2024-04-19T00:49:10.871 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-19T00:49:10.871 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi --description rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} --archive /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663223 --owner scheduled_yuriw@teuthology --verbose -- /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663223/orig.config.yaml 2024-04-19T00:49:10.914 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-19T00:49:11.003 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663223 branch: wip-yuri5-testing-2024-04-17-1400 description: rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} email: yweinste@redhat.com first_in_suite: false job_id: '7663223' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-yuri5-testing-2024-04-17-1400 ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 2500 ms tcp read timeout: 5 osd_async_recovery_min_cost: 1 osd_object_clean_region_max_num_intervals: 1000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore block size: 96636764160 bluestore compression algorithm: zlib bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd max backfills: 3 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - slow request - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum sha1: 8b4ab94f9c30da63487fa2f241ce12f483378860 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 8b4ab94f9c30da63487fa2f241ce12f483378860 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri5-testing-2024-04-17-1400 sha1: 8b4ab94f9c30da63487fa2f241ce12f483378860 owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.4 - osd.8 - osd.12 - - mon.b - osd.1 - osd.5 - osd.9 - osd.13 - - mon.c - osd.2 - osd.6 - osd.10 - osd.14 - - mgr.x - osd.3 - osd.7 - osd.11 - osd.15 - client.0 seed: 1937 sha1: 8b4ab94f9c30da63487fa2f241ce12f483378860 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri5-testing-2024-04-17-1400 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_8b4ab94f9c30da63487fa2f241ce12f483378860/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 8b4ab94f9c30da63487fa2f241ce12f483378860 targets: smithi033.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLpj9Qk/UG0JRgamdP9/Iob2qWzoJv6O9SKRCTlLstPpnJX2lcDw5VMd2tdADQnRV0XFb8kkOmy/hZOWz91kKvA= smithi110.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMpT4SWuPk3vehHOSyYehjC5rmtrs9bqSDnqYxND2vcHtiEpikXMu814iob8v7tQQX3zntntxYotfbUXKXYd85A= smithi118.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFDPX1v+vTbcLofsXdEKMfDHoJAKbxRJZMaJtKb4xIkhtkXyQQxepVG8bmh95EH8s6MV4IwVR2BmzHuyM+6UE8A= smithi155.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMQRDx7ifdlfs6P34mYZ0RVkeh/us0Z15Iz4nF0HRtCT+ftN1LRVRF4gmhst0dx6WW4tHqOwvAgRJtl/cPZWoR8= tasks: - install: null - ceph: conf: osd: debug monc: 20 - thrashosds: aggressive_pg_num_changes: false chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 min_in: 8 timeout: 1200 - rados: clients: - client.0 ec_pool: true erasure_code_profile: c: 2 crush-failure-domain: osd k: 4 m: 3 name: shecprofile plugin: shec objects: 50 op_weights: append: 100 copy_from: 50 delete: 50 read: 100 rmattr: 25 rollback: 50 setattr: 25 snap_create: 50 snap_remove: 50 write: 0 ops: 400 write_append_excl: false teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-18_22:49:05 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-19T00:49:11.003 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_8b4ab94f9c30da63487fa2f241ce12f483378860/qa; will attempt to use it 2024-04-19T00:49:11.004 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_8b4ab94f9c30da63487fa2f241ce12f483378860/qa/tasks 2024-04-19T00:49:11.004 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-19T00:49:11.006 INFO:teuthology.task.internal:Checking packages... 2024-04-19T00:49:11.030 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '8b4ab94f9c30da63487fa2f241ce12f483378860' 2024-04-19T00:49:11.030 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-19T00:49:11.030 INFO:teuthology.packaging:ref: None 2024-04-19T00:49:11.031 INFO:teuthology.packaging:tag: None 2024-04-19T00:49:11.031 INFO:teuthology.packaging:branch: wip-yuri5-testing-2024-04-17-1400 2024-04-19T00:49:11.031 INFO:teuthology.packaging:sha1: 8b4ab94f9c30da63487fa2f241ce12f483378860 2024-04-19T00:49:11.031 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-yuri5-testing-2024-04-17-1400 2024-04-19T00:49:11.263 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3154-g8b4ab94f-1jammy 2024-04-19T00:49:11.264 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-19T00:49:11.272 INFO:teuthology.task.internal:no buildpackages task found 2024-04-19T00:49:11.272 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-19T00:49:11.294 INFO:teuthology.task.internal:Saving configuration 2024-04-19T00:49:11.310 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-19T00:49:11.316 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-19T00:49:11.340 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi033.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663223', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-19 00:42:36.985771', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKELus2sUWh2YMpyp8DsfBOyEq7NqKSKGnqJBXnlt+NT'} 2024-04-19T00:49:11.360 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi110.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663223', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-19 00:42:36.988283', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOuhFmDb/6ni7X2Vs/sqKDSmDX3lrCLlYSwGh4k5ovqF'} 2024-04-19T00:49:11.385 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi118.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663223', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-19 00:42:36.983853', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBQ1fbeB3aFz9sq5/s0tQY/HhafmucitoFgoBw0gR6tT'} 2024-04-19T00:49:11.404 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi155.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663223', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-19 00:42:36.987208', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBbAQu8VGnr8MtkgofjvlhK8Ou/z6BzJiJ9Jfz6x+Gec'} 2024-04-19T00:49:11.404 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-19T00:49:11.410 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'] 2024-04-19T00:49:11.410 INFO:teuthology.task.internal:roles: ubuntu@smithi110.front.sepia.ceph.com - ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'] 2024-04-19T00:49:11.410 INFO:teuthology.task.internal:roles: ubuntu@smithi118.front.sepia.ceph.com - ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'] 2024-04-19T00:49:11.410 INFO:teuthology.task.internal:roles: ubuntu@smithi155.front.sepia.ceph.com - ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0'] 2024-04-19T00:49:11.410 INFO:teuthology.run_tasks:Running task console_log... 2024-04-19T00:49:11.544 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f52672b23a0>, signals=[15]) 2024-04-19T00:49:11.545 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-19T00:49:11.551 INFO:teuthology.task.internal:Opening connections... 2024-04-19T00:49:11.551 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi033.front.sepia.ceph.com 2024-04-19T00:49:11.553 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T00:49:11.638 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi110.front.sepia.ceph.com 2024-04-19T00:49:11.639 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T00:49:11.716 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi118.front.sepia.ceph.com 2024-04-19T00:49:11.717 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T00:49:11.797 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi155.front.sepia.ceph.com 2024-04-19T00:49:11.798 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T00:49:11.877 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-19T00:49:11.884 DEBUG:teuthology.orchestra.run.smithi033:> uname -m 2024-04-19T00:49:11.891 INFO:teuthology.orchestra.run.smithi033.stdout:x86_64 2024-04-19T00:49:11.891 DEBUG:teuthology.orchestra.run.smithi033:> cat /etc/os-release 2024-04-19T00:49:11.936 INFO:teuthology.orchestra.run.smithi033.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-19T00:49:11.936 INFO:teuthology.orchestra.run.smithi033.stdout:NAME="Ubuntu" 2024-04-19T00:49:11.936 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION_ID="22.04" 2024-04-19T00:49:11.936 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-19T00:49:11.936 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION_CODENAME=jammy 2024-04-19T00:49:11.936 INFO:teuthology.orchestra.run.smithi033.stdout:ID=ubuntu 2024-04-19T00:49:11.937 INFO:teuthology.orchestra.run.smithi033.stdout:ID_LIKE=debian 2024-04-19T00:49:11.937 INFO:teuthology.orchestra.run.smithi033.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-19T00:49:11.937 INFO:teuthology.orchestra.run.smithi033.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-19T00:49:11.937 INFO:teuthology.orchestra.run.smithi033.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-19T00:49:11.937 INFO:teuthology.orchestra.run.smithi033.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-19T00:49:11.937 INFO:teuthology.orchestra.run.smithi033.stdout:UBUNTU_CODENAME=jammy 2024-04-19T00:49:11.938 INFO:teuthology.lock.ops:Updating smithi033.front.sepia.ceph.com on lock server 2024-04-19T00:49:11.961 DEBUG:teuthology.orchestra.run.smithi110:> uname -m 2024-04-19T00:49:11.967 INFO:teuthology.orchestra.run.smithi110.stdout:x86_64 2024-04-19T00:49:11.967 DEBUG:teuthology.orchestra.run.smithi110:> cat /etc/os-release 2024-04-19T00:49:12.014 INFO:teuthology.orchestra.run.smithi110.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-19T00:49:12.014 INFO:teuthology.orchestra.run.smithi110.stdout:NAME="Ubuntu" 2024-04-19T00:49:12.014 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION_ID="22.04" 2024-04-19T00:49:12.015 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-19T00:49:12.015 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION_CODENAME=jammy 2024-04-19T00:49:12.015 INFO:teuthology.orchestra.run.smithi110.stdout:ID=ubuntu 2024-04-19T00:49:12.015 INFO:teuthology.orchestra.run.smithi110.stdout:ID_LIKE=debian 2024-04-19T00:49:12.015 INFO:teuthology.orchestra.run.smithi110.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-19T00:49:12.015 INFO:teuthology.orchestra.run.smithi110.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-19T00:49:12.015 INFO:teuthology.orchestra.run.smithi110.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-19T00:49:12.015 INFO:teuthology.orchestra.run.smithi110.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-19T00:49:12.015 INFO:teuthology.orchestra.run.smithi110.stdout:UBUNTU_CODENAME=jammy 2024-04-19T00:49:12.015 INFO:teuthology.lock.ops:Updating smithi110.front.sepia.ceph.com on lock server 2024-04-19T00:49:12.036 DEBUG:teuthology.orchestra.run.smithi118:> uname -m 2024-04-19T00:49:12.042 INFO:teuthology.orchestra.run.smithi118.stdout:x86_64 2024-04-19T00:49:12.042 DEBUG:teuthology.orchestra.run.smithi118:> cat /etc/os-release 2024-04-19T00:49:12.089 INFO:teuthology.orchestra.run.smithi118.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-19T00:49:12.089 INFO:teuthology.orchestra.run.smithi118.stdout:NAME="Ubuntu" 2024-04-19T00:49:12.089 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION_ID="22.04" 2024-04-19T00:49:12.090 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-19T00:49:12.090 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION_CODENAME=jammy 2024-04-19T00:49:12.090 INFO:teuthology.orchestra.run.smithi118.stdout:ID=ubuntu 2024-04-19T00:49:12.090 INFO:teuthology.orchestra.run.smithi118.stdout:ID_LIKE=debian 2024-04-19T00:49:12.090 INFO:teuthology.orchestra.run.smithi118.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-19T00:49:12.090 INFO:teuthology.orchestra.run.smithi118.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-19T00:49:12.090 INFO:teuthology.orchestra.run.smithi118.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-19T00:49:12.090 INFO:teuthology.orchestra.run.smithi118.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-19T00:49:12.090 INFO:teuthology.orchestra.run.smithi118.stdout:UBUNTU_CODENAME=jammy 2024-04-19T00:49:12.090 INFO:teuthology.lock.ops:Updating smithi118.front.sepia.ceph.com on lock server 2024-04-19T00:49:12.111 DEBUG:teuthology.orchestra.run.smithi155:> uname -m 2024-04-19T00:49:12.116 INFO:teuthology.orchestra.run.smithi155.stdout:x86_64 2024-04-19T00:49:12.117 DEBUG:teuthology.orchestra.run.smithi155:> cat /etc/os-release 2024-04-19T00:49:12.165 INFO:teuthology.orchestra.run.smithi155.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-19T00:49:12.165 INFO:teuthology.orchestra.run.smithi155.stdout:NAME="Ubuntu" 2024-04-19T00:49:12.165 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION_ID="22.04" 2024-04-19T00:49:12.165 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-19T00:49:12.165 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION_CODENAME=jammy 2024-04-19T00:49:12.165 INFO:teuthology.orchestra.run.smithi155.stdout:ID=ubuntu 2024-04-19T00:49:12.165 INFO:teuthology.orchestra.run.smithi155.stdout:ID_LIKE=debian 2024-04-19T00:49:12.165 INFO:teuthology.orchestra.run.smithi155.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-19T00:49:12.165 INFO:teuthology.orchestra.run.smithi155.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-19T00:49:12.165 INFO:teuthology.orchestra.run.smithi155.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-19T00:49:12.165 INFO:teuthology.orchestra.run.smithi155.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-19T00:49:12.166 INFO:teuthology.orchestra.run.smithi155.stdout:UBUNTU_CODENAME=jammy 2024-04-19T00:49:12.166 INFO:teuthology.lock.ops:Updating smithi155.front.sepia.ceph.com on lock server 2024-04-19T00:49:12.186 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-19T00:49:12.195 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-19T00:49:12.201 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-19T00:49:12.201 DEBUG:teuthology.orchestra.run.smithi033:> test '!' -e /home/ubuntu/cephtest 2024-04-19T00:49:12.203 DEBUG:teuthology.orchestra.run.smithi110:> test '!' -e /home/ubuntu/cephtest 2024-04-19T00:49:12.206 DEBUG:teuthology.orchestra.run.smithi118:> test '!' -e /home/ubuntu/cephtest 2024-04-19T00:49:12.209 DEBUG:teuthology.orchestra.run.smithi155:> test '!' -e /home/ubuntu/cephtest 2024-04-19T00:49:12.214 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-19T00:49:12.219 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-19T00:49:12.219 DEBUG:teuthology.orchestra.run.smithi033:> test -z $(ls -A /var/lib/ceph) 2024-04-19T00:49:12.249 DEBUG:teuthology.orchestra.run.smithi110:> test -z $(ls -A /var/lib/ceph) 2024-04-19T00:49:12.252 DEBUG:teuthology.orchestra.run.smithi118:> test -z $(ls -A /var/lib/ceph) 2024-04-19T00:49:12.255 DEBUG:teuthology.orchestra.run.smithi155:> test -z $(ls -A /var/lib/ceph) 2024-04-19T00:49:12.290 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-19T00:49:12.385 INFO:teuthology.run_tasks:Running task kernel... 2024-04-19T00:49:12.399 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-19T00:49:12.399 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}, 'mgr.x': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-19T00:49:12.400 DEBUG:teuthology.orchestra.run.smithi033:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-19T00:49:12.400 DEBUG:teuthology.orchestra.run.smithi110:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-19T00:49:12.401 DEBUG:teuthology.orchestra.run.smithi118:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-19T00:49:12.401 DEBUG:teuthology.orchestra.run.smithi155:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-19T00:49:12.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T00:49:12.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T00:49:12.407 DEBUG:teuthology.orchestra.run.smithi033:> uname -r 2024-04-19T00:49:12.407 DEBUG:teuthology.orchestra.run.smithi118:> uname -r 2024-04-19T00:49:12.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T00:49:12.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T00:49:12.409 DEBUG:teuthology.orchestra.run.smithi155:> uname -r 2024-04-19T00:49:12.409 DEBUG:teuthology.orchestra.run.smithi110:> uname -r 2024-04-19T00:49:12.453 INFO:teuthology.orchestra.run.smithi033.stdout:5.15.0-56-generic 2024-04-19T00:49:12.453 INFO:teuthology.task.kernel:Running kernel on smithi033: 5.15.0-56-generic 2024-04-19T00:49:12.453 DEBUG:teuthology.orchestra.run.smithi033:> sudo apt-get clean 2024-04-19T00:49:12.455 INFO:teuthology.orchestra.run.smithi118.stdout:5.15.0-56-generic 2024-04-19T00:49:12.456 INFO:teuthology.orchestra.run.smithi110.stdout:5.15.0-56-generic 2024-04-19T00:49:12.456 INFO:teuthology.task.kernel:Running kernel on smithi118: 5.15.0-56-generic 2024-04-19T00:49:12.456 DEBUG:teuthology.orchestra.run.smithi118:> sudo apt-get clean 2024-04-19T00:49:12.456 INFO:teuthology.task.kernel:Running kernel on smithi110: 5.15.0-56-generic 2024-04-19T00:49:12.456 DEBUG:teuthology.orchestra.run.smithi110:> sudo apt-get clean 2024-04-19T00:49:12.457 INFO:teuthology.orchestra.run.smithi155.stdout:5.15.0-56-generic 2024-04-19T00:49:12.458 INFO:teuthology.task.kernel:Running kernel on smithi155: 5.15.0-56-generic 2024-04-19T00:49:12.458 DEBUG:teuthology.orchestra.run.smithi155:> sudo apt-get clean 2024-04-19T00:49:12.581 DEBUG:teuthology.orchestra.run.smithi110:> sudo apt-get update 2024-04-19T00:49:12.598 DEBUG:teuthology.orchestra.run.smithi033:> sudo apt-get update 2024-04-19T00:49:12.599 DEBUG:teuthology.orchestra.run.smithi155:> sudo apt-get update 2024-04-19T00:49:12.624 DEBUG:teuthology.orchestra.run.smithi118:> sudo apt-get update 2024-04-19T00:49:12.758 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-19T00:49:12.766 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-19T00:49:12.771 INFO:teuthology.orchestra.run.smithi033.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-19T00:49:12.774 INFO:teuthology.orchestra.run.smithi118.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-19T00:49:12.788 INFO:teuthology.orchestra.run.smithi155.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-19T00:49:12.850 INFO:teuthology.orchestra.run.smithi033.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-19T00:49:12.860 INFO:teuthology.orchestra.run.smithi155.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-19T00:49:12.866 INFO:teuthology.orchestra.run.smithi033.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-19T00:49:12.874 INFO:teuthology.orchestra.run.smithi155.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-19T00:49:12.881 INFO:teuthology.orchestra.run.smithi118.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-19T00:49:12.884 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-19T00:49:12.885 INFO:teuthology.orchestra.run.smithi110.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-19T00:49:12.930 INFO:teuthology.orchestra.run.smithi110.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-19T00:49:12.963 INFO:teuthology.orchestra.run.smithi118.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-19T00:49:12.994 INFO:teuthology.orchestra.run.smithi118.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,351 kB] 2024-04-19T00:49:13.012 INFO:teuthology.orchestra.run.smithi155.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,597 kB] 2024-04-19T00:49:13.017 INFO:teuthology.orchestra.run.smithi033.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [616 kB] 2024-04-19T00:49:13.098 INFO:teuthology.orchestra.run.smithi033.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,597 kB] 2024-04-19T00:49:13.109 INFO:teuthology.orchestra.run.smithi118.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [445 kB] 2024-04-19T00:49:13.115 INFO:teuthology.orchestra.run.smithi118.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [238 kB] 2024-04-19T00:49:13.118 INFO:teuthology.orchestra.run.smithi118.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-19T00:49:13.118 INFO:teuthology.orchestra.run.smithi118.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-19T00:49:13.118 INFO:teuthology.orchestra.run.smithi118.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,680 kB] 2024-04-19T00:49:13.140 INFO:teuthology.orchestra.run.smithi033.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [302 kB] 2024-04-19T00:49:13.143 INFO:teuthology.orchestra.run.smithi118.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [281 kB] 2024-04-19T00:49:13.147 INFO:teuthology.orchestra.run.smithi033.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-19T00:49:13.147 INFO:teuthology.orchestra.run.smithi033.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-19T00:49:13.147 INFO:teuthology.orchestra.run.smithi118.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-19T00:49:13.148 INFO:teuthology.orchestra.run.smithi118.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-19T00:49:13.148 INFO:teuthology.orchestra.run.smithi033.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,806 kB] 2024-04-19T00:49:13.155 INFO:teuthology.orchestra.run.smithi155.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [616 kB] 2024-04-19T00:49:13.166 INFO:teuthology.orchestra.run.smithi118.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [600 kB] 2024-04-19T00:49:13.173 INFO:teuthology.orchestra.run.smithi155.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [302 kB] 2024-04-19T00:49:13.178 INFO:teuthology.orchestra.run.smithi118.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [162 kB] 2024-04-19T00:49:13.180 INFO:teuthology.orchestra.run.smithi118.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-19T00:49:13.181 INFO:teuthology.orchestra.run.smithi155.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-19T00:49:13.182 INFO:teuthology.orchestra.run.smithi155.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-19T00:49:13.183 INFO:teuthology.orchestra.run.smithi155.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,806 kB] 2024-04-19T00:49:13.187 INFO:teuthology.orchestra.run.smithi118.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-19T00:49:13.188 INFO:teuthology.orchestra.run.smithi118.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-19T00:49:13.188 INFO:teuthology.orchestra.run.smithi118.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-19T00:49:13.189 INFO:teuthology.orchestra.run.smithi033.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [305 kB] 2024-04-19T00:49:13.192 INFO:teuthology.orchestra.run.smithi033.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-19T00:49:13.193 INFO:teuthology.orchestra.run.smithi033.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,070 kB] 2024-04-19T00:49:13.205 INFO:teuthology.orchestra.run.smithi033.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [699 kB] 2024-04-19T00:49:13.226 INFO:teuthology.orchestra.run.smithi155.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [305 kB] 2024-04-19T00:49:13.229 INFO:teuthology.orchestra.run.smithi155.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-19T00:49:13.229 INFO:teuthology.orchestra.run.smithi155.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,070 kB] 2024-04-19T00:49:13.241 INFO:teuthology.orchestra.run.smithi155.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [699 kB] 2024-04-19T00:49:13.245 INFO:teuthology.orchestra.run.smithi118.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-19T00:49:13.249 INFO:teuthology.orchestra.run.smithi155.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-19T00:49:13.252 INFO:teuthology.orchestra.run.smithi155.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-19T00:49:13.252 INFO:teuthology.orchestra.run.smithi155.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-19T00:49:13.253 INFO:teuthology.orchestra.run.smithi155.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-19T00:49:13.270 INFO:teuthology.orchestra.run.smithi155.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-19T00:49:13.271 INFO:teuthology.orchestra.run.smithi155.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-19T00:49:13.271 INFO:teuthology.orchestra.run.smithi155.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-19T00:49:13.272 INFO:teuthology.orchestra.run.smithi155.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-04-19T00:49:13.273 INFO:teuthology.orchestra.run.smithi155.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-19T00:49:13.280 INFO:teuthology.orchestra.run.smithi033.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [445 kB] 2024-04-19T00:49:13.285 INFO:teuthology.orchestra.run.smithi155.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-19T00:49:13.285 INFO:teuthology.orchestra.run.smithi155.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-04-19T00:49:13.286 INFO:teuthology.orchestra.run.smithi155.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-04-19T00:49:13.288 INFO:teuthology.orchestra.run.smithi110.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,351 kB] 2024-04-19T00:49:13.296 INFO:teuthology.orchestra.run.smithi110.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-19T00:49:13.305 INFO:teuthology.orchestra.run.smithi155.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-19T00:49:13.305 INFO:teuthology.orchestra.run.smithi155.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-19T00:49:13.308 INFO:teuthology.orchestra.run.smithi155.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,351 kB] 2024-04-19T00:49:13.311 INFO:teuthology.orchestra.run.smithi033.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-19T00:49:13.318 INFO:teuthology.orchestra.run.smithi033.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-19T00:49:13.320 INFO:teuthology.orchestra.run.smithi033.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-19T00:49:13.321 INFO:teuthology.orchestra.run.smithi033.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-19T00:49:13.338 INFO:teuthology.orchestra.run.smithi118.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-19T00:49:13.349 INFO:teuthology.orchestra.run.smithi033.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-19T00:49:13.350 INFO:teuthology.orchestra.run.smithi033.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-19T00:49:13.351 INFO:teuthology.orchestra.run.smithi033.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-04-19T00:49:13.353 INFO:teuthology.orchestra.run.smithi033.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-19T00:49:13.354 INFO:teuthology.orchestra.run.smithi033.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-19T00:49:13.355 INFO:teuthology.orchestra.run.smithi033.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-19T00:49:13.355 INFO:teuthology.orchestra.run.smithi033.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-04-19T00:49:13.356 INFO:teuthology.orchestra.run.smithi033.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-04-19T00:49:13.372 INFO:teuthology.orchestra.run.smithi033.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-19T00:49:13.373 INFO:teuthology.orchestra.run.smithi033.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-19T00:49:13.435 INFO:teuthology.orchestra.run.smithi110.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,597 kB] 2024-04-19T00:49:13.478 INFO:teuthology.orchestra.run.smithi118.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,597 kB] 2024-04-19T00:49:13.562 INFO:teuthology.orchestra.run.smithi033.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,351 kB] 2024-04-19T00:49:13.716 INFO:teuthology.orchestra.run.smithi155.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [445 kB] 2024-04-19T00:49:13.729 INFO:teuthology.orchestra.run.smithi033.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [238 kB] 2024-04-19T00:49:13.734 INFO:teuthology.orchestra.run.smithi155.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [238 kB] 2024-04-19T00:49:13.742 INFO:teuthology.orchestra.run.smithi033.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-19T00:49:13.742 INFO:teuthology.orchestra.run.smithi033.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-19T00:49:13.743 INFO:teuthology.orchestra.run.smithi155.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-19T00:49:13.744 INFO:teuthology.orchestra.run.smithi033.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,680 kB] 2024-04-19T00:49:13.744 INFO:teuthology.orchestra.run.smithi155.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-19T00:49:13.746 INFO:teuthology.orchestra.run.smithi155.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,680 kB] 2024-04-19T00:49:13.756 INFO:teuthology.orchestra.run.smithi110.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [616 kB] 2024-04-19T00:49:13.783 INFO:teuthology.orchestra.run.smithi110.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [302 kB] 2024-04-19T00:49:13.798 INFO:teuthology.orchestra.run.smithi110.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-19T00:49:13.799 INFO:teuthology.orchestra.run.smithi110.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-19T00:49:13.800 INFO:teuthology.orchestra.run.smithi118.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [616 kB] 2024-04-19T00:49:13.801 INFO:teuthology.orchestra.run.smithi110.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,806 kB] 2024-04-19T00:49:13.825 INFO:teuthology.orchestra.run.smithi033.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [281 kB] 2024-04-19T00:49:13.832 INFO:teuthology.orchestra.run.smithi033.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-19T00:49:13.833 INFO:teuthology.orchestra.run.smithi033.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [600 kB] 2024-04-19T00:49:13.838 INFO:teuthology.orchestra.run.smithi155.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [281 kB] 2024-04-19T00:49:13.842 INFO:teuthology.orchestra.run.smithi118.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [302 kB] 2024-04-19T00:49:13.848 INFO:teuthology.orchestra.run.smithi033.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-19T00:49:13.849 INFO:teuthology.orchestra.run.smithi155.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-19T00:49:13.849 INFO:teuthology.orchestra.run.smithi155.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [600 kB] 2024-04-19T00:49:13.861 INFO:teuthology.orchestra.run.smithi118.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-19T00:49:13.861 INFO:teuthology.orchestra.run.smithi118.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-19T00:49:13.862 INFO:teuthology.orchestra.run.smithi110.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [305 kB] 2024-04-19T00:49:13.863 INFO:teuthology.orchestra.run.smithi118.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,806 kB] 2024-04-19T00:49:13.869 INFO:teuthology.orchestra.run.smithi110.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-19T00:49:13.870 INFO:teuthology.orchestra.run.smithi155.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-19T00:49:14.032 INFO:teuthology.orchestra.run.smithi033.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [162 kB] 2024-04-19T00:49:14.033 INFO:teuthology.orchestra.run.smithi110.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [699 kB] 2024-04-19T00:49:14.056 INFO:teuthology.orchestra.run.smithi155.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [162 kB] 2024-04-19T00:49:14.438 INFO:teuthology.orchestra.run.smithi033.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-19T00:49:14.444 INFO:teuthology.orchestra.run.smithi033.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-19T00:49:14.458 INFO:teuthology.orchestra.run.smithi033.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-19T00:49:14.458 INFO:teuthology.orchestra.run.smithi033.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-19T00:49:14.461 INFO:teuthology.orchestra.run.smithi118.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [305 kB] 2024-04-19T00:49:14.461 INFO:teuthology.orchestra.run.smithi033.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-19T00:49:14.462 INFO:teuthology.orchestra.run.smithi110.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [445 kB] 2024-04-19T00:49:14.464 INFO:teuthology.orchestra.run.smithi155.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-19T00:49:14.470 INFO:teuthology.orchestra.run.smithi155.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-19T00:49:14.471 INFO:teuthology.orchestra.run.smithi155.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-19T00:49:14.485 INFO:teuthology.orchestra.run.smithi155.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-19T00:49:14.488 INFO:teuthology.orchestra.run.smithi155.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-19T00:49:14.540 INFO:teuthology.orchestra.run.smithi118.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-19T00:49:14.616 INFO:teuthology.orchestra.run.smithi110.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,070 kB] 2024-04-19T00:49:14.658 INFO:teuthology.orchestra.run.smithi110.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [238 kB] 2024-04-19T00:49:14.705 INFO:teuthology.orchestra.run.smithi118.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [699 kB] 2024-04-19T00:49:14.727 INFO:teuthology.orchestra.run.smithi110.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-19T00:49:14.729 INFO:teuthology.orchestra.run.smithi110.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-19T00:49:14.739 INFO:teuthology.orchestra.run.smithi110.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,680 kB] 2024-04-19T00:49:14.884 INFO:teuthology.orchestra.run.smithi110.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-19T00:49:14.904 INFO:teuthology.orchestra.run.smithi110.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-19T00:49:14.906 INFO:teuthology.orchestra.run.smithi110.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-19T00:49:14.910 INFO:teuthology.orchestra.run.smithi110.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-19T00:49:14.910 INFO:teuthology.orchestra.run.smithi110.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-19T00:49:14.911 INFO:teuthology.orchestra.run.smithi110.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-19T00:49:14.912 INFO:teuthology.orchestra.run.smithi110.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-19T00:49:14.917 INFO:teuthology.orchestra.run.smithi110.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-04-19T00:49:14.922 INFO:teuthology.orchestra.run.smithi110.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-19T00:49:15.066 INFO:teuthology.orchestra.run.smithi110.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [281 kB] 2024-04-19T00:49:15.089 INFO:teuthology.orchestra.run.smithi110.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-19T00:49:15.107 INFO:teuthology.orchestra.run.smithi110.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-19T00:49:15.107 INFO:teuthology.orchestra.run.smithi110.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [600 kB] 2024-04-19T00:49:15.171 INFO:teuthology.orchestra.run.smithi110.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-04-19T00:49:15.184 INFO:teuthology.orchestra.run.smithi110.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-19T00:49:15.273 INFO:teuthology.orchestra.run.smithi110.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-04-19T00:49:15.292 INFO:teuthology.orchestra.run.smithi118.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,070 kB] 2024-04-19T00:49:15.334 INFO:teuthology.orchestra.run.smithi110.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-19T00:49:15.363 INFO:teuthology.orchestra.run.smithi110.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-19T00:49:15.403 INFO:teuthology.orchestra.run.smithi118.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-19T00:49:15.423 INFO:teuthology.orchestra.run.smithi118.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-19T00:49:15.425 INFO:teuthology.orchestra.run.smithi118.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-19T00:49:15.428 INFO:teuthology.orchestra.run.smithi118.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-19T00:49:15.429 INFO:teuthology.orchestra.run.smithi118.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-19T00:49:15.430 INFO:teuthology.orchestra.run.smithi118.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-19T00:49:15.430 INFO:teuthology.orchestra.run.smithi118.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-04-19T00:49:15.433 INFO:teuthology.orchestra.run.smithi118.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-19T00:49:15.438 INFO:teuthology.orchestra.run.smithi118.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-19T00:49:15.443 INFO:teuthology.orchestra.run.smithi110.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [162 kB] 2024-04-19T00:49:15.602 INFO:teuthology.orchestra.run.smithi118.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-19T00:49:15.685 INFO:teuthology.orchestra.run.smithi118.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-04-19T00:49:15.769 INFO:teuthology.orchestra.run.smithi118.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-04-19T00:49:15.850 INFO:teuthology.orchestra.run.smithi118.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-19T00:49:15.857 INFO:teuthology.orchestra.run.smithi110.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-19T00:49:15.863 INFO:teuthology.orchestra.run.smithi110.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-19T00:49:15.864 INFO:teuthology.orchestra.run.smithi110.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-19T00:49:15.878 INFO:teuthology.orchestra.run.smithi110.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-19T00:49:15.881 INFO:teuthology.orchestra.run.smithi110.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-19T00:49:15.885 INFO:teuthology.orchestra.run.smithi118.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-19T00:49:19.103 INFO:teuthology.orchestra.run.smithi155.stdout:Fetched 13.0 MB in 2s (6,855 kB/s) 2024-04-19T00:49:19.158 INFO:teuthology.orchestra.run.smithi033.stdout:Fetched 13.0 MB in 2s (6,850 kB/s) 2024-04-19T00:49:20.067 INFO:teuthology.orchestra.run.smithi118.stdout:Fetched 13.0 MB in 3s (4,086 kB/s) 2024-04-19T00:49:20.136 INFO:teuthology.orchestra.run.smithi110.stdout:Fetched 13.0 MB in 3s (4,051 kB/s) 2024-04-19T00:49:20.598 INFO:teuthology.orchestra.run.smithi155.stdout:Reading package lists... 2024-04-19T00:49:20.617 DEBUG:teuthology.orchestra.run.smithi155:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-19T00:49:20.632 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2024-04-19T00:49:20.650 DEBUG:teuthology.orchestra.run.smithi033:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-19T00:49:20.681 INFO:teuthology.orchestra.run.smithi155.stdout:Reading package lists... 2024-04-19T00:49:20.715 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2024-04-19T00:49:20.880 INFO:teuthology.orchestra.run.smithi155.stdout:Building dependency tree... 2024-04-19T00:49:20.881 INFO:teuthology.orchestra.run.smithi155.stdout:Reading state information... 2024-04-19T00:49:20.922 INFO:teuthology.orchestra.run.smithi033.stdout:Building dependency tree... 2024-04-19T00:49:20.923 INFO:teuthology.orchestra.run.smithi033.stdout:Reading state information... 2024-04-19T00:49:21.058 INFO:teuthology.orchestra.run.smithi155.stdout:The following packages were automatically installed and are no longer required: 2024-04-19T00:49:21.058 INFO:teuthology.orchestra.run.smithi155.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-19T00:49:21.058 INFO:teuthology.orchestra.run.smithi155.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-19T00:49:21.059 INFO:teuthology.orchestra.run.smithi155.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-19T00:49:21.059 INFO:teuthology.orchestra.run.smithi155.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-19T00:49:21.059 INFO:teuthology.orchestra.run.smithi155.stdout: python2.7-minimal 2024-04-19T00:49:21.059 INFO:teuthology.orchestra.run.smithi155.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-19T00:49:21.060 INFO:teuthology.orchestra.run.smithi155.stdout:The following additional packages will be installed: 2024-04-19T00:49:21.060 INFO:teuthology.orchestra.run.smithi155.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-19T00:49:21.060 INFO:teuthology.orchestra.run.smithi155.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-19T00:49:21.061 INFO:teuthology.orchestra.run.smithi155.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-19T00:49:21.062 INFO:teuthology.orchestra.run.smithi155.stdout:Suggested packages: 2024-04-19T00:49:21.062 INFO:teuthology.orchestra.run.smithi155.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-19T00:49:21.062 INFO:teuthology.orchestra.run.smithi155.stdout:Recommended packages: 2024-04-19T00:49:21.062 INFO:teuthology.orchestra.run.smithi155.stdout: thermald 2024-04-19T00:49:21.103 INFO:teuthology.orchestra.run.smithi033.stdout:The following packages were automatically installed and are no longer required: 2024-04-19T00:49:21.103 INFO:teuthology.orchestra.run.smithi033.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-19T00:49:21.103 INFO:teuthology.orchestra.run.smithi033.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-19T00:49:21.104 INFO:teuthology.orchestra.run.smithi033.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-19T00:49:21.104 INFO:teuthology.orchestra.run.smithi033.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-19T00:49:21.104 INFO:teuthology.orchestra.run.smithi033.stdout: python2.7-minimal 2024-04-19T00:49:21.104 INFO:teuthology.orchestra.run.smithi033.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-19T00:49:21.105 INFO:teuthology.orchestra.run.smithi155.stdout:The following NEW packages will be installed: 2024-04-19T00:49:21.105 INFO:teuthology.orchestra.run.smithi155.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-19T00:49:21.106 INFO:teuthology.orchestra.run.smithi155.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-19T00:49:21.106 INFO:teuthology.orchestra.run.smithi155.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-19T00:49:21.106 INFO:teuthology.orchestra.run.smithi033.stdout:The following additional packages will be installed: 2024-04-19T00:49:21.106 INFO:teuthology.orchestra.run.smithi033.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-19T00:49:21.107 INFO:teuthology.orchestra.run.smithi033.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-19T00:49:21.107 INFO:teuthology.orchestra.run.smithi033.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-19T00:49:21.107 INFO:teuthology.orchestra.run.smithi155.stdout:The following packages will be upgraded: 2024-04-19T00:49:21.107 INFO:teuthology.orchestra.run.smithi033.stdout:Suggested packages: 2024-04-19T00:49:21.107 INFO:teuthology.orchestra.run.smithi033.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-19T00:49:21.108 INFO:teuthology.orchestra.run.smithi033.stdout:Recommended packages: 2024-04-19T00:49:21.108 INFO:teuthology.orchestra.run.smithi033.stdout: thermald 2024-04-19T00:49:21.108 INFO:teuthology.orchestra.run.smithi155.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-19T00:49:21.153 INFO:teuthology.orchestra.run.smithi033.stdout:The following NEW packages will be installed: 2024-04-19T00:49:21.153 INFO:teuthology.orchestra.run.smithi033.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-19T00:49:21.154 INFO:teuthology.orchestra.run.smithi033.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-19T00:49:21.154 INFO:teuthology.orchestra.run.smithi033.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-19T00:49:21.155 INFO:teuthology.orchestra.run.smithi033.stdout:The following packages will be upgraded: 2024-04-19T00:49:21.156 INFO:teuthology.orchestra.run.smithi033.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-19T00:49:21.287 INFO:teuthology.orchestra.run.smithi155.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-19T00:49:21.287 INFO:teuthology.orchestra.run.smithi155.stdout:Need to get 113 MB of archives. 2024-04-19T00:49:21.287 INFO:teuthology.orchestra.run.smithi155.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-19T00:49:21.287 INFO:teuthology.orchestra.run.smithi155.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-19T00:49:21.337 INFO:teuthology.orchestra.run.smithi033.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-19T00:49:21.337 INFO:teuthology.orchestra.run.smithi033.stdout:Need to get 113 MB of archives. 2024-04-19T00:49:21.337 INFO:teuthology.orchestra.run.smithi033.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-19T00:49:21.337 INFO:teuthology.orchestra.run.smithi033.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-19T00:49:21.356 INFO:teuthology.orchestra.run.smithi118.stdout:Reading package lists... 2024-04-19T00:49:21.372 DEBUG:teuthology.orchestra.run.smithi118:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-19T00:49:21.433 INFO:teuthology.orchestra.run.smithi118.stdout:Reading package lists... 2024-04-19T00:49:21.524 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2024-04-19T00:49:21.541 DEBUG:teuthology.orchestra.run.smithi110:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-19T00:49:21.599 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2024-04-19T00:49:21.633 INFO:teuthology.orchestra.run.smithi118.stdout:Building dependency tree... 2024-04-19T00:49:21.633 INFO:teuthology.orchestra.run.smithi118.stdout:Reading state information... 2024-04-19T00:49:21.798 INFO:teuthology.orchestra.run.smithi110.stdout:Building dependency tree... 2024-04-19T00:49:21.799 INFO:teuthology.orchestra.run.smithi110.stdout:Reading state information... 2024-04-19T00:49:21.803 INFO:teuthology.orchestra.run.smithi118.stdout:The following packages were automatically installed and are no longer required: 2024-04-19T00:49:21.803 INFO:teuthology.orchestra.run.smithi118.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-19T00:49:21.804 INFO:teuthology.orchestra.run.smithi118.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-19T00:49:21.804 INFO:teuthology.orchestra.run.smithi118.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-19T00:49:21.804 INFO:teuthology.orchestra.run.smithi118.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-19T00:49:21.804 INFO:teuthology.orchestra.run.smithi118.stdout: python2.7-minimal 2024-04-19T00:49:21.804 INFO:teuthology.orchestra.run.smithi118.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-19T00:49:21.805 INFO:teuthology.orchestra.run.smithi118.stdout:The following additional packages will be installed: 2024-04-19T00:49:21.806 INFO:teuthology.orchestra.run.smithi118.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-19T00:49:21.806 INFO:teuthology.orchestra.run.smithi118.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-19T00:49:21.806 INFO:teuthology.orchestra.run.smithi118.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-19T00:49:21.807 INFO:teuthology.orchestra.run.smithi118.stdout:Suggested packages: 2024-04-19T00:49:21.807 INFO:teuthology.orchestra.run.smithi118.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-19T00:49:21.807 INFO:teuthology.orchestra.run.smithi118.stdout:Recommended packages: 2024-04-19T00:49:21.807 INFO:teuthology.orchestra.run.smithi118.stdout: thermald 2024-04-19T00:49:21.849 INFO:teuthology.orchestra.run.smithi118.stdout:The following NEW packages will be installed: 2024-04-19T00:49:21.849 INFO:teuthology.orchestra.run.smithi118.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-19T00:49:21.849 INFO:teuthology.orchestra.run.smithi118.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-19T00:49:21.850 INFO:teuthology.orchestra.run.smithi118.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-19T00:49:21.851 INFO:teuthology.orchestra.run.smithi118.stdout:The following packages will be upgraded: 2024-04-19T00:49:21.851 INFO:teuthology.orchestra.run.smithi118.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-19T00:49:21.975 INFO:teuthology.orchestra.run.smithi110.stdout:The following packages were automatically installed and are no longer required: 2024-04-19T00:49:21.975 INFO:teuthology.orchestra.run.smithi110.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-19T00:49:21.975 INFO:teuthology.orchestra.run.smithi110.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-19T00:49:21.975 INFO:teuthology.orchestra.run.smithi110.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-19T00:49:21.975 INFO:teuthology.orchestra.run.smithi110.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-19T00:49:21.975 INFO:teuthology.orchestra.run.smithi110.stdout: python2.7-minimal 2024-04-19T00:49:21.975 INFO:teuthology.orchestra.run.smithi110.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-19T00:49:21.976 INFO:teuthology.orchestra.run.smithi110.stdout:The following additional packages will be installed: 2024-04-19T00:49:21.976 INFO:teuthology.orchestra.run.smithi110.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-19T00:49:21.976 INFO:teuthology.orchestra.run.smithi110.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-19T00:49:21.976 INFO:teuthology.orchestra.run.smithi110.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-19T00:49:21.976 INFO:teuthology.orchestra.run.smithi110.stdout:Suggested packages: 2024-04-19T00:49:21.977 INFO:teuthology.orchestra.run.smithi110.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-19T00:49:21.977 INFO:teuthology.orchestra.run.smithi110.stdout:Recommended packages: 2024-04-19T00:49:21.977 INFO:teuthology.orchestra.run.smithi110.stdout: thermald 2024-04-19T00:49:22.026 INFO:teuthology.orchestra.run.smithi110.stdout:The following NEW packages will be installed: 2024-04-19T00:49:22.026 INFO:teuthology.orchestra.run.smithi110.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-19T00:49:22.026 INFO:teuthology.orchestra.run.smithi110.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-19T00:49:22.026 INFO:teuthology.orchestra.run.smithi110.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-19T00:49:22.028 INFO:teuthology.orchestra.run.smithi110.stdout:The following packages will be upgraded: 2024-04-19T00:49:22.028 INFO:teuthology.orchestra.run.smithi110.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-19T00:49:22.031 INFO:teuthology.orchestra.run.smithi118.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-19T00:49:22.031 INFO:teuthology.orchestra.run.smithi118.stdout:Need to get 113 MB of archives. 2024-04-19T00:49:22.031 INFO:teuthology.orchestra.run.smithi118.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-19T00:49:22.031 INFO:teuthology.orchestra.run.smithi118.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-19T00:49:22.087 INFO:teuthology.orchestra.run.smithi110.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-19T00:49:22.087 INFO:teuthology.orchestra.run.smithi110.stdout:Need to get 113 MB of archives. 2024-04-19T00:49:22.087 INFO:teuthology.orchestra.run.smithi110.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-19T00:49:22.087 INFO:teuthology.orchestra.run.smithi110.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-19T00:49:22.580 INFO:teuthology.orchestra.run.smithi110.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-19T00:49:22.689 INFO:teuthology.orchestra.run.smithi110.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-19T00:49:22.732 INFO:teuthology.orchestra.run.smithi155.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-19T00:49:23.087 INFO:teuthology.orchestra.run.smithi155.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-19T00:49:23.422 INFO:teuthology.orchestra.run.smithi118.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-19T00:49:23.530 INFO:teuthology.orchestra.run.smithi110.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-19T00:49:23.530 INFO:teuthology.orchestra.run.smithi110.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-19T00:49:23.531 INFO:teuthology.orchestra.run.smithi110.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-19T00:49:23.777 INFO:teuthology.orchestra.run.smithi118.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-19T00:49:23.998 INFO:teuthology.orchestra.run.smithi110.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-19T00:49:24.044 INFO:teuthology.orchestra.run.smithi110.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-19T00:49:24.449 INFO:teuthology.orchestra.run.smithi110.stdout:Fetched 113 MB in 2s (56.4 MB/s) 2024-04-19T00:49:24.616 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-19T00:49:25.362 INFO:teuthology.orchestra.run.smithi155.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-19T00:49:25.362 INFO:teuthology.orchestra.run.smithi155.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-19T00:49:25.363 INFO:teuthology.orchestra.run.smithi155.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-19T00:49:25.796 INFO:teuthology.orchestra.run.smithi155.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-19T00:49:25.878 INFO:teuthology.orchestra.run.smithi155.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-19T00:49:25.961 INFO:teuthology.orchestra.run.smithi118.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-19T00:49:25.961 INFO:teuthology.orchestra.run.smithi118.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-19T00:49:26.041 INFO:teuthology.orchestra.run.smithi118.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-19T00:49:26.266 INFO:teuthology.orchestra.run.smithi155.stdout:Fetched 113 MB in 5s (23.8 MB/s) 2024-04-19T00:49:26.472 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-19T00:49:26.625 INFO:teuthology.orchestra.run.smithi110.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-19T00:49:26.629 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T00:49:26.723 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:49:27.065 INFO:teuthology.orchestra.run.smithi118.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-19T00:49:27.154 INFO:teuthology.orchestra.run.smithi118.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-19T00:49:27.571 INFO:teuthology.orchestra.run.smithi118.stdout:Fetched 113 MB in 5s (21.4 MB/s) 2024-04-19T00:49:27.747 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-19T00:49:28.463 INFO:teuthology.orchestra.run.smithi155.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-19T00:49:28.468 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T00:49:28.629 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:49:28.967 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-19T00:49:28.988 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T00:49:29.070 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:49:29.529 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-19T00:49:29.549 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T00:49:29.578 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:49:29.710 INFO:teuthology.orchestra.run.smithi118.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-19T00:49:29.714 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T00:49:29.778 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:49:30.873 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-19T00:49:30.894 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T00:49:30.995 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:49:31.477 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-19T00:49:31.497 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T00:49:31.693 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:49:31.995 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-19T00:49:32.016 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T00:49:32.102 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:49:32.656 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-19T00:49:32.677 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T00:49:32.875 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:49:36.732 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-19T00:49:36.884 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-19T00:49:37.209 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-19T00:49:37.321 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-19T00:49:37.650 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-19T00:49:37.672 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-19T00:49:37.707 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-19T00:49:38.505 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-19T00:49:38.641 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-19T00:49:39.008 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-19T00:49:39.161 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-19T00:49:39.299 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-19T00:49:39.424 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-19T00:49:39.473 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-19T00:49:39.495 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-19T00:49:39.531 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-19T00:49:39.766 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-19T00:49:39.887 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-19T00:49:40.181 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-19T00:49:40.192 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-19T00:49:40.231 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-19T00:49:43.763 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-19T00:49:43.800 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T00:49:43.837 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:49:45.830 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-19T00:49:45.864 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T00:49:45.905 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:49:46.334 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-19T00:49:46.371 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T00:49:46.408 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:49:46.800 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-19T00:49:46.901 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-19T00:49:47.251 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-19T00:49:47.381 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:49:47.499 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-19T00:49:47.627 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:49:48.972 INFO:teuthology.orchestra.run.smithi110.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-19T00:49:48.972 INFO:teuthology.orchestra.run.smithi110.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-19T00:49:48.972 INFO:teuthology.orchestra.run.smithi110.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-19T00:49:48.972 INFO:teuthology.orchestra.run.smithi110.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-19T00:49:49.122 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:49:49.822 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-19T00:49:49.923 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-19T00:49:50.135 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-19T00:49:50.223 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-19T00:49:50.261 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-19T00:49:50.337 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:49:50.438 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-19T00:49:50.533 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:49:50.541 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:49:50.660 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-19T00:49:50.774 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:49:50.934 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-19T00:49:51.062 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:49:51.854 INFO:teuthology.orchestra.run.smithi118.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-19T00:49:51.854 INFO:teuthology.orchestra.run.smithi118.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-19T00:49:51.854 INFO:teuthology.orchestra.run.smithi118.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-19T00:49:51.854 INFO:teuthology.orchestra.run.smithi118.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-19T00:49:51.984 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-19T00:49:52.004 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:49:52.118 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-19T00:49:52.269 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:49:52.329 INFO:teuthology.orchestra.run.smithi110.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-19T00:49:52.329 INFO:teuthology.orchestra.run.smithi110.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-19T00:49:52.442 INFO:teuthology.orchestra.run.smithi155.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-19T00:49:52.442 INFO:teuthology.orchestra.run.smithi155.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-19T00:49:52.443 INFO:teuthology.orchestra.run.smithi155.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-19T00:49:52.443 INFO:teuthology.orchestra.run.smithi155.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-19T00:49:52.598 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:49:53.365 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:49:54.009 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:49:55.399 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-19T00:49:55.517 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-19T00:49:55.643 INFO:teuthology.orchestra.run.smithi118.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:49:55.720 INFO:teuthology.orchestra.run.smithi118.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-19T00:49:55.720 INFO:teuthology.orchestra.run.smithi118.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-19T00:49:56.210 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-19T00:49:56.353 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-19T00:49:56.479 INFO:teuthology.orchestra.run.smithi155.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:49:56.580 INFO:teuthology.orchestra.run.smithi155.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-19T00:49:56.580 INFO:teuthology.orchestra.run.smithi155.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-19T00:50:02.932 INFO:teuthology.orchestra.run.smithi033.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-19T00:50:03.337 INFO:teuthology.orchestra.run.smithi033.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-19T00:50:04.568 INFO:teuthology.orchestra.run.smithi033.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-19T00:50:04.589 INFO:teuthology.orchestra.run.smithi033.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-19T00:50:04.609 INFO:teuthology.orchestra.run.smithi033.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-19T00:50:04.784 INFO:teuthology.orchestra.run.smithi033.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-19T00:50:04.840 INFO:teuthology.orchestra.run.smithi033.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-19T00:50:05.261 INFO:teuthology.orchestra.run.smithi033.stdout:Fetched 113 MB in 44s (2,594 kB/s) 2024-04-19T00:50:05.457 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-19T00:50:06.754 INFO:teuthology.orchestra.run.smithi110.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-19T00:50:06.754 INFO:teuthology.orchestra.run.smithi110.stdout:Sourcing file `/etc/default/grub' 2024-04-19T00:50:06.767 INFO:teuthology.orchestra.run.smithi110.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-19T00:50:06.789 INFO:teuthology.orchestra.run.smithi110.stdout:Generating grub configuration file ... 2024-04-19T00:50:07.141 INFO:teuthology.orchestra.run.smithi110.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-19T00:50:07.162 INFO:teuthology.orchestra.run.smithi110.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-19T00:50:07.473 INFO:teuthology.orchestra.run.smithi033.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-19T00:50:07.478 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T00:50:07.564 INFO:teuthology.orchestra.run.smithi110.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-19T00:50:07.569 INFO:teuthology.orchestra.run.smithi110.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-19T00:50:07.605 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:50:07.644 INFO:teuthology.orchestra.run.smithi110.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-19T00:50:07.649 INFO:teuthology.orchestra.run.smithi110.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-19T00:50:07.847 INFO:teuthology.orchestra.run.smithi110.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-19T00:50:07.847 INFO:teuthology.orchestra.run.smithi110.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-19T00:50:07.847 INFO:teuthology.orchestra.run.smithi110.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-19T00:50:07.896 INFO:teuthology.orchestra.run.smithi110.stdout:done 2024-04-19T00:50:08.380 DEBUG:teuthology.orchestra.run.smithi110:> dpkg -s linux-image-generic 2024-04-19T00:50:08.405 INFO:teuthology.orchestra.run.smithi110.stdout:Package: linux-image-generic 2024-04-19T00:50:08.405 INFO:teuthology.orchestra.run.smithi110.stdout:Status: install ok installed 2024-04-19T00:50:08.405 INFO:teuthology.orchestra.run.smithi110.stdout:Priority: optional 2024-04-19T00:50:08.405 INFO:teuthology.orchestra.run.smithi110.stdout:Section: kernel 2024-04-19T00:50:08.405 INFO:teuthology.orchestra.run.smithi110.stdout:Installed-Size: 21 2024-04-19T00:50:08.405 INFO:teuthology.orchestra.run.smithi110.stdout:Maintainer: Ubuntu Kernel Team 2024-04-19T00:50:08.406 INFO:teuthology.orchestra.run.smithi110.stdout:Architecture: amd64 2024-04-19T00:50:08.406 INFO:teuthology.orchestra.run.smithi110.stdout:Source: linux-meta 2024-04-19T00:50:08.406 INFO:teuthology.orchestra.run.smithi110.stdout:Version: 5.15.0.105.102 2024-04-19T00:50:08.406 INFO:teuthology.orchestra.run.smithi110.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-19T00:50:08.406 INFO:teuthology.orchestra.run.smithi110.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-19T00:50:08.406 INFO:teuthology.orchestra.run.smithi110.stdout:Recommends: thermald 2024-04-19T00:50:08.406 INFO:teuthology.orchestra.run.smithi110.stdout:Description: Generic Linux kernel image 2024-04-19T00:50:08.406 INFO:teuthology.orchestra.run.smithi110.stdout: This package will always depend on the latest generic kernel image 2024-04-19T00:50:08.406 INFO:teuthology.orchestra.run.smithi110.stdout: available. 2024-04-19T00:50:08.406 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-19T00:50:08.407 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-19T00:50:08.407 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-19T00:50:08.407 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-19T00:50:08.407 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi110.front.sepia.ceph.com, path=None, version=distro) 2024-04-19T00:50:08.407 DEBUG:teuthology.orchestra.run.smithi110:> sudo apt-get clean 2024-04-19T00:50:08.602 DEBUG:teuthology.orchestra.run.smithi110:> sudo apt-get update 2024-04-19T00:50:08.721 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-19T00:50:08.721 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-19T00:50:08.734 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-19T00:50:08.755 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-19T00:50:09.833 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-19T00:50:09.845 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T00:50:09.937 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:50:09.986 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2024-04-19T00:50:10.004 DEBUG:teuthology.orchestra.run.smithi110:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-19T00:50:10.056 INFO:teuthology.orchestra.run.smithi118.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-19T00:50:10.056 INFO:teuthology.orchestra.run.smithi118.stdout:Sourcing file `/etc/default/grub' 2024-04-19T00:50:10.063 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2024-04-19T00:50:10.065 INFO:teuthology.orchestra.run.smithi118.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-19T00:50:10.084 INFO:teuthology.orchestra.run.smithi118.stdout:Generating grub configuration file ... 2024-04-19T00:50:10.263 INFO:teuthology.orchestra.run.smithi110.stdout:Building dependency tree... 2024-04-19T00:50:10.264 INFO:teuthology.orchestra.run.smithi110.stdout:Reading state information... 2024-04-19T00:50:10.367 INFO:teuthology.orchestra.run.smithi118.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-19T00:50:10.382 INFO:teuthology.orchestra.run.smithi118.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-19T00:50:10.442 INFO:teuthology.orchestra.run.smithi110.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-19T00:50:10.442 INFO:teuthology.orchestra.run.smithi110.stdout:The following packages were automatically installed and are no longer required: 2024-04-19T00:50:10.442 INFO:teuthology.orchestra.run.smithi110.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-19T00:50:10.443 INFO:teuthology.orchestra.run.smithi110.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-19T00:50:10.443 INFO:teuthology.orchestra.run.smithi110.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-19T00:50:10.443 INFO:teuthology.orchestra.run.smithi110.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-19T00:50:10.443 INFO:teuthology.orchestra.run.smithi110.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-19T00:50:10.443 INFO:teuthology.orchestra.run.smithi110.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-19T00:50:10.486 INFO:teuthology.orchestra.run.smithi110.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-19T00:50:10.488 DEBUG:teuthology.orchestra.run.smithi110:> dpkg -s linux-image-generic 2024-04-19T00:50:10.503 INFO:teuthology.orchestra.run.smithi110.stdout:Package: linux-image-generic 2024-04-19T00:50:10.503 INFO:teuthology.orchestra.run.smithi110.stdout:Status: install ok installed 2024-04-19T00:50:10.503 INFO:teuthology.orchestra.run.smithi110.stdout:Priority: optional 2024-04-19T00:50:10.504 INFO:teuthology.orchestra.run.smithi110.stdout:Section: kernel 2024-04-19T00:50:10.504 INFO:teuthology.orchestra.run.smithi110.stdout:Installed-Size: 21 2024-04-19T00:50:10.504 INFO:teuthology.orchestra.run.smithi110.stdout:Maintainer: Ubuntu Kernel Team 2024-04-19T00:50:10.504 INFO:teuthology.orchestra.run.smithi110.stdout:Architecture: amd64 2024-04-19T00:50:10.504 INFO:teuthology.orchestra.run.smithi110.stdout:Source: linux-meta 2024-04-19T00:50:10.504 INFO:teuthology.orchestra.run.smithi110.stdout:Version: 5.15.0.105.102 2024-04-19T00:50:10.504 INFO:teuthology.orchestra.run.smithi110.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-19T00:50:10.504 INFO:teuthology.orchestra.run.smithi110.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-19T00:50:10.504 INFO:teuthology.orchestra.run.smithi110.stdout:Recommends: thermald 2024-04-19T00:50:10.504 INFO:teuthology.orchestra.run.smithi110.stdout:Description: Generic Linux kernel image 2024-04-19T00:50:10.504 INFO:teuthology.orchestra.run.smithi110.stdout: This package will always depend on the latest generic kernel image 2024-04-19T00:50:10.504 INFO:teuthology.orchestra.run.smithi110.stdout: available. 2024-04-19T00:50:10.505 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2024-04-19T00:50:10.549 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.i4IT29hMIE 2024-04-19T00:50:10.549 DEBUG:teuthology.orchestra.run.smithi110:> sudo cp /boot/grub/grub.cfg /tmp/tmp.i4IT29hMIE 2024-04-19T00:50:10.604 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.i4IT29hMIE 2024-04-19T00:50:10.663 INFO:teuthology.orchestra.run.smithi118.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-19T00:50:10.668 INFO:teuthology.orchestra.run.smithi118.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-19T00:50:10.727 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.i4IT29hMIE is 10KB 2024-04-19T00:50:10.736 INFO:teuthology.orchestra.run.smithi118.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-19T00:50:10.740 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.i4IT29hMIE 2024-04-19T00:50:10.741 INFO:teuthology.orchestra.run.smithi118.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-19T00:50:10.746 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-19T00:50:10.802 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-19T00:50:10.802 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-19T00:50:10.802 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-19T00:50:10.853 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-19T00:50:10.874 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T00:50:10.881 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-19T00:50:10.881 DEBUG:teuthology.orchestra.run.smithi110:> sudo update-grub 2024-04-19T00:50:10.898 INFO:teuthology.orchestra.run.smithi118.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-19T00:50:10.898 INFO:teuthology.orchestra.run.smithi118.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-19T00:50:10.898 INFO:teuthology.orchestra.run.smithi118.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-19T00:50:10.911 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:50:10.913 INFO:teuthology.orchestra.run.smithi155.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-19T00:50:10.913 INFO:teuthology.orchestra.run.smithi155.stdout:Sourcing file `/etc/default/grub' 2024-04-19T00:50:10.941 INFO:teuthology.orchestra.run.smithi118.stdout:done 2024-04-19T00:50:10.943 INFO:teuthology.orchestra.run.smithi155.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-19T00:50:10.963 INFO:teuthology.orchestra.run.smithi155.stdout:Generating grub configuration file ... 2024-04-19T00:50:11.273 INFO:teuthology.orchestra.run.smithi155.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-19T00:50:11.301 INFO:teuthology.orchestra.run.smithi155.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-19T00:50:11.399 DEBUG:teuthology.orchestra.run.smithi118:> dpkg -s linux-image-generic 2024-04-19T00:50:11.424 INFO:teuthology.orchestra.run.smithi118.stdout:Package: linux-image-generic 2024-04-19T00:50:11.424 INFO:teuthology.orchestra.run.smithi118.stdout:Status: install ok installed 2024-04-19T00:50:11.424 INFO:teuthology.orchestra.run.smithi118.stdout:Priority: optional 2024-04-19T00:50:11.424 INFO:teuthology.orchestra.run.smithi118.stdout:Section: kernel 2024-04-19T00:50:11.424 INFO:teuthology.orchestra.run.smithi118.stdout:Installed-Size: 21 2024-04-19T00:50:11.425 INFO:teuthology.orchestra.run.smithi118.stdout:Maintainer: Ubuntu Kernel Team 2024-04-19T00:50:11.425 INFO:teuthology.orchestra.run.smithi118.stdout:Architecture: amd64 2024-04-19T00:50:11.425 INFO:teuthology.orchestra.run.smithi118.stdout:Source: linux-meta 2024-04-19T00:50:11.425 INFO:teuthology.orchestra.run.smithi118.stdout:Version: 5.15.0.105.102 2024-04-19T00:50:11.425 INFO:teuthology.orchestra.run.smithi118.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-19T00:50:11.425 INFO:teuthology.orchestra.run.smithi118.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-19T00:50:11.425 INFO:teuthology.orchestra.run.smithi118.stdout:Recommends: thermald 2024-04-19T00:50:11.425 INFO:teuthology.orchestra.run.smithi118.stdout:Description: Generic Linux kernel image 2024-04-19T00:50:11.425 INFO:teuthology.orchestra.run.smithi118.stdout: This package will always depend on the latest generic kernel image 2024-04-19T00:50:11.425 INFO:teuthology.orchestra.run.smithi118.stdout: available. 2024-04-19T00:50:11.425 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-19T00:50:11.425 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-19T00:50:11.425 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-19T00:50:11.426 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-04-19T00:50:11.426 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi118.front.sepia.ceph.com, path=None, version=distro) 2024-04-19T00:50:11.426 DEBUG:teuthology.orchestra.run.smithi118:> sudo apt-get clean 2024-04-19T00:50:11.593 INFO:teuthology.orchestra.run.smithi155.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-19T00:50:11.598 INFO:teuthology.orchestra.run.smithi155.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-19T00:50:11.657 DEBUG:teuthology.orchestra.run.smithi118:> sudo apt-get update 2024-04-19T00:50:11.667 INFO:teuthology.orchestra.run.smithi155.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-19T00:50:11.672 INFO:teuthology.orchestra.run.smithi155.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-19T00:50:11.798 INFO:teuthology.orchestra.run.smithi118.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-19T00:50:11.799 INFO:teuthology.orchestra.run.smithi118.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-19T00:50:11.813 INFO:teuthology.orchestra.run.smithi118.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-19T00:50:11.831 INFO:teuthology.orchestra.run.smithi155.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-19T00:50:11.831 INFO:teuthology.orchestra.run.smithi155.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-19T00:50:11.831 INFO:teuthology.orchestra.run.smithi155.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-19T00:50:11.833 INFO:teuthology.orchestra.run.smithi118.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-19T00:50:11.873 INFO:teuthology.orchestra.run.smithi155.stdout:done 2024-04-19T00:50:12.148 INFO:teuthology.orchestra.run.smithi110.stderr:Sourcing file `/etc/default/grub' 2024-04-19T00:50:12.149 INFO:teuthology.orchestra.run.smithi110.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-19T00:50:12.151 INFO:teuthology.orchestra.run.smithi110.stderr:Generating grub configuration file ... 2024-04-19T00:50:12.375 DEBUG:teuthology.orchestra.run.smithi155:> dpkg -s linux-image-generic 2024-04-19T00:50:12.396 INFO:teuthology.orchestra.run.smithi155.stdout:Package: linux-image-generic 2024-04-19T00:50:12.396 INFO:teuthology.orchestra.run.smithi155.stdout:Status: install ok installed 2024-04-19T00:50:12.397 INFO:teuthology.orchestra.run.smithi155.stdout:Priority: optional 2024-04-19T00:50:12.397 INFO:teuthology.orchestra.run.smithi155.stdout:Section: kernel 2024-04-19T00:50:12.397 INFO:teuthology.orchestra.run.smithi155.stdout:Installed-Size: 21 2024-04-19T00:50:12.397 INFO:teuthology.orchestra.run.smithi155.stdout:Maintainer: Ubuntu Kernel Team 2024-04-19T00:50:12.397 INFO:teuthology.orchestra.run.smithi155.stdout:Architecture: amd64 2024-04-19T00:50:12.397 INFO:teuthology.orchestra.run.smithi155.stdout:Source: linux-meta 2024-04-19T00:50:12.397 INFO:teuthology.orchestra.run.smithi155.stdout:Version: 5.15.0.105.102 2024-04-19T00:50:12.397 INFO:teuthology.orchestra.run.smithi155.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-19T00:50:12.397 INFO:teuthology.orchestra.run.smithi155.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-19T00:50:12.397 INFO:teuthology.orchestra.run.smithi155.stdout:Recommends: thermald 2024-04-19T00:50:12.397 INFO:teuthology.orchestra.run.smithi155.stdout:Description: Generic Linux kernel image 2024-04-19T00:50:12.397 INFO:teuthology.orchestra.run.smithi155.stdout: This package will always depend on the latest generic kernel image 2024-04-19T00:50:12.397 INFO:teuthology.orchestra.run.smithi155.stdout: available. 2024-04-19T00:50:12.398 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-19T00:50:12.398 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-19T00:50:12.398 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-19T00:50:12.398 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2024-04-19T00:50:12.398 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi155.front.sepia.ceph.com, path=None, version=distro) 2024-04-19T00:50:12.398 DEBUG:teuthology.orchestra.run.smithi155:> sudo apt-get clean 2024-04-19T00:50:12.461 INFO:teuthology.orchestra.run.smithi110.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-19T00:50:12.468 INFO:teuthology.orchestra.run.smithi110.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-19T00:50:12.572 DEBUG:teuthology.orchestra.run.smithi155:> sudo apt-get update 2024-04-19T00:50:12.707 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-19T00:50:12.707 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-19T00:50:12.722 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-19T00:50:12.746 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-19T00:50:12.752 INFO:teuthology.orchestra.run.smithi110.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-19T00:50:12.757 INFO:teuthology.orchestra.run.smithi110.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-19T00:50:12.801 INFO:teuthology.orchestra.run.smithi110.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-19T00:50:12.806 INFO:teuthology.orchestra.run.smithi110.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-19T00:50:12.927 INFO:teuthology.orchestra.run.smithi110.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-19T00:50:12.927 INFO:teuthology.orchestra.run.smithi110.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-19T00:50:12.927 INFO:teuthology.orchestra.run.smithi110.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-19T00:50:12.944 INFO:teuthology.orchestra.run.smithi110.stderr:done 2024-04-19T00:50:12.946 DEBUG:teuthology.orchestra.run.smithi110:> sudo shutdown -r now 2024-04-19T00:50:13.213 INFO:teuthology.orchestra.run.smithi118.stdout:Reading package lists... 2024-04-19T00:50:13.231 DEBUG:teuthology.orchestra.run.smithi118:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-19T00:50:13.282 INFO:teuthology.orchestra.run.smithi118.stdout:Reading package lists... 2024-04-19T00:50:13.461 INFO:teuthology.orchestra.run.smithi118.stdout:Building dependency tree... 2024-04-19T00:50:13.461 INFO:teuthology.orchestra.run.smithi118.stdout:Reading state information... 2024-04-19T00:50:13.620 INFO:teuthology.orchestra.run.smithi118.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-19T00:50:13.620 INFO:teuthology.orchestra.run.smithi118.stdout:The following packages were automatically installed and are no longer required: 2024-04-19T00:50:13.620 INFO:teuthology.orchestra.run.smithi118.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-19T00:50:13.620 INFO:teuthology.orchestra.run.smithi118.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-19T00:50:13.620 INFO:teuthology.orchestra.run.smithi118.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-19T00:50:13.620 INFO:teuthology.orchestra.run.smithi118.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-19T00:50:13.621 INFO:teuthology.orchestra.run.smithi118.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-19T00:50:13.621 INFO:teuthology.orchestra.run.smithi118.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-19T00:50:13.659 INFO:teuthology.orchestra.run.smithi118.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-19T00:50:13.661 DEBUG:teuthology.orchestra.run.smithi118:> dpkg -s linux-image-generic 2024-04-19T00:50:13.720 INFO:teuthology.orchestra.run.smithi118.stdout:Package: linux-image-generic 2024-04-19T00:50:13.720 INFO:teuthology.orchestra.run.smithi118.stdout:Status: install ok installed 2024-04-19T00:50:13.720 INFO:teuthology.orchestra.run.smithi118.stdout:Priority: optional 2024-04-19T00:50:13.720 INFO:teuthology.orchestra.run.smithi118.stdout:Section: kernel 2024-04-19T00:50:13.720 INFO:teuthology.orchestra.run.smithi118.stdout:Installed-Size: 21 2024-04-19T00:50:13.720 INFO:teuthology.orchestra.run.smithi118.stdout:Maintainer: Ubuntu Kernel Team 2024-04-19T00:50:13.720 INFO:teuthology.orchestra.run.smithi118.stdout:Architecture: amd64 2024-04-19T00:50:13.720 INFO:teuthology.orchestra.run.smithi118.stdout:Source: linux-meta 2024-04-19T00:50:13.720 INFO:teuthology.orchestra.run.smithi118.stdout:Version: 5.15.0.105.102 2024-04-19T00:50:13.721 INFO:teuthology.orchestra.run.smithi118.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-19T00:50:13.721 INFO:teuthology.orchestra.run.smithi118.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-19T00:50:13.721 INFO:teuthology.orchestra.run.smithi118.stdout:Recommends: thermald 2024-04-19T00:50:13.721 INFO:teuthology.orchestra.run.smithi118.stdout:Description: Generic Linux kernel image 2024-04-19T00:50:13.721 INFO:teuthology.orchestra.run.smithi118.stdout: This package will always depend on the latest generic kernel image 2024-04-19T00:50:13.721 INFO:teuthology.orchestra.run.smithi118.stdout: available. 2024-04-19T00:50:13.721 DEBUG:teuthology.orchestra.run.smithi118:> mktemp 2024-04-19T00:50:13.768 INFO:teuthology.orchestra.run.smithi118.stdout:/tmp/tmp.2fQAybm7Qw 2024-04-19T00:50:13.769 DEBUG:teuthology.orchestra.run.smithi118:> sudo cp /boot/grub/grub.cfg /tmp/tmp.2fQAybm7Qw 2024-04-19T00:50:13.825 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0666 /tmp/tmp.2fQAybm7Qw 2024-04-19T00:50:13.935 DEBUG:teuthology.orchestra.remote:smithi118:/tmp/tmp.2fQAybm7Qw is 10KB 2024-04-19T00:50:13.947 DEBUG:teuthology.orchestra.run.smithi118:> rm -fr /tmp/tmp.2fQAybm7Qw 2024-04-19T00:50:13.953 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-19T00:50:14.009 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-04-19T00:50:14.009 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-19T00:50:14.009 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-19T00:50:14.075 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-19T00:50:14.076 DEBUG:teuthology.orchestra.run.smithi118:> sudo update-grub 2024-04-19T00:50:14.088 INFO:teuthology.orchestra.run.smithi155.stdout:Reading package lists... 2024-04-19T00:50:14.106 DEBUG:teuthology.orchestra.run.smithi155:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-19T00:50:14.169 INFO:teuthology.orchestra.run.smithi155.stdout:Reading package lists... 2024-04-19T00:50:14.374 INFO:teuthology.orchestra.run.smithi155.stdout:Building dependency tree... 2024-04-19T00:50:14.375 INFO:teuthology.orchestra.run.smithi155.stdout:Reading state information... 2024-04-19T00:50:14.553 INFO:teuthology.orchestra.run.smithi155.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-19T00:50:14.553 INFO:teuthology.orchestra.run.smithi155.stdout:The following packages were automatically installed and are no longer required: 2024-04-19T00:50:14.553 INFO:teuthology.orchestra.run.smithi155.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-19T00:50:14.554 INFO:teuthology.orchestra.run.smithi155.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-19T00:50:14.554 INFO:teuthology.orchestra.run.smithi155.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-19T00:50:14.554 INFO:teuthology.orchestra.run.smithi155.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-19T00:50:14.554 INFO:teuthology.orchestra.run.smithi155.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-19T00:50:14.554 INFO:teuthology.orchestra.run.smithi155.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-19T00:50:14.593 INFO:teuthology.orchestra.run.smithi155.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-19T00:50:14.595 DEBUG:teuthology.orchestra.run.smithi155:> dpkg -s linux-image-generic 2024-04-19T00:50:14.611 INFO:teuthology.orchestra.run.smithi155.stdout:Package: linux-image-generic 2024-04-19T00:50:14.611 INFO:teuthology.orchestra.run.smithi155.stdout:Status: install ok installed 2024-04-19T00:50:14.611 INFO:teuthology.orchestra.run.smithi155.stdout:Priority: optional 2024-04-19T00:50:14.611 INFO:teuthology.orchestra.run.smithi155.stdout:Section: kernel 2024-04-19T00:50:14.611 INFO:teuthology.orchestra.run.smithi155.stdout:Installed-Size: 21 2024-04-19T00:50:14.611 INFO:teuthology.orchestra.run.smithi155.stdout:Maintainer: Ubuntu Kernel Team 2024-04-19T00:50:14.611 INFO:teuthology.orchestra.run.smithi155.stdout:Architecture: amd64 2024-04-19T00:50:14.611 INFO:teuthology.orchestra.run.smithi155.stdout:Source: linux-meta 2024-04-19T00:50:14.611 INFO:teuthology.orchestra.run.smithi155.stdout:Version: 5.15.0.105.102 2024-04-19T00:50:14.611 INFO:teuthology.orchestra.run.smithi155.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-19T00:50:14.611 INFO:teuthology.orchestra.run.smithi155.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-19T00:50:14.612 INFO:teuthology.orchestra.run.smithi155.stdout:Recommends: thermald 2024-04-19T00:50:14.612 INFO:teuthology.orchestra.run.smithi155.stdout:Description: Generic Linux kernel image 2024-04-19T00:50:14.612 INFO:teuthology.orchestra.run.smithi155.stdout: This package will always depend on the latest generic kernel image 2024-04-19T00:50:14.612 INFO:teuthology.orchestra.run.smithi155.stdout: available. 2024-04-19T00:50:14.612 DEBUG:teuthology.orchestra.run.smithi155:> mktemp 2024-04-19T00:50:14.655 INFO:teuthology.orchestra.run.smithi155.stdout:/tmp/tmp.rb95fhIOvb 2024-04-19T00:50:14.656 DEBUG:teuthology.orchestra.run.smithi155:> sudo cp /boot/grub/grub.cfg /tmp/tmp.rb95fhIOvb 2024-04-19T00:50:14.711 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0666 /tmp/tmp.rb95fhIOvb 2024-04-19T00:50:14.839 DEBUG:teuthology.orchestra.remote:smithi155:/tmp/tmp.rb95fhIOvb is 10KB 2024-04-19T00:50:14.852 DEBUG:teuthology.orchestra.run.smithi155:> rm -fr /tmp/tmp.rb95fhIOvb 2024-04-19T00:50:14.857 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-19T00:50:14.911 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-19T00:50:14.911 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-19T00:50:14.912 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-19T00:50:14.995 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-19T00:50:14.995 DEBUG:teuthology.orchestra.run.smithi155:> sudo update-grub 2024-04-19T00:50:15.228 INFO:teuthology.orchestra.run.smithi118.stderr:Sourcing file `/etc/default/grub' 2024-04-19T00:50:15.229 INFO:teuthology.orchestra.run.smithi118.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-19T00:50:15.232 INFO:teuthology.orchestra.run.smithi118.stderr:Generating grub configuration file ... 2024-04-19T00:50:15.408 INFO:teuthology.orchestra.run.smithi118.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-19T00:50:15.414 INFO:teuthology.orchestra.run.smithi118.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-19T00:50:15.631 INFO:teuthology.orchestra.run.smithi118.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-19T00:50:15.636 INFO:teuthology.orchestra.run.smithi118.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-19T00:50:15.673 INFO:teuthology.orchestra.run.smithi118.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-19T00:50:15.678 INFO:teuthology.orchestra.run.smithi118.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-19T00:50:15.814 INFO:teuthology.orchestra.run.smithi118.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-19T00:50:15.814 INFO:teuthology.orchestra.run.smithi118.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-19T00:50:15.814 INFO:teuthology.orchestra.run.smithi118.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-19T00:50:15.825 INFO:teuthology.orchestra.run.smithi118.stderr:done 2024-04-19T00:50:15.839 DEBUG:teuthology.orchestra.run.smithi118:> sudo shutdown -r now 2024-04-19T00:50:16.327 INFO:teuthology.orchestra.run.smithi155.stderr:Sourcing file `/etc/default/grub' 2024-04-19T00:50:16.328 INFO:teuthology.orchestra.run.smithi155.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-19T00:50:16.332 INFO:teuthology.orchestra.run.smithi155.stderr:Generating grub configuration file ... 2024-04-19T00:50:16.554 INFO:teuthology.orchestra.run.smithi155.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-19T00:50:16.560 INFO:teuthology.orchestra.run.smithi155.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-19T00:50:16.826 INFO:teuthology.orchestra.run.smithi155.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-19T00:50:16.831 INFO:teuthology.orchestra.run.smithi155.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-19T00:50:16.875 INFO:teuthology.orchestra.run.smithi155.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-19T00:50:16.880 INFO:teuthology.orchestra.run.smithi155.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-19T00:50:17.031 INFO:teuthology.orchestra.run.smithi155.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-19T00:50:17.031 INFO:teuthology.orchestra.run.smithi155.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-19T00:50:17.031 INFO:teuthology.orchestra.run.smithi155.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-19T00:50:17.041 INFO:teuthology.orchestra.run.smithi155.stderr:done 2024-04-19T00:50:17.043 DEBUG:teuthology.orchestra.run.smithi155:> sudo shutdown -r now 2024-04-19T00:50:17.540 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-19T00:50:17.666 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-19T00:50:18.026 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-19T00:50:18.171 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-19T00:50:18.483 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-19T00:50:18.494 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-19T00:50:18.549 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-19T00:50:24.587 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-19T00:50:24.624 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T00:50:24.662 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:50:27.523 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-19T00:50:27.647 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-19T00:50:28.014 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-19T00:50:28.136 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:50:28.287 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-19T00:50:28.415 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:50:29.779 INFO:teuthology.orchestra.run.smithi033.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-19T00:50:29.780 INFO:teuthology.orchestra.run.smithi033.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-19T00:50:29.780 INFO:teuthology.orchestra.run.smithi033.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-19T00:50:29.780 INFO:teuthology.orchestra.run.smithi033.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-19T00:50:29.958 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:50:31.402 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:50:32.828 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-19T00:50:32.962 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-19T00:50:33.088 INFO:teuthology.orchestra.run.smithi033.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:50:33.162 INFO:teuthology.orchestra.run.smithi033.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-19T00:50:33.162 INFO:teuthology.orchestra.run.smithi033.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-19T00:50:42.948 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-19T00:50:42.949 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-04-19T00:50:42.950 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T00:50:45.842 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-04-19T00:50:45.842 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-04-19T00:50:45.843 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T00:50:47.046 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2024-04-19T00:50:47.046 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-04-19T00:50:47.047 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T00:50:48.092 INFO:teuthology.orchestra.run.smithi033.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-19T00:50:48.092 INFO:teuthology.orchestra.run.smithi033.stdout:Sourcing file `/etc/default/grub' 2024-04-19T00:50:48.120 INFO:teuthology.orchestra.run.smithi033.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-19T00:50:48.146 INFO:teuthology.orchestra.run.smithi033.stdout:Generating grub configuration file ... 2024-04-19T00:50:48.429 INFO:teuthology.orchestra.run.smithi033.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-19T00:50:48.456 INFO:teuthology.orchestra.run.smithi033.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-19T00:50:48.742 INFO:teuthology.orchestra.run.smithi033.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-19T00:50:48.748 INFO:teuthology.orchestra.run.smithi033.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-19T00:50:48.822 INFO:teuthology.orchestra.run.smithi033.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-19T00:50:48.830 INFO:teuthology.orchestra.run.smithi033.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-19T00:50:49.007 INFO:teuthology.orchestra.run.smithi033.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-19T00:50:49.007 INFO:teuthology.orchestra.run.smithi033.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-19T00:50:49.007 INFO:teuthology.orchestra.run.smithi033.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-19T00:50:49.051 INFO:teuthology.orchestra.run.smithi033.stdout:done 2024-04-19T00:50:49.562 DEBUG:teuthology.orchestra.run.smithi033:> dpkg -s linux-image-generic 2024-04-19T00:50:49.589 INFO:teuthology.orchestra.run.smithi033.stdout:Package: linux-image-generic 2024-04-19T00:50:49.589 INFO:teuthology.orchestra.run.smithi033.stdout:Status: install ok installed 2024-04-19T00:50:49.589 INFO:teuthology.orchestra.run.smithi033.stdout:Priority: optional 2024-04-19T00:50:49.589 INFO:teuthology.orchestra.run.smithi033.stdout:Section: kernel 2024-04-19T00:50:49.589 INFO:teuthology.orchestra.run.smithi033.stdout:Installed-Size: 21 2024-04-19T00:50:49.589 INFO:teuthology.orchestra.run.smithi033.stdout:Maintainer: Ubuntu Kernel Team 2024-04-19T00:50:49.589 INFO:teuthology.orchestra.run.smithi033.stdout:Architecture: amd64 2024-04-19T00:50:49.590 INFO:teuthology.orchestra.run.smithi033.stdout:Source: linux-meta 2024-04-19T00:50:49.590 INFO:teuthology.orchestra.run.smithi033.stdout:Version: 5.15.0.105.102 2024-04-19T00:50:49.590 INFO:teuthology.orchestra.run.smithi033.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-19T00:50:49.590 INFO:teuthology.orchestra.run.smithi033.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-19T00:50:49.590 INFO:teuthology.orchestra.run.smithi033.stdout:Recommends: thermald 2024-04-19T00:50:49.590 INFO:teuthology.orchestra.run.smithi033.stdout:Description: Generic Linux kernel image 2024-04-19T00:50:49.590 INFO:teuthology.orchestra.run.smithi033.stdout: This package will always depend on the latest generic kernel image 2024-04-19T00:50:49.590 INFO:teuthology.orchestra.run.smithi033.stdout: available. 2024-04-19T00:50:49.590 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-19T00:50:49.590 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-19T00:50:49.591 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-19T00:50:49.591 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-19T00:50:49.591 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi033.front.sepia.ceph.com, path=None, version=distro) 2024-04-19T00:50:49.591 DEBUG:teuthology.orchestra.run.smithi033:> sudo apt-get clean 2024-04-19T00:50:49.751 DEBUG:teuthology.orchestra.run.smithi033:> sudo apt-get update 2024-04-19T00:50:50.027 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-19T00:50:50.036 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-19T00:50:50.089 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-19T00:50:50.170 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-19T00:50:51.469 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2024-04-19T00:50:51.487 DEBUG:teuthology.orchestra.run.smithi033:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-19T00:50:51.553 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2024-04-19T00:50:51.765 INFO:teuthology.orchestra.run.smithi033.stdout:Building dependency tree... 2024-04-19T00:50:51.765 INFO:teuthology.orchestra.run.smithi033.stdout:Reading state information... 2024-04-19T00:50:51.953 INFO:teuthology.orchestra.run.smithi033.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-19T00:50:51.953 INFO:teuthology.orchestra.run.smithi033.stdout:The following packages were automatically installed and are no longer required: 2024-04-19T00:50:51.953 INFO:teuthology.orchestra.run.smithi033.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-19T00:50:51.953 INFO:teuthology.orchestra.run.smithi033.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-19T00:50:51.953 INFO:teuthology.orchestra.run.smithi033.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-19T00:50:51.954 INFO:teuthology.orchestra.run.smithi033.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-19T00:50:51.954 INFO:teuthology.orchestra.run.smithi033.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-19T00:50:51.954 INFO:teuthology.orchestra.run.smithi033.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-19T00:50:51.996 INFO:teuthology.orchestra.run.smithi033.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-19T00:50:51.998 DEBUG:teuthology.orchestra.run.smithi033:> dpkg -s linux-image-generic 2024-04-19T00:50:52.014 INFO:teuthology.orchestra.run.smithi033.stdout:Package: linux-image-generic 2024-04-19T00:50:52.014 INFO:teuthology.orchestra.run.smithi033.stdout:Status: install ok installed 2024-04-19T00:50:52.014 INFO:teuthology.orchestra.run.smithi033.stdout:Priority: optional 2024-04-19T00:50:52.014 INFO:teuthology.orchestra.run.smithi033.stdout:Section: kernel 2024-04-19T00:50:52.014 INFO:teuthology.orchestra.run.smithi033.stdout:Installed-Size: 21 2024-04-19T00:50:52.015 INFO:teuthology.orchestra.run.smithi033.stdout:Maintainer: Ubuntu Kernel Team 2024-04-19T00:50:52.015 INFO:teuthology.orchestra.run.smithi033.stdout:Architecture: amd64 2024-04-19T00:50:52.015 INFO:teuthology.orchestra.run.smithi033.stdout:Source: linux-meta 2024-04-19T00:50:52.015 INFO:teuthology.orchestra.run.smithi033.stdout:Version: 5.15.0.105.102 2024-04-19T00:50:52.015 INFO:teuthology.orchestra.run.smithi033.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-19T00:50:52.015 INFO:teuthology.orchestra.run.smithi033.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-19T00:50:52.015 INFO:teuthology.orchestra.run.smithi033.stdout:Recommends: thermald 2024-04-19T00:50:52.015 INFO:teuthology.orchestra.run.smithi033.stdout:Description: Generic Linux kernel image 2024-04-19T00:50:52.015 INFO:teuthology.orchestra.run.smithi033.stdout: This package will always depend on the latest generic kernel image 2024-04-19T00:50:52.015 INFO:teuthology.orchestra.run.smithi033.stdout: available. 2024-04-19T00:50:52.015 DEBUG:teuthology.orchestra.run.smithi033:> mktemp 2024-04-19T00:50:52.059 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/tmp.dk7yPadzeT 2024-04-19T00:50:52.059 DEBUG:teuthology.orchestra.run.smithi033:> sudo cp /boot/grub/grub.cfg /tmp/tmp.dk7yPadzeT 2024-04-19T00:50:52.115 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0666 /tmp/tmp.dk7yPadzeT 2024-04-19T00:50:52.240 DEBUG:teuthology.orchestra.remote:smithi033:/tmp/tmp.dk7yPadzeT is 10KB 2024-04-19T00:50:52.254 DEBUG:teuthology.orchestra.run.smithi033:> rm -fr /tmp/tmp.dk7yPadzeT 2024-04-19T00:50:52.260 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-19T00:50:52.315 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-19T00:50:52.316 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-19T00:50:52.316 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-19T00:50:52.392 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-19T00:50:52.392 DEBUG:teuthology.orchestra.run.smithi033:> sudo update-grub 2024-04-19T00:50:53.715 INFO:teuthology.orchestra.run.smithi033.stderr:Sourcing file `/etc/default/grub' 2024-04-19T00:50:53.716 INFO:teuthology.orchestra.run.smithi033.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-19T00:50:53.719 INFO:teuthology.orchestra.run.smithi033.stderr:Generating grub configuration file ... 2024-04-19T00:50:53.916 INFO:teuthology.orchestra.run.smithi033.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-19T00:50:53.922 INFO:teuthology.orchestra.run.smithi033.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-19T00:50:54.210 INFO:teuthology.orchestra.run.smithi033.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-19T00:50:54.216 INFO:teuthology.orchestra.run.smithi033.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-19T00:50:54.258 INFO:teuthology.orchestra.run.smithi033.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-19T00:50:54.263 INFO:teuthology.orchestra.run.smithi033.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-19T00:50:54.426 INFO:teuthology.orchestra.run.smithi033.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-19T00:50:54.426 INFO:teuthology.orchestra.run.smithi033.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-19T00:50:54.426 INFO:teuthology.orchestra.run.smithi033.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-19T00:50:54.466 INFO:teuthology.orchestra.run.smithi033.stderr:done 2024-04-19T00:50:54.475 DEBUG:teuthology.orchestra.run.smithi033:> sudo shutdown -r now 2024-04-19T00:51:01.283 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.110 2024-04-19T00:51:04.350 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2024-04-19T00:51:05.374 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-04-19T00:51:10.286 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-04-19T00:51:10.287 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T00:51:13.342 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.110 2024-04-19T00:51:13.351 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-04-19T00:51:13.351 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T00:51:14.376 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-04-19T00:51:14.377 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T00:51:16.414 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2024-04-19T00:51:17.438 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-04-19T00:51:24.485 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-19T00:51:24.486 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-04-19T00:51:24.486 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T00:51:25.344 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-04-19T00:51:25.345 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T00:51:28.418 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-04-19T00:51:28.419 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T00:51:29.440 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-04-19T00:51:29.441 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T00:51:31.486 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2024-04-19T00:51:43.006 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.33 2024-04-19T00:51:46.490 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-04-19T00:51:46.491 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T00:51:52.010 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-04-19T00:51:52.011 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T00:51:57.161 DEBUG:teuthology.orchestra.run.smithi110:> true 2024-04-19T00:51:57.788 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-04-19T00:51:57.788 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-105-generic"... 2024-04-19T00:51:57.788 DEBUG:teuthology.orchestra.run.smithi110:> uname -r 2024-04-19T00:51:57.834 INFO:teuthology.orchestra.run.smithi110.stdout:5.15.0-105-generic 2024-04-19T00:51:57.835 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-19T00:51:57.835 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-19T00:51:57.835 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-19T00:51:58.836 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-19T00:51:58.836 DEBUG:teuthology.orchestra.run.smithi110:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-19T00:51:58.913 INFO:teuthology.orchestra.run.smithi110.stdout:ttyS1 2024-04-19T00:51:58.936 DEBUG:teuthology.parallel:result is None 2024-04-19T00:52:05.022 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2024-04-19T00:52:23.023 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-04-19T00:52:23.025 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T00:52:23.279 DEBUG:teuthology.orchestra.run.smithi118:> true 2024-04-19T00:52:23.819 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-04-19T00:52:23.819 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.15.0-105-generic"... 2024-04-19T00:52:23.820 DEBUG:teuthology.orchestra.run.smithi118:> uname -r 2024-04-19T00:52:23.866 INFO:teuthology.orchestra.run.smithi118.stdout:5.15.0-105-generic 2024-04-19T00:52:23.866 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-19T00:52:23.866 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-19T00:52:23.866 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-19T00:52:24.866 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-04-19T00:52:24.867 DEBUG:teuthology.orchestra.run.smithi118:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-19T00:52:24.943 INFO:teuthology.orchestra.run.smithi118.stdout:ttyS1 2024-04-19T00:52:24.967 DEBUG:teuthology.parallel:result is None 2024-04-19T00:52:29.444 DEBUG:teuthology.orchestra.remote:timed out 2024-04-19T00:52:44.445 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-04-19T00:52:44.446 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T00:52:44.680 DEBUG:teuthology.orchestra.run.smithi155:> true 2024-04-19T00:52:45.279 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-04-19T00:52:45.279 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.15.0-105-generic"... 2024-04-19T00:52:45.279 DEBUG:teuthology.orchestra.run.smithi155:> uname -r 2024-04-19T00:52:45.326 INFO:teuthology.orchestra.run.smithi155.stdout:5.15.0-105-generic 2024-04-19T00:52:45.327 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-19T00:52:45.327 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-19T00:52:45.327 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-19T00:52:46.328 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2024-04-19T00:52:46.328 DEBUG:teuthology.orchestra.run.smithi155:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-19T00:52:46.396 INFO:teuthology.orchestra.run.smithi155.stdout:ttyS1 2024-04-19T00:52:46.416 DEBUG:teuthology.parallel:result is None 2024-04-19T00:52:52.014 DEBUG:teuthology.orchestra.remote:timed out 2024-04-19T00:53:04.015 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-04-19T00:53:04.016 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T00:53:04.258 DEBUG:teuthology.orchestra.run.smithi033:> true 2024-04-19T00:53:04.897 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-04-19T00:53:04.897 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-105-generic"... 2024-04-19T00:53:04.898 DEBUG:teuthology.orchestra.run.smithi033:> uname -r 2024-04-19T00:53:04.947 INFO:teuthology.orchestra.run.smithi033.stdout:5.15.0-105-generic 2024-04-19T00:53:04.947 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-19T00:53:04.948 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-19T00:53:04.948 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-19T00:53:05.949 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-19T00:53:05.949 DEBUG:teuthology.orchestra.run.smithi033:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-19T00:53:06.017 INFO:teuthology.orchestra.run.smithi033.stdout:ttyS1 2024-04-19T00:53:06.040 DEBUG:teuthology.parallel:result is None 2024-04-19T00:53:06.041 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-19T00:53:06.049 INFO:teuthology.task.internal:Creating test directory... 2024-04-19T00:53:06.049 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-19T00:53:06.051 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-19T00:53:06.054 DEBUG:teuthology.orchestra.run.smithi118:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-19T00:53:06.057 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-19T00:53:06.064 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-19T00:53:06.076 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-19T00:53:06.136 INFO:teuthology.task.internal:Creating archive directory... 2024-04-19T00:53:06.136 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-19T00:53:06.138 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-19T00:53:06.140 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-19T00:53:06.143 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-19T00:53:06.230 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-19T00:53:06.237 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-19T00:53:06.237 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-19T00:53:06.240 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-19T00:53:06.242 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-19T00:53:06.244 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-19T00:53:06.256 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-19T00:53:06.259 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-19T00:53:06.263 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-19T00:53:06.265 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-19T00:53:06.268 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-19T00:53:06.268 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-19T00:53:06.272 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-19T00:53:06.277 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-19T00:53:06.279 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-19T00:53:06.286 INFO:teuthology.task.internal:Configuring sudo... 2024-04-19T00:53:06.286 DEBUG:teuthology.orchestra.run.smithi033:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-19T00:53:06.313 DEBUG:teuthology.orchestra.run.smithi110:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-19T00:53:06.315 DEBUG:teuthology.orchestra.run.smithi118:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-19T00:53:06.322 DEBUG:teuthology.orchestra.run.smithi155:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-19T00:53:06.354 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-19T00:53:06.363 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-19T00:53:06.363 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-19T00:53:06.372 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-19T00:53:06.375 DEBUG:teuthology.orchestra.run.smithi118:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-19T00:53:06.378 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-19T00:53:06.402 DEBUG:teuthology.orchestra.run.smithi033:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-19T00:53:06.424 DEBUG:teuthology.orchestra.run.smithi033:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-19T00:53:06.472 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-19T00:53:06.472 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-19T00:53:06.535 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-19T00:53:06.541 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-19T00:53:06.590 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-19T00:53:06.590 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-19T00:53:06.652 DEBUG:teuthology.orchestra.run.smithi118:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-19T00:53:06.657 DEBUG:teuthology.orchestra.run.smithi118:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-19T00:53:06.705 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-04-19T00:53:06.706 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-19T00:53:06.775 DEBUG:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-19T00:53:06.780 DEBUG:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-19T00:53:06.827 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-19T00:53:06.827 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-19T00:53:06.897 DEBUG:teuthology.orchestra.run.smithi033:> sudo service rsyslog restart 2024-04-19T00:53:06.900 DEBUG:teuthology.orchestra.run.smithi110:> sudo service rsyslog restart 2024-04-19T00:53:06.902 DEBUG:teuthology.orchestra.run.smithi118:> sudo service rsyslog restart 2024-04-19T00:53:06.905 DEBUG:teuthology.orchestra.run.smithi155:> sudo service rsyslog restart 2024-04-19T00:53:06.977 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-19T00:53:06.985 INFO:teuthology.task.internal:Starting timer... 2024-04-19T00:53:06.985 INFO:teuthology.run_tasks:Running task pcp... 2024-04-19T00:53:07.021 INFO:teuthology.run_tasks:Running task selinux... 2024-04-19T00:53:07.090 DEBUG:teuthology.task.selinux:Excluding smithi033: OS 'ubuntu' does not support SELinux 2024-04-19T00:53:07.090 DEBUG:teuthology.task.selinux:Excluding smithi110: OS 'ubuntu' does not support SELinux 2024-04-19T00:53:07.090 DEBUG:teuthology.task.selinux:Excluding smithi118: OS 'ubuntu' does not support SELinux 2024-04-19T00:53:07.090 DEBUG:teuthology.task.selinux:Excluding smithi155: OS 'ubuntu' does not support SELinux 2024-04-19T00:53:07.091 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-19T00:53:07.091 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-04-19T00:53:07.091 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-19T00:53:07.091 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-19T00:53:07.232 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-19T00:53:07.237 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-19T00:53:07.363 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-19T00:53:07.378 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-19T00:53:07.380 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi033.front.sepia.ceph.com,smithi110.front.sepia.ceph.com,smithi118.front.sepia.ceph.com,smithi155.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-19T01:00:18.356 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi033.front.sepia.ceph.com'), Remote(name='ubuntu@smithi110.front.sepia.ceph.com'), Remote(name='ubuntu@smithi118.front.sepia.ceph.com'), Remote(name='ubuntu@smithi155.front.sepia.ceph.com')] 2024-04-19T01:00:18.357 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-04-19T01:00:18.358 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T01:00:18.443 DEBUG:teuthology.orchestra.run.smithi033:> true 2024-04-19T01:00:18.526 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-04-19T01:00:18.527 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-04-19T01:00:18.527 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T01:00:18.607 DEBUG:teuthology.orchestra.run.smithi110:> true 2024-04-19T01:00:18.682 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-04-19T01:00:18.682 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-04-19T01:00:18.683 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T01:00:18.763 DEBUG:teuthology.orchestra.run.smithi118:> true 2024-04-19T01:00:18.838 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-04-19T01:00:18.838 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-04-19T01:00:18.839 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T01:00:18.922 DEBUG:teuthology.orchestra.run.smithi155:> true 2024-04-19T01:00:19.002 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-04-19T01:00:19.003 INFO:teuthology.run_tasks:Running task clock... 2024-04-19T01:00:19.015 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-19T01:00:19.015 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-19T01:00:19.016 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-19T01:00:19.018 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-19T01:00:19.018 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-19T01:00:19.021 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-19T01:00:19.021 DEBUG:teuthology.orchestra.run.smithi118:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-19T01:00:19.024 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-19T01:00:19.024 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-19T01:00:19.046 INFO:teuthology.orchestra.run.smithi033.stdout:19 Apr 01:00:19 ntpd[17082]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-19T01:00:19.046 INFO:teuthology.orchestra.run.smithi033.stdout:19 Apr 01:00:19 ntpd[17082]: Command line: ntpd -gq 2024-04-19T01:00:19.046 INFO:teuthology.orchestra.run.smithi033.stdout:19 Apr 01:00:19 ntpd[17082]: ---------------------------------------------------- 2024-04-19T01:00:19.046 INFO:teuthology.orchestra.run.smithi033.stdout:19 Apr 01:00:19 ntpd[17082]: ntp-4 is maintained by Network Time Foundation, 2024-04-19T01:00:19.046 INFO:teuthology.orchestra.run.smithi033.stdout:19 Apr 01:00:19 ntpd[17082]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-19T01:00:19.046 INFO:teuthology.orchestra.run.smithi033.stdout:19 Apr 01:00:19 ntpd[17082]: corporation. Support and training for ntp-4 are 2024-04-19T01:00:19.047 INFO:teuthology.orchestra.run.smithi033.stdout:19 Apr 01:00:19 ntpd[17082]: available at https://www.nwtime.org/support 2024-04-19T01:00:19.047 INFO:teuthology.orchestra.run.smithi033.stdout:19 Apr 01:00:19 ntpd[17082]: ---------------------------------------------------- 2024-04-19T01:00:19.047 INFO:teuthology.orchestra.run.smithi033.stdout:19 Apr 01:00:19 ntpd[17082]: proto: precision = 0.045 usec (-24) 2024-04-19T01:00:19.048 INFO:teuthology.orchestra.run.smithi033.stdout:19 Apr 01:00:19 ntpd[17082]: basedate set to 2022-02-04 2024-04-19T01:00:19.048 INFO:teuthology.orchestra.run.smithi033.stdout:19 Apr 01:00:19 ntpd[17082]: gps base set to 2022-02-06 (week 2196) 2024-04-19T01:00:19.048 INFO:teuthology.orchestra.run.smithi033.stdout:19 Apr 01:00:19 ntpd[17082]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-19T01:00:19.048 INFO:teuthology.orchestra.run.smithi033.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-19T01:00:19.048 INFO:teuthology.orchestra.run.smithi033.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-19T01:00:19.049 INFO:teuthology.orchestra.run.smithi033.stdout:19 Apr 01:00:19 ntpd[17082]: restrict ::: KOD does nothing without LIMITED. 2024-04-19T01:00:19.049 INFO:teuthology.orchestra.run.smithi033.stdout:19 Apr 01:00:19 ntpd[17082]: Listen and drop on 0 v6wildcard [::]:123 2024-04-19T01:00:19.049 INFO:teuthology.orchestra.run.smithi033.stdout:19 Apr 01:00:19 ntpd[17082]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-19T01:00:19.049 INFO:teuthology.orchestra.run.smithi033.stdout:19 Apr 01:00:19 ntpd[17082]: Listen normally on 2 lo 127.0.0.1:123 2024-04-19T01:00:19.050 INFO:teuthology.orchestra.run.smithi033.stdout:19 Apr 01:00:19 ntpd[17082]: Listen normally on 3 ens1f0 172.21.15.33:123 2024-04-19T01:00:19.050 INFO:teuthology.orchestra.run.smithi033.stdout:19 Apr 01:00:19 ntpd[17082]: Listen normally on 4 lo [::1]:123 2024-04-19T01:00:19.050 INFO:teuthology.orchestra.run.smithi033.stdout:19 Apr 01:00:19 ntpd[17082]: Listen normally on 5 ens1f0 [fe80::ae1f:6bff:fea5:d294%4]:123 2024-04-19T01:00:19.050 INFO:teuthology.orchestra.run.smithi033.stdout:19 Apr 01:00:19 ntpd[17082]: Listening on routing socket on fd #22 for interface updates 2024-04-19T01:00:19.051 INFO:teuthology.orchestra.run.smithi110.stdout:19 Apr 01:00:19 ntpd[17219]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-19T01:00:19.051 INFO:teuthology.orchestra.run.smithi110.stdout:19 Apr 01:00:19 ntpd[17219]: Command line: ntpd -gq 2024-04-19T01:00:19.051 INFO:teuthology.orchestra.run.smithi110.stdout:19 Apr 01:00:19 ntpd[17219]: ---------------------------------------------------- 2024-04-19T01:00:19.051 INFO:teuthology.orchestra.run.smithi110.stdout:19 Apr 01:00:19 ntpd[17219]: ntp-4 is maintained by Network Time Foundation, 2024-04-19T01:00:19.051 INFO:teuthology.orchestra.run.smithi110.stdout:19 Apr 01:00:19 ntpd[17219]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-19T01:00:19.051 INFO:teuthology.orchestra.run.smithi110.stdout:19 Apr 01:00:19 ntpd[17219]: corporation. Support and training for ntp-4 are 2024-04-19T01:00:19.051 INFO:teuthology.orchestra.run.smithi110.stdout:19 Apr 01:00:19 ntpd[17219]: available at https://www.nwtime.org/support 2024-04-19T01:00:19.051 INFO:teuthology.orchestra.run.smithi110.stdout:19 Apr 01:00:19 ntpd[17219]: ---------------------------------------------------- 2024-04-19T01:00:19.051 INFO:teuthology.orchestra.run.smithi110.stdout:19 Apr 01:00:19 ntpd[17219]: proto: precision = 0.057 usec (-24) 2024-04-19T01:00:19.052 INFO:teuthology.orchestra.run.smithi110.stdout:19 Apr 01:00:19 ntpd[17219]: basedate set to 2022-02-04 2024-04-19T01:00:19.052 INFO:teuthology.orchestra.run.smithi110.stdout:19 Apr 01:00:19 ntpd[17219]: gps base set to 2022-02-06 (week 2196) 2024-04-19T01:00:19.052 INFO:teuthology.orchestra.run.smithi110.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-19T01:00:19.052 INFO:teuthology.orchestra.run.smithi110.stdout:19 Apr 01:00:19 ntpd[17219]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-19T01:00:19.052 INFO:teuthology.orchestra.run.smithi110.stdout:19 Apr 01:00:19 ntpd[17219]: restrict ::: KOD does nothing without LIMITED. 2024-04-19T01:00:19.053 INFO:teuthology.orchestra.run.smithi110.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-19T01:00:19.054 INFO:teuthology.orchestra.run.smithi110.stdout:19 Apr 01:00:19 ntpd[17219]: Listen and drop on 0 v6wildcard [::]:123 2024-04-19T01:00:19.054 INFO:teuthology.orchestra.run.smithi110.stdout:19 Apr 01:00:19 ntpd[17219]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-19T01:00:19.054 INFO:teuthology.orchestra.run.smithi110.stdout:19 Apr 01:00:19 ntpd[17219]: Listen normally on 2 lo 127.0.0.1:123 2024-04-19T01:00:19.054 INFO:teuthology.orchestra.run.smithi110.stdout:19 Apr 01:00:19 ntpd[17219]: Listen normally on 3 enp3s0f1 172.21.15.110:123 2024-04-19T01:00:19.054 INFO:teuthology.orchestra.run.smithi110.stdout:19 Apr 01:00:19 ntpd[17219]: Listen normally on 4 lo [::1]:123 2024-04-19T01:00:19.054 INFO:teuthology.orchestra.run.smithi110.stdout:19 Apr 01:00:19 ntpd[17219]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:cc35%5]:123 2024-04-19T01:00:19.054 INFO:teuthology.orchestra.run.smithi110.stdout:19 Apr 01:00:19 ntpd[17219]: Listening on routing socket on fd #22 for interface updates 2024-04-19T01:00:19.055 INFO:teuthology.orchestra.run.smithi118.stdout:19 Apr 01:00:19 ntpd[17212]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-19T01:00:19.055 INFO:teuthology.orchestra.run.smithi118.stdout:19 Apr 01:00:19 ntpd[17212]: Command line: ntpd -gq 2024-04-19T01:00:19.055 INFO:teuthology.orchestra.run.smithi118.stdout:19 Apr 01:00:19 ntpd[17212]: ---------------------------------------------------- 2024-04-19T01:00:19.055 INFO:teuthology.orchestra.run.smithi118.stdout:19 Apr 01:00:19 ntpd[17212]: ntp-4 is maintained by Network Time Foundation, 2024-04-19T01:00:19.055 INFO:teuthology.orchestra.run.smithi118.stdout:19 Apr 01:00:19 ntpd[17212]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-19T01:00:19.056 INFO:teuthology.orchestra.run.smithi118.stdout:19 Apr 01:00:19 ntpd[17212]: corporation. Support and training for ntp-4 are 2024-04-19T01:00:19.056 INFO:teuthology.orchestra.run.smithi118.stdout:19 Apr 01:00:19 ntpd[17212]: available at https://www.nwtime.org/support 2024-04-19T01:00:19.056 INFO:teuthology.orchestra.run.smithi118.stdout:19 Apr 01:00:19 ntpd[17212]: ---------------------------------------------------- 2024-04-19T01:00:19.056 INFO:teuthology.orchestra.run.smithi118.stdout:19 Apr 01:00:19 ntpd[17212]: proto: precision = 0.062 usec (-24) 2024-04-19T01:00:19.056 INFO:teuthology.orchestra.run.smithi118.stdout:19 Apr 01:00:19 ntpd[17212]: basedate set to 2022-02-04 2024-04-19T01:00:19.056 INFO:teuthology.orchestra.run.smithi118.stdout:19 Apr 01:00:19 ntpd[17212]: gps base set to 2022-02-06 (week 2196) 2024-04-19T01:00:19.056 INFO:teuthology.orchestra.run.smithi118.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-19T01:00:19.057 INFO:teuthology.orchestra.run.smithi118.stdout:19 Apr 01:00:19 ntpd[17212]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-19T01:00:19.057 INFO:teuthology.orchestra.run.smithi118.stdout:19 Apr 01:00:19 ntpd[17212]: restrict ::: KOD does nothing without LIMITED. 2024-04-19T01:00:19.057 INFO:teuthology.orchestra.run.smithi118.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-19T01:00:19.058 INFO:teuthology.orchestra.run.smithi118.stdout:19 Apr 01:00:19 ntpd[17212]: Listen and drop on 0 v6wildcard [::]:123 2024-04-19T01:00:19.058 INFO:teuthology.orchestra.run.smithi118.stdout:19 Apr 01:00:19 ntpd[17212]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-19T01:00:19.058 INFO:teuthology.orchestra.run.smithi118.stdout:19 Apr 01:00:19 ntpd[17212]: Listen normally on 2 lo 127.0.0.1:123 2024-04-19T01:00:19.058 INFO:teuthology.orchestra.run.smithi118.stdout:19 Apr 01:00:19 ntpd[17212]: Listen normally on 3 enp3s0f1 172.21.15.118:123 2024-04-19T01:00:19.058 INFO:teuthology.orchestra.run.smithi118.stdout:19 Apr 01:00:19 ntpd[17212]: Listen normally on 4 lo [::1]:123 2024-04-19T01:00:19.058 INFO:teuthology.orchestra.run.smithi118.stdout:19 Apr 01:00:19 ntpd[17212]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d725%5]:123 2024-04-19T01:00:19.058 INFO:teuthology.orchestra.run.smithi118.stdout:19 Apr 01:00:19 ntpd[17212]: Listening on routing socket on fd #22 for interface updates 2024-04-19T01:00:19.069 INFO:teuthology.orchestra.run.smithi155.stdout:19 Apr 01:00:19 ntpd[17252]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-19T01:00:19.070 INFO:teuthology.orchestra.run.smithi155.stdout:19 Apr 01:00:19 ntpd[17252]: Command line: ntpd -gq 2024-04-19T01:00:19.070 INFO:teuthology.orchestra.run.smithi155.stdout:19 Apr 01:00:19 ntpd[17252]: ---------------------------------------------------- 2024-04-19T01:00:19.070 INFO:teuthology.orchestra.run.smithi155.stdout:19 Apr 01:00:19 ntpd[17252]: ntp-4 is maintained by Network Time Foundation, 2024-04-19T01:00:19.070 INFO:teuthology.orchestra.run.smithi155.stdout:19 Apr 01:00:19 ntpd[17252]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-19T01:00:19.070 INFO:teuthology.orchestra.run.smithi155.stdout:19 Apr 01:00:19 ntpd[17252]: corporation. Support and training for ntp-4 are 2024-04-19T01:00:19.070 INFO:teuthology.orchestra.run.smithi155.stdout:19 Apr 01:00:19 ntpd[17252]: available at https://www.nwtime.org/support 2024-04-19T01:00:19.070 INFO:teuthology.orchestra.run.smithi155.stdout:19 Apr 01:00:19 ntpd[17252]: ---------------------------------------------------- 2024-04-19T01:00:19.071 INFO:teuthology.orchestra.run.smithi155.stdout:19 Apr 01:00:19 ntpd[17252]: proto: precision = 0.046 usec (-24) 2024-04-19T01:00:19.071 INFO:teuthology.orchestra.run.smithi155.stdout:19 Apr 01:00:19 ntpd[17252]: basedate set to 2022-02-04 2024-04-19T01:00:19.071 INFO:teuthology.orchestra.run.smithi155.stdout:19 Apr 01:00:19 ntpd[17252]: gps base set to 2022-02-06 (week 2196) 2024-04-19T01:00:19.071 INFO:teuthology.orchestra.run.smithi155.stdout:19 Apr 01:00:19 ntpd[17252]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-19T01:00:19.071 INFO:teuthology.orchestra.run.smithi155.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-19T01:00:19.071 INFO:teuthology.orchestra.run.smithi155.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-19T01:00:19.072 INFO:teuthology.orchestra.run.smithi155.stdout:19 Apr 01:00:19 ntpd[17252]: restrict ::: KOD does nothing without LIMITED. 2024-04-19T01:00:19.073 INFO:teuthology.orchestra.run.smithi155.stdout:19 Apr 01:00:19 ntpd[17252]: Listen and drop on 0 v6wildcard [::]:123 2024-04-19T01:00:19.073 INFO:teuthology.orchestra.run.smithi155.stdout:19 Apr 01:00:19 ntpd[17252]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-19T01:00:19.073 INFO:teuthology.orchestra.run.smithi155.stdout:19 Apr 01:00:19 ntpd[17252]: Listen normally on 2 lo 127.0.0.1:123 2024-04-19T01:00:19.073 INFO:teuthology.orchestra.run.smithi155.stdout:19 Apr 01:00:19 ntpd[17252]: Listen normally on 3 enp3s0f1 172.21.15.155:123 2024-04-19T01:00:19.073 INFO:teuthology.orchestra.run.smithi155.stdout:19 Apr 01:00:19 ntpd[17252]: Listen normally on 4 lo [::1]:123 2024-04-19T01:00:19.073 INFO:teuthology.orchestra.run.smithi155.stdout:19 Apr 01:00:19 ntpd[17252]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6ded%5]:123 2024-04-19T01:00:19.073 INFO:teuthology.orchestra.run.smithi155.stdout:19 Apr 01:00:19 ntpd[17252]: Listening on routing socket on fd #22 for interface updates 2024-04-19T01:00:20.047 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:20 ntpd[17082]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:20.047 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:20 ntpd[17082]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:20.047 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:20 ntpd[17082]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:20.047 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:20 ntpd[17082]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:20.047 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:20 ntpd[17082]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:00:20.047 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:20 ntpd[17082]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:00:20.050 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:20 ntpd[17219]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:20.051 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:20 ntpd[17219]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:20.051 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:20 ntpd[17219]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:20.051 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:20 ntpd[17219]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:20.051 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:20 ntpd[17219]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:20.051 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:20 ntpd[17219]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:20.051 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:20 ntpd[17219]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:00:20.051 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:20 ntpd[17219]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:00:20.051 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:20 ntpd[17219]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:20.051 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:20 ntpd[17219]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:20.051 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:20 ntpd[17219]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:00:20.051 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:20 ntpd[17219]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:00:20.070 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:20 ntpd[17252]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:20.070 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:20 ntpd[17252]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:20.070 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:20 ntpd[17252]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:20.071 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:20 ntpd[17252]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:20.071 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:20 ntpd[17252]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:00:20.071 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:20 ntpd[17252]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:00:21.047 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:21 ntpd[17082]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:21.047 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:21 ntpd[17082]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:21.047 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:21 ntpd[17082]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:21.047 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:21 ntpd[17082]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:21.047 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:21 ntpd[17082]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:00:21.047 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:21 ntpd[17082]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:00:21.052 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:21 ntpd[17212]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:21.052 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:21 ntpd[17212]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:22.047 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:22 ntpd[17082]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:22.047 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:22 ntpd[17082]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:22.047 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:22 ntpd[17082]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:22.047 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:22 ntpd[17082]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:22.047 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:22 ntpd[17082]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:22.048 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:22 ntpd[17082]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:22.048 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:22 ntpd[17082]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:00:22.048 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:22 ntpd[17082]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:00:22.051 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:22 ntpd[17219]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:22.051 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:22 ntpd[17219]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:22.051 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:22 ntpd[17219]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:22.051 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:22 ntpd[17219]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:22.051 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:22 ntpd[17219]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:22.051 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:22 ntpd[17219]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:22.051 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:22 ntpd[17219]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:00:22.051 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:22 ntpd[17219]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:00:22.051 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:22 ntpd[17219]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:22.051 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:22 ntpd[17219]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:22.051 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:22 ntpd[17219]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:00:22.051 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:22 ntpd[17219]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:00:22.053 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:22 ntpd[17212]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:22.053 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:22 ntpd[17212]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:22.053 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:22 ntpd[17212]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:22.053 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:22 ntpd[17212]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:22.053 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:22 ntpd[17212]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:22.053 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:22 ntpd[17212]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:22.053 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:22 ntpd[17212]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:00:22.053 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:22 ntpd[17212]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:00:22.053 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:22 ntpd[17212]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:22.053 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:22 ntpd[17212]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:22.053 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:22 ntpd[17212]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:00:22.054 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:22 ntpd[17212]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:00:22.071 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:22 ntpd[17252]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:22.071 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:22 ntpd[17252]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:22.071 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:22 ntpd[17252]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:22.071 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:22 ntpd[17252]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:22.071 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:22 ntpd[17252]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:22.071 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:22 ntpd[17252]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:22.071 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:22 ntpd[17252]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:00:22.071 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:22 ntpd[17252]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:00:22.071 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:22 ntpd[17252]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:22.072 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:22 ntpd[17252]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:22.072 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:22 ntpd[17252]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:00:22.072 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:22 ntpd[17252]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:00:23.047 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:23 ntpd[17082]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:23.047 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:23 ntpd[17082]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:23.047 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:23 ntpd[17082]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:23.047 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:23 ntpd[17082]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:23.047 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:23 ntpd[17082]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:00:23.047 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:23 ntpd[17082]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:00:23.050 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:23 ntpd[17219]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:23.050 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:23 ntpd[17219]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:23.052 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:23 ntpd[17212]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:23.052 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:23 ntpd[17212]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:23.053 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:23 ntpd[17212]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:23.053 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:23 ntpd[17212]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:23.053 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:23 ntpd[17212]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:00:23.053 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:23 ntpd[17212]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:00:23.070 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:23 ntpd[17252]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:23.070 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:23 ntpd[17252]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:23.071 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:23 ntpd[17252]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:23.071 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:23 ntpd[17252]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:23.071 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:23 ntpd[17252]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:00:23.071 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:23 ntpd[17252]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:00:24.047 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:24 ntpd[17082]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:24.048 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:24 ntpd[17082]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:24.048 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:24 ntpd[17082]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:24.048 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:24 ntpd[17082]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:24.048 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:24 ntpd[17082]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:24.048 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:24 ntpd[17082]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:24.048 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:24 ntpd[17082]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:00:24.048 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:24 ntpd[17082]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:00:24.048 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:24 ntpd[17082]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:24.048 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:24 ntpd[17082]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:24.048 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:24 ntpd[17082]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:00:24.048 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:24 ntpd[17082]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:00:24.051 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:24 ntpd[17219]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:24.051 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:24 ntpd[17219]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:24.051 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:24 ntpd[17219]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:24.051 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:24 ntpd[17219]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:24.051 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:24 ntpd[17219]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:24.051 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:24 ntpd[17219]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:24.051 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:24 ntpd[17219]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:24.052 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:24 ntpd[17219]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:24.052 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:24 ntpd[17219]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:00:24.052 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:24 ntpd[17219]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:00:24.052 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:24 ntpd[17219]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:24.052 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:24 ntpd[17219]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:24.052 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:24 ntpd[17219]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:00:24.052 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:24 ntpd[17219]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:00:24.052 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:24 ntpd[17219]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:24.052 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:24 ntpd[17219]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:24.052 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:24 ntpd[17219]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:00:24.052 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:24 ntpd[17219]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:00:24.053 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:24 ntpd[17212]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:24.053 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:24 ntpd[17212]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:24.054 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:24 ntpd[17212]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:24.054 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:24 ntpd[17212]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:24.054 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:24 ntpd[17212]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:24.054 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:24 ntpd[17212]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:24.054 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:24 ntpd[17212]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:00:24.054 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:24 ntpd[17212]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:00:24.054 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:24 ntpd[17212]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:24.054 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:24 ntpd[17212]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:24.054 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:24 ntpd[17212]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:00:24.054 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:24 ntpd[17212]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:00:24.071 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:24 ntpd[17252]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:24.071 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:24 ntpd[17252]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:24.071 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:24 ntpd[17252]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:24.071 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:24 ntpd[17252]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:24.071 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:24 ntpd[17252]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:24.071 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:24 ntpd[17252]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:24.071 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:24 ntpd[17252]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:24.071 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:24 ntpd[17252]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:24.071 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:24 ntpd[17252]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:00:24.072 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:24 ntpd[17252]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:00:24.072 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:24 ntpd[17252]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:24.072 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:24 ntpd[17252]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:24.072 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:24 ntpd[17252]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:00:24.072 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:24 ntpd[17252]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:00:25.047 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:25 ntpd[17082]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:25.047 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:25 ntpd[17082]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:25.047 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:25 ntpd[17082]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:25.047 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:25 ntpd[17082]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:25.047 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:25 ntpd[17082]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:00:25.048 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:25 ntpd[17082]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:00:25.052 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:25 ntpd[17212]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:25.052 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:25 ntpd[17212]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:25.052 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:25 ntpd[17212]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:25.053 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:25 ntpd[17212]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:25.053 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:25 ntpd[17212]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:00:25.053 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:25 ntpd[17212]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:00:25.070 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:25 ntpd[17252]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:25.070 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:25 ntpd[17252]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:25.070 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:25 ntpd[17252]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:25.071 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:25 ntpd[17252]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:25.071 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:25 ntpd[17252]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:00:25.071 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:25 ntpd[17252]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:00:26.047 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:26 ntpd[17082]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:26.047 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:26 ntpd[17082]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:26.047 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:26 ntpd[17082]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:26.048 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:26 ntpd[17082]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:26.048 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:26 ntpd[17082]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:26.048 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:26 ntpd[17082]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:26.048 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:26 ntpd[17082]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:00:26.048 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:26 ntpd[17082]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:00:26.048 INFO:teuthology.orchestra.run.smithi033.stdout:19 Apr 01:00:26 ntpd[17082]: ntpd: time slew -0.000725 s 2024-04-19T01:00:26.048 INFO:teuthology.orchestra.run.smithi033.stdout:ntpd: time slew -0.000725s 2024-04-19T01:00:26.049 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:26 ntpd[17082]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-19T01:00:26.049 INFO:teuthology.orchestra.run.smithi033.stderr:19 Apr 01:00:26 ntpd[17082]: can't open /var/log/ntpstats/loopstats.20240419: Permission denied 2024-04-19T01:00:26.051 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:26 ntpd[17219]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:26.051 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:26 ntpd[17219]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:26.051 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:26 ntpd[17219]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:26.051 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:26 ntpd[17219]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:26.051 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:26 ntpd[17219]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:26.051 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:26 ntpd[17219]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:26.051 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:26 ntpd[17219]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:26.051 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:26 ntpd[17219]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:26.051 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:26 ntpd[17219]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:00:26.052 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:26 ntpd[17219]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:00:26.052 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:26 ntpd[17219]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:26.052 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:26 ntpd[17219]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:26.052 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:26 ntpd[17219]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:00:26.052 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:26 ntpd[17219]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:00:26.052 INFO:teuthology.orchestra.run.smithi110.stdout:19 Apr 01:00:26 ntpd[17219]: ntpd: time slew -0.000757 s 2024-04-19T01:00:26.052 INFO:teuthology.orchestra.run.smithi110.stdout:ntpd: time slew -0.000757s 2024-04-19T01:00:26.053 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:26 ntpd[17219]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-19T01:00:26.053 INFO:teuthology.orchestra.run.smithi110.stderr:19 Apr 01:00:26 ntpd[17219]: can't open /var/log/ntpstats/loopstats.20240419: Permission denied 2024-04-19T01:00:26.054 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:26 ntpd[17212]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:26.054 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:26 ntpd[17212]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:26.054 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:26 ntpd[17212]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:26.054 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:26 ntpd[17212]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:26.054 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:26 ntpd[17212]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:26.054 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:26 ntpd[17212]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:26.054 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:26 ntpd[17212]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:00:26.054 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:26 ntpd[17212]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:00:26.055 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:26 ntpd[17212]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:26.055 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:26 ntpd[17212]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:26.055 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:26 ntpd[17212]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:00:26.055 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:26 ntpd[17212]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:00:26.071 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:26 ntpd[17252]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:26.071 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:26 ntpd[17252]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:26.071 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:26 ntpd[17252]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:26.071 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:26 ntpd[17252]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:26.071 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:26 ntpd[17252]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:26.071 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:26 ntpd[17252]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:26.071 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:26 ntpd[17252]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:00:26.071 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:26 ntpd[17252]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:00:26.072 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:26 ntpd[17252]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:26.072 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:26 ntpd[17252]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:26.072 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:26 ntpd[17252]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:00:26.072 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:26 ntpd[17252]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:00:26.072 INFO:teuthology.orchestra.run.smithi155.stdout:19 Apr 01:00:26 ntpd[17252]: ntpd: time slew -0.001440 s 2024-04-19T01:00:26.072 INFO:teuthology.orchestra.run.smithi155.stdout:ntpd: time slew -0.001440s 2024-04-19T01:00:26.073 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:26 ntpd[17252]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-19T01:00:26.073 INFO:teuthology.orchestra.run.smithi155.stderr:19 Apr 01:00:26 ntpd[17252]: can't open /var/log/ntpstats/loopstats.20240419: Permission denied 2024-04-19T01:00:26.093 INFO:teuthology.orchestra.run.smithi033.stdout: remote refid st t when poll reach delay offset jitter 2024-04-19T01:00:26.094 INFO:teuthology.orchestra.run.smithi033.stdout:============================================================================== 2024-04-19T01:00:26.094 INFO:teuthology.orchestra.run.smithi033.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:00:26.094 INFO:teuthology.orchestra.run.smithi033.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:00:26.094 INFO:teuthology.orchestra.run.smithi033.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:00:26.094 INFO:teuthology.orchestra.run.smithi033.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:00:26.111 INFO:teuthology.orchestra.run.smithi110.stdout: remote refid st t when poll reach delay offset jitter 2024-04-19T01:00:26.111 INFO:teuthology.orchestra.run.smithi110.stdout:============================================================================== 2024-04-19T01:00:26.111 INFO:teuthology.orchestra.run.smithi110.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:00:26.111 INFO:teuthology.orchestra.run.smithi110.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:00:26.111 INFO:teuthology.orchestra.run.smithi110.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:00:26.111 INFO:teuthology.orchestra.run.smithi110.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:00:26.117 INFO:teuthology.orchestra.run.smithi155.stdout: remote refid st t when poll reach delay offset jitter 2024-04-19T01:00:26.117 INFO:teuthology.orchestra.run.smithi155.stdout:============================================================================== 2024-04-19T01:00:26.117 INFO:teuthology.orchestra.run.smithi155.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:00:26.117 INFO:teuthology.orchestra.run.smithi155.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:00:26.117 INFO:teuthology.orchestra.run.smithi155.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:00:26.118 INFO:teuthology.orchestra.run.smithi155.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:00:27.053 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:27 ntpd[17212]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:27.053 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:27 ntpd[17212]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:27.053 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:27 ntpd[17212]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:27.053 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:27 ntpd[17212]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:27.053 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:27 ntpd[17212]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:00:27.053 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:27 ntpd[17212]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:00:28.052 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:28 ntpd[17212]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:28.053 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:28 ntpd[17212]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:28.053 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:28 ntpd[17212]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:28.053 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:28 ntpd[17212]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:28.053 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:28 ntpd[17212]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:00:28.053 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:28 ntpd[17212]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:00:28.053 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:28 ntpd[17212]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:00:28.053 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:28 ntpd[17212]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:00:28.053 INFO:teuthology.orchestra.run.smithi118.stdout:19 Apr 01:00:28 ntpd[17212]: ntpd: time slew -0.000365 s 2024-04-19T01:00:28.053 INFO:teuthology.orchestra.run.smithi118.stdout:ntpd: time slew -0.000365s 2024-04-19T01:00:28.054 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:28 ntpd[17212]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-19T01:00:28.054 INFO:teuthology.orchestra.run.smithi118.stderr:19 Apr 01:00:28 ntpd[17212]: can't open /var/log/ntpstats/loopstats.20240419: Permission denied 2024-04-19T01:00:28.101 INFO:teuthology.orchestra.run.smithi118.stdout: remote refid st t when poll reach delay offset jitter 2024-04-19T01:00:28.101 INFO:teuthology.orchestra.run.smithi118.stdout:============================================================================== 2024-04-19T01:00:28.101 INFO:teuthology.orchestra.run.smithi118.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:00:28.101 INFO:teuthology.orchestra.run.smithi118.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:00:28.101 INFO:teuthology.orchestra.run.smithi118.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:00:28.101 INFO:teuthology.orchestra.run.smithi118.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:00:28.102 INFO:teuthology.run_tasks:Running task install... 2024-04-19T01:00:28.112 DEBUG:teuthology.task.install:project ceph 2024-04-19T01:00:28.112 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8b4ab94f9c30da63487fa2f241ce12f483378860'}} 2024-04-19T01:00:28.112 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '8b4ab94f9c30da63487fa2f241ce12f483378860'} 2024-04-19T01:00:28.112 INFO:teuthology.task.install:Using flavor: default 2024-04-19T01:00:28.118 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-19T01:00:28.118 INFO:teuthology.task.install:extra packages: [] 2024-04-19T01:00:28.119 DEBUG:teuthology.orchestra.run.smithi033:> sudo apt-key list | grep Ceph 2024-04-19T01:00:28.119 DEBUG:teuthology.orchestra.run.smithi110:> sudo apt-key list | grep Ceph 2024-04-19T01:00:28.120 DEBUG:teuthology.orchestra.run.smithi118:> sudo apt-key list | grep Ceph 2024-04-19T01:00:28.120 DEBUG:teuthology.orchestra.run.smithi155:> sudo apt-key list | grep Ceph 2024-04-19T01:00:28.183 INFO:teuthology.orchestra.run.smithi110.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-19T01:00:28.183 INFO:teuthology.orchestra.run.smithi155.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-19T01:00:28.188 INFO:teuthology.orchestra.run.smithi033.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-19T01:00:28.200 INFO:teuthology.orchestra.run.smithi118.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-19T01:00:28.211 INFO:teuthology.orchestra.run.smithi110.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-19T01:00:28.212 INFO:teuthology.orchestra.run.smithi110.stdout:uid [ unknown] Ceph.com (release key) 2024-04-19T01:00:28.212 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-19T01:00:28.212 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8b4ab94f9c30da63487fa2f241ce12f483378860 2024-04-19T01:00:28.217 INFO:teuthology.orchestra.run.smithi155.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-19T01:00:28.217 INFO:teuthology.orchestra.run.smithi155.stdout:uid [ unknown] Ceph.com (release key) 2024-04-19T01:00:28.217 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-19T01:00:28.217 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8b4ab94f9c30da63487fa2f241ce12f483378860 2024-04-19T01:00:28.220 INFO:teuthology.orchestra.run.smithi033.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-19T01:00:28.220 INFO:teuthology.orchestra.run.smithi033.stdout:uid [ unknown] Ceph.com (release key) 2024-04-19T01:00:28.221 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-19T01:00:28.221 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8b4ab94f9c30da63487fa2f241ce12f483378860 2024-04-19T01:00:28.226 INFO:teuthology.orchestra.run.smithi118.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-19T01:00:28.226 INFO:teuthology.orchestra.run.smithi118.stdout:uid [ unknown] Ceph.com (release key) 2024-04-19T01:00:28.226 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-19T01:00:28.227 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8b4ab94f9c30da63487fa2f241ce12f483378860 2024-04-19T01:00:28.380 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default/ 2024-04-19T01:00:28.380 INFO:teuthology.task.install.deb:Package version is 19.0.0-3154-g8b4ab94f-1jammy 2024-04-19T01:00:28.386 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default/ 2024-04-19T01:00:28.386 INFO:teuthology.task.install.deb:Package version is 19.0.0-3154-g8b4ab94f-1jammy 2024-04-19T01:00:28.447 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default/ 2024-04-19T01:00:28.447 INFO:teuthology.task.install.deb:Package version is 19.0.0-3154-g8b4ab94f-1jammy 2024-04-19T01:00:28.451 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default/ 2024-04-19T01:00:28.451 INFO:teuthology.task.install.deb:Package version is 19.0.0-3154-g8b4ab94f-1jammy 2024-04-19T01:00:28.566 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-04-19T01:00:28.566 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-19T01:00:28.574 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-19T01:00:28.574 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-19T01:00:28.580 DEBUG:teuthology.orchestra.run.smithi118:> sudo apt-get update 2024-04-19T01:00:28.588 DEBUG:teuthology.orchestra.run.smithi110:> sudo apt-get update 2024-04-19T01:00:28.645 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-19T01:00:28.645 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-19T01:00:28.653 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-19T01:00:28.653 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-19T01:00:28.659 DEBUG:teuthology.orchestra.run.smithi155:> sudo apt-get update 2024-04-19T01:00:28.667 DEBUG:teuthology.orchestra.run.smithi033:> sudo apt-get update 2024-04-19T01:00:28.802 INFO:teuthology.orchestra.run.smithi118.stdout:Ign:1 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy InRelease 2024-04-19T01:00:28.823 INFO:teuthology.orchestra.run.smithi118.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-19T01:00:28.847 INFO:teuthology.orchestra.run.smithi118.stdout:Ign:3 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-19T01:00:28.868 INFO:teuthology.orchestra.run.smithi118.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,408 B] 2024-04-19T01:00:28.876 INFO:teuthology.orchestra.run.smithi155.stdout:Ign:1 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy InRelease 2024-04-19T01:00:28.895 INFO:teuthology.orchestra.run.smithi033.stdout:Ign:1 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy InRelease 2024-04-19T01:00:28.898 INFO:teuthology.orchestra.run.smithi155.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-19T01:00:28.900 INFO:teuthology.orchestra.run.smithi118.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-04-19T01:00:28.916 INFO:teuthology.orchestra.run.smithi033.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-19T01:00:28.921 INFO:teuthology.orchestra.run.smithi155.stdout:Ign:3 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-19T01:00:28.940 INFO:teuthology.orchestra.run.smithi033.stdout:Ign:3 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-19T01:00:28.943 INFO:teuthology.orchestra.run.smithi155.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-04-19T01:00:28.951 INFO:teuthology.orchestra.run.smithi110.stdout:Ign:1 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy InRelease 2024-04-19T01:00:28.960 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:5 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-19T01:00:28.961 INFO:teuthology.orchestra.run.smithi033.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,408 B] 2024-04-19T01:00:28.982 INFO:teuthology.orchestra.run.smithi155.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,408 B] 2024-04-19T01:00:28.993 INFO:teuthology.orchestra.run.smithi033.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-04-19T01:00:29.027 INFO:teuthology.orchestra.run.smithi110.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-19T01:00:29.103 INFO:teuthology.orchestra.run.smithi110.stdout:Ign:3 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-19T01:00:29.166 INFO:teuthology.orchestra.run.smithi110.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,408 B] 2024-04-19T01:00:29.235 INFO:teuthology.orchestra.run.smithi110.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-04-19T01:00:29.468 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-19T01:00:29.541 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:7 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-19T01:00:29.622 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-19T01:00:31.375 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:7 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-19T01:00:31.436 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-19T01:00:31.462 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-19T01:00:31.495 INFO:teuthology.orchestra.run.smithi118.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-19T01:00:31.507 INFO:teuthology.orchestra.run.smithi118.stdout:Hit:7 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-19T01:00:31.521 INFO:teuthology.orchestra.run.smithi118.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-19T01:00:31.567 INFO:teuthology.orchestra.run.smithi118.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-19T01:00:31.698 INFO:teuthology.orchestra.run.smithi155.stdout:Fetched 29.2 kB in 3s (10.2 kB/s) 2024-04-19T01:00:31.864 INFO:teuthology.orchestra.run.smithi118.stdout:Fetched 29.2 kB in 3s (9,429 B/s) 2024-04-19T01:00:32.025 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-19T01:00:32.053 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:7 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-19T01:00:32.054 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-19T01:00:32.093 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-19T01:00:32.377 INFO:teuthology.orchestra.run.smithi110.stdout:Fetched 29.2 kB in 4s (8,091 B/s) 2024-04-19T01:00:32.648 INFO:teuthology.orchestra.run.smithi155.stdout:Reading package lists... 2024-04-19T01:00:32.665 DEBUG:teuthology.orchestra.run.smithi155:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-3154-g8b4ab94f-1jammy cephadm=19.0.0-3154-g8b4ab94f-1jammy ceph-mds=19.0.0-3154-g8b4ab94f-1jammy ceph-mgr=19.0.0-3154-g8b4ab94f-1jammy ceph-common=19.0.0-3154-g8b4ab94f-1jammy ceph-fuse=19.0.0-3154-g8b4ab94f-1jammy ceph-test=19.0.0-3154-g8b4ab94f-1jammy ceph-volume=19.0.0-3154-g8b4ab94f-1jammy radosgw=19.0.0-3154-g8b4ab94f-1jammy python3-rados=19.0.0-3154-g8b4ab94f-1jammy python3-rgw=19.0.0-3154-g8b4ab94f-1jammy python3-cephfs=19.0.0-3154-g8b4ab94f-1jammy python3-rbd=19.0.0-3154-g8b4ab94f-1jammy libcephfs2=19.0.0-3154-g8b4ab94f-1jammy libcephfs-dev=19.0.0-3154-g8b4ab94f-1jammy librados2=19.0.0-3154-g8b4ab94f-1jammy librbd1=19.0.0-3154-g8b4ab94f-1jammy rbd-fuse=19.0.0-3154-g8b4ab94f-1jammy 2024-04-19T01:00:32.726 INFO:teuthology.orchestra.run.smithi155.stdout:Reading package lists... 2024-04-19T01:00:32.821 INFO:teuthology.orchestra.run.smithi118.stdout:Reading package lists... 2024-04-19T01:00:32.838 DEBUG:teuthology.orchestra.run.smithi118:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-3154-g8b4ab94f-1jammy cephadm=19.0.0-3154-g8b4ab94f-1jammy ceph-mds=19.0.0-3154-g8b4ab94f-1jammy ceph-mgr=19.0.0-3154-g8b4ab94f-1jammy ceph-common=19.0.0-3154-g8b4ab94f-1jammy ceph-fuse=19.0.0-3154-g8b4ab94f-1jammy ceph-test=19.0.0-3154-g8b4ab94f-1jammy ceph-volume=19.0.0-3154-g8b4ab94f-1jammy radosgw=19.0.0-3154-g8b4ab94f-1jammy python3-rados=19.0.0-3154-g8b4ab94f-1jammy python3-rgw=19.0.0-3154-g8b4ab94f-1jammy python3-cephfs=19.0.0-3154-g8b4ab94f-1jammy python3-rbd=19.0.0-3154-g8b4ab94f-1jammy libcephfs2=19.0.0-3154-g8b4ab94f-1jammy libcephfs-dev=19.0.0-3154-g8b4ab94f-1jammy librados2=19.0.0-3154-g8b4ab94f-1jammy librbd1=19.0.0-3154-g8b4ab94f-1jammy rbd-fuse=19.0.0-3154-g8b4ab94f-1jammy 2024-04-19T01:00:32.898 INFO:teuthology.orchestra.run.smithi118.stdout:Reading package lists... 2024-04-19T01:00:32.911 INFO:teuthology.orchestra.run.smithi155.stdout:Building dependency tree... 2024-04-19T01:00:32.912 INFO:teuthology.orchestra.run.smithi155.stdout:Reading state information... 2024-04-19T01:00:33.066 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:9 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-19T01:00:33.084 INFO:teuthology.orchestra.run.smithi155.stdout:The following packages were automatically installed and are no longer required: 2024-04-19T01:00:33.084 INFO:teuthology.orchestra.run.smithi155.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-19T01:00:33.084 INFO:teuthology.orchestra.run.smithi155.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-19T01:00:33.084 INFO:teuthology.orchestra.run.smithi155.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-19T01:00:33.085 INFO:teuthology.orchestra.run.smithi155.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-19T01:00:33.085 INFO:teuthology.orchestra.run.smithi155.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-19T01:00:33.085 INFO:teuthology.orchestra.run.smithi155.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-19T01:00:33.085 INFO:teuthology.orchestra.run.smithi155.stdout:The following additional packages will be installed: 2024-04-19T01:00:33.086 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-19T01:00:33.086 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-19T01:00:33.086 INFO:teuthology.orchestra.run.smithi155.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-19T01:00:33.086 INFO:teuthology.orchestra.run.smithi155.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-19T01:00:33.087 INFO:teuthology.orchestra.run.smithi155.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-19T01:00:33.087 INFO:teuthology.orchestra.run.smithi155.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-19T01:00:33.087 INFO:teuthology.orchestra.run.smithi155.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-19T01:00:33.087 INFO:teuthology.orchestra.run.smithi155.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-19T01:00:33.087 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-19T01:00:33.087 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-19T01:00:33.087 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-19T01:00:33.087 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-19T01:00:33.087 INFO:teuthology.orchestra.run.smithi155.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-19T01:00:33.088 INFO:teuthology.orchestra.run.smithi155.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-19T01:00:33.088 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-19T01:00:33.088 INFO:teuthology.orchestra.run.smithi155.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-19T01:00:33.088 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-19T01:00:33.088 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-19T01:00:33.088 INFO:teuthology.orchestra.run.smithi155.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-19T01:00:33.088 INFO:teuthology.orchestra.run.smithi155.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-19T01:00:33.088 INFO:teuthology.orchestra.run.smithi155.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-19T01:00:33.088 INFO:teuthology.orchestra.run.smithi155.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-19T01:00:33.088 INFO:teuthology.orchestra.run.smithi155.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-19T01:00:33.088 INFO:teuthology.orchestra.run.smithi155.stdout: xmlstarlet zip 2024-04-19T01:00:33.088 INFO:teuthology.orchestra.run.smithi155.stdout:Suggested packages: 2024-04-19T01:00:33.089 INFO:teuthology.orchestra.run.smithi155.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-19T01:00:33.089 INFO:teuthology.orchestra.run.smithi155.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-19T01:00:33.089 INFO:teuthology.orchestra.run.smithi155.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-19T01:00:33.089 INFO:teuthology.orchestra.run.smithi155.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-19T01:00:33.089 INFO:teuthology.orchestra.run.smithi155.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-19T01:00:33.089 INFO:teuthology.orchestra.run.smithi155.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-19T01:00:33.089 INFO:teuthology.orchestra.run.smithi155.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-19T01:00:33.089 INFO:teuthology.orchestra.run.smithi155.stdout:Recommended packages: 2024-04-19T01:00:33.089 INFO:teuthology.orchestra.run.smithi155.stdout: btrfs-tools 2024-04-19T01:00:33.097 INFO:teuthology.orchestra.run.smithi118.stdout:Building dependency tree... 2024-04-19T01:00:33.098 INFO:teuthology.orchestra.run.smithi118.stdout:Reading state information... 2024-04-19T01:00:33.190 INFO:teuthology.orchestra.run.smithi155.stdout:The following NEW packages will be installed: 2024-04-19T01:00:33.190 INFO:teuthology.orchestra.run.smithi155.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-19T01:00:33.190 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-19T01:00:33.190 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-19T01:00:33.191 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-19T01:00:33.191 INFO:teuthology.orchestra.run.smithi155.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-19T01:00:33.191 INFO:teuthology.orchestra.run.smithi155.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-19T01:00:33.191 INFO:teuthology.orchestra.run.smithi155.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-19T01:00:33.191 INFO:teuthology.orchestra.run.smithi155.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-19T01:00:33.191 INFO:teuthology.orchestra.run.smithi155.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-19T01:00:33.192 INFO:teuthology.orchestra.run.smithi155.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-19T01:00:33.192 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-19T01:00:33.192 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-19T01:00:33.192 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-19T01:00:33.192 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-19T01:00:33.192 INFO:teuthology.orchestra.run.smithi155.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-19T01:00:33.192 INFO:teuthology.orchestra.run.smithi155.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-19T01:00:33.192 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-19T01:00:33.192 INFO:teuthology.orchestra.run.smithi155.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-19T01:00:33.192 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-19T01:00:33.192 INFO:teuthology.orchestra.run.smithi155.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-19T01:00:33.193 INFO:teuthology.orchestra.run.smithi155.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-19T01:00:33.193 INFO:teuthology.orchestra.run.smithi155.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-19T01:00:33.193 INFO:teuthology.orchestra.run.smithi155.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-19T01:00:33.193 INFO:teuthology.orchestra.run.smithi155.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-19T01:00:33.193 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-19T01:00:33.193 INFO:teuthology.orchestra.run.smithi155.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-19T01:00:33.239 INFO:teuthology.orchestra.run.smithi033.stdout:Fetched 29.2 kB in 4s (6,669 B/s) 2024-04-19T01:00:33.291 INFO:teuthology.orchestra.run.smithi118.stdout:The following packages were automatically installed and are no longer required: 2024-04-19T01:00:33.291 INFO:teuthology.orchestra.run.smithi118.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-19T01:00:33.291 INFO:teuthology.orchestra.run.smithi118.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-19T01:00:33.291 INFO:teuthology.orchestra.run.smithi118.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-19T01:00:33.291 INFO:teuthology.orchestra.run.smithi118.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-19T01:00:33.292 INFO:teuthology.orchestra.run.smithi118.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-19T01:00:33.292 INFO:teuthology.orchestra.run.smithi118.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-19T01:00:33.292 INFO:teuthology.orchestra.run.smithi118.stdout:The following additional packages will be installed: 2024-04-19T01:00:33.292 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-19T01:00:33.292 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-19T01:00:33.294 INFO:teuthology.orchestra.run.smithi118.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-19T01:00:33.294 INFO:teuthology.orchestra.run.smithi118.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-19T01:00:33.294 INFO:teuthology.orchestra.run.smithi118.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-19T01:00:33.294 INFO:teuthology.orchestra.run.smithi118.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-19T01:00:33.294 INFO:teuthology.orchestra.run.smithi118.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-19T01:00:33.294 INFO:teuthology.orchestra.run.smithi118.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-19T01:00:33.294 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-19T01:00:33.294 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-19T01:00:33.294 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-19T01:00:33.294 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-19T01:00:33.294 INFO:teuthology.orchestra.run.smithi118.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-19T01:00:33.295 INFO:teuthology.orchestra.run.smithi118.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-19T01:00:33.295 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-19T01:00:33.295 INFO:teuthology.orchestra.run.smithi118.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-19T01:00:33.295 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-19T01:00:33.295 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-19T01:00:33.295 INFO:teuthology.orchestra.run.smithi118.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-19T01:00:33.295 INFO:teuthology.orchestra.run.smithi118.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-19T01:00:33.295 INFO:teuthology.orchestra.run.smithi118.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-19T01:00:33.295 INFO:teuthology.orchestra.run.smithi118.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-19T01:00:33.295 INFO:teuthology.orchestra.run.smithi118.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-19T01:00:33.295 INFO:teuthology.orchestra.run.smithi118.stdout: xmlstarlet zip 2024-04-19T01:00:33.295 INFO:teuthology.orchestra.run.smithi118.stdout:Suggested packages: 2024-04-19T01:00:33.296 INFO:teuthology.orchestra.run.smithi118.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-19T01:00:33.296 INFO:teuthology.orchestra.run.smithi118.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-19T01:00:33.296 INFO:teuthology.orchestra.run.smithi118.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-19T01:00:33.296 INFO:teuthology.orchestra.run.smithi118.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-19T01:00:33.296 INFO:teuthology.orchestra.run.smithi118.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-19T01:00:33.296 INFO:teuthology.orchestra.run.smithi118.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-19T01:00:33.296 INFO:teuthology.orchestra.run.smithi118.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-19T01:00:33.296 INFO:teuthology.orchestra.run.smithi118.stdout:Recommended packages: 2024-04-19T01:00:33.296 INFO:teuthology.orchestra.run.smithi118.stdout: btrfs-tools 2024-04-19T01:00:33.332 INFO:teuthology.orchestra.run.smithi155.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-19T01:00:33.332 INFO:teuthology.orchestra.run.smithi155.stdout:Need to get 181 MB of archives. 2024-04-19T01:00:33.332 INFO:teuthology.orchestra.run.smithi155.stdout:After this operation, 829 MB of additional disk space will be used. 2024-04-19T01:00:33.332 INFO:teuthology.orchestra.run.smithi155.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-3154-g8b4ab94f-1jammy [3,555 kB] 2024-04-19T01:00:33.357 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2024-04-19T01:00:33.374 DEBUG:teuthology.orchestra.run.smithi110:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-3154-g8b4ab94f-1jammy cephadm=19.0.0-3154-g8b4ab94f-1jammy ceph-mds=19.0.0-3154-g8b4ab94f-1jammy ceph-mgr=19.0.0-3154-g8b4ab94f-1jammy ceph-common=19.0.0-3154-g8b4ab94f-1jammy ceph-fuse=19.0.0-3154-g8b4ab94f-1jammy ceph-test=19.0.0-3154-g8b4ab94f-1jammy ceph-volume=19.0.0-3154-g8b4ab94f-1jammy radosgw=19.0.0-3154-g8b4ab94f-1jammy python3-rados=19.0.0-3154-g8b4ab94f-1jammy python3-rgw=19.0.0-3154-g8b4ab94f-1jammy python3-cephfs=19.0.0-3154-g8b4ab94f-1jammy python3-rbd=19.0.0-3154-g8b4ab94f-1jammy libcephfs2=19.0.0-3154-g8b4ab94f-1jammy libcephfs-dev=19.0.0-3154-g8b4ab94f-1jammy librados2=19.0.0-3154-g8b4ab94f-1jammy librbd1=19.0.0-3154-g8b4ab94f-1jammy rbd-fuse=19.0.0-3154-g8b4ab94f-1jammy 2024-04-19T01:00:33.380 INFO:teuthology.orchestra.run.smithi155.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-19T01:00:33.398 INFO:teuthology.orchestra.run.smithi118.stdout:The following NEW packages will be installed: 2024-04-19T01:00:33.398 INFO:teuthology.orchestra.run.smithi118.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-19T01:00:33.398 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-19T01:00:33.398 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-19T01:00:33.399 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-19T01:00:33.399 INFO:teuthology.orchestra.run.smithi118.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-19T01:00:33.400 INFO:teuthology.orchestra.run.smithi118.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-19T01:00:33.401 INFO:teuthology.orchestra.run.smithi118.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-19T01:00:33.401 INFO:teuthology.orchestra.run.smithi118.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-19T01:00:33.401 INFO:teuthology.orchestra.run.smithi118.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-19T01:00:33.401 INFO:teuthology.orchestra.run.smithi118.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-19T01:00:33.401 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-19T01:00:33.401 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-19T01:00:33.401 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-19T01:00:33.401 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-19T01:00:33.401 INFO:teuthology.orchestra.run.smithi118.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-19T01:00:33.402 INFO:teuthology.orchestra.run.smithi118.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-19T01:00:33.402 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-19T01:00:33.402 INFO:teuthology.orchestra.run.smithi118.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-19T01:00:33.402 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-19T01:00:33.402 INFO:teuthology.orchestra.run.smithi118.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-19T01:00:33.402 INFO:teuthology.orchestra.run.smithi118.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-19T01:00:33.402 INFO:teuthology.orchestra.run.smithi118.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-19T01:00:33.402 INFO:teuthology.orchestra.run.smithi118.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-19T01:00:33.402 INFO:teuthology.orchestra.run.smithi118.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-19T01:00:33.402 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-19T01:00:33.402 INFO:teuthology.orchestra.run.smithi118.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-19T01:00:33.435 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2024-04-19T01:00:33.540 INFO:teuthology.orchestra.run.smithi118.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-19T01:00:33.540 INFO:teuthology.orchestra.run.smithi118.stdout:Need to get 181 MB of archives. 2024-04-19T01:00:33.540 INFO:teuthology.orchestra.run.smithi118.stdout:After this operation, 829 MB of additional disk space will be used. 2024-04-19T01:00:33.540 INFO:teuthology.orchestra.run.smithi118.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-3154-g8b4ab94f-1jammy [3,555 kB] 2024-04-19T01:00:33.569 INFO:teuthology.orchestra.run.smithi155.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-3154-g8b4ab94f-1jammy [3,224 kB] 2024-04-19T01:00:33.629 INFO:teuthology.orchestra.run.smithi110.stdout:Building dependency tree... 2024-04-19T01:00:33.630 INFO:teuthology.orchestra.run.smithi110.stdout:Reading state information... 2024-04-19T01:00:33.724 INFO:teuthology.orchestra.run.smithi155.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-3154-g8b4ab94f-1jammy [712 kB] 2024-04-19T01:00:33.764 INFO:teuthology.orchestra.run.smithi155.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-3154-g8b4ab94f-1jammy [356 kB] 2024-04-19T01:00:33.775 INFO:teuthology.orchestra.run.smithi155.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-3154-g8b4ab94f-1jammy [32.8 kB] 2024-04-19T01:00:33.777 INFO:teuthology.orchestra.run.smithi155.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-3154-g8b4ab94f-1jammy [180 kB] 2024-04-19T01:00:33.782 INFO:teuthology.orchestra.run.smithi155.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-3154-g8b4ab94f-1jammy [66.9 kB] 2024-04-19T01:00:33.784 INFO:teuthology.orchestra.run.smithi155.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-3154-g8b4ab94f-1jammy [330 kB] 2024-04-19T01:00:33.808 INFO:teuthology.orchestra.run.smithi155.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-19T01:00:33.813 INFO:teuthology.orchestra.run.smithi155.stdout:Get:11 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-3154-g8b4ab94f-1jammy [6,894 kB] 2024-04-19T01:00:33.817 INFO:teuthology.orchestra.run.smithi110.stdout:The following packages were automatically installed and are no longer required: 2024-04-19T01:00:33.817 INFO:teuthology.orchestra.run.smithi110.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-19T01:00:33.817 INFO:teuthology.orchestra.run.smithi110.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-19T01:00:33.817 INFO:teuthology.orchestra.run.smithi110.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-19T01:00:33.818 INFO:teuthology.orchestra.run.smithi110.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-19T01:00:33.818 INFO:teuthology.orchestra.run.smithi110.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-19T01:00:33.818 INFO:teuthology.orchestra.run.smithi110.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-19T01:00:33.820 INFO:teuthology.orchestra.run.smithi110.stdout:The following additional packages will be installed: 2024-04-19T01:00:33.820 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-19T01:00:33.820 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-19T01:00:33.820 INFO:teuthology.orchestra.run.smithi110.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-19T01:00:33.820 INFO:teuthology.orchestra.run.smithi110.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-19T01:00:33.820 INFO:teuthology.orchestra.run.smithi110.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-19T01:00:33.820 INFO:teuthology.orchestra.run.smithi110.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-19T01:00:33.820 INFO:teuthology.orchestra.run.smithi110.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-19T01:00:33.820 INFO:teuthology.orchestra.run.smithi110.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-19T01:00:33.820 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-19T01:00:33.820 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-19T01:00:33.820 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-19T01:00:33.821 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-19T01:00:33.821 INFO:teuthology.orchestra.run.smithi110.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-19T01:00:33.821 INFO:teuthology.orchestra.run.smithi110.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-19T01:00:33.821 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-19T01:00:33.821 INFO:teuthology.orchestra.run.smithi110.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-19T01:00:33.821 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-19T01:00:33.821 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-19T01:00:33.821 INFO:teuthology.orchestra.run.smithi110.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-19T01:00:33.821 INFO:teuthology.orchestra.run.smithi110.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-19T01:00:33.821 INFO:teuthology.orchestra.run.smithi110.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-19T01:00:33.821 INFO:teuthology.orchestra.run.smithi110.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-19T01:00:33.821 INFO:teuthology.orchestra.run.smithi110.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-19T01:00:33.821 INFO:teuthology.orchestra.run.smithi110.stdout: xmlstarlet zip 2024-04-19T01:00:33.822 INFO:teuthology.orchestra.run.smithi155.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-19T01:00:33.822 INFO:teuthology.orchestra.run.smithi110.stdout:Suggested packages: 2024-04-19T01:00:33.822 INFO:teuthology.orchestra.run.smithi110.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-19T01:00:33.822 INFO:teuthology.orchestra.run.smithi110.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-19T01:00:33.822 INFO:teuthology.orchestra.run.smithi110.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-19T01:00:33.822 INFO:teuthology.orchestra.run.smithi110.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-19T01:00:33.823 INFO:teuthology.orchestra.run.smithi110.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-19T01:00:33.823 INFO:teuthology.orchestra.run.smithi110.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-19T01:00:33.823 INFO:teuthology.orchestra.run.smithi110.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-19T01:00:33.823 INFO:teuthology.orchestra.run.smithi110.stdout:Recommended packages: 2024-04-19T01:00:33.823 INFO:teuthology.orchestra.run.smithi110.stdout: btrfs-tools 2024-04-19T01:00:33.913 INFO:teuthology.orchestra.run.smithi118.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-3154-g8b4ab94f-1jammy [3,224 kB] 2024-04-19T01:00:33.919 INFO:teuthology.orchestra.run.smithi110.stdout:The following NEW packages will be installed: 2024-04-19T01:00:33.920 INFO:teuthology.orchestra.run.smithi110.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-19T01:00:33.920 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-19T01:00:33.920 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-19T01:00:33.921 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-19T01:00:33.921 INFO:teuthology.orchestra.run.smithi110.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-19T01:00:33.921 INFO:teuthology.orchestra.run.smithi110.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-19T01:00:33.921 INFO:teuthology.orchestra.run.smithi110.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-19T01:00:33.921 INFO:teuthology.orchestra.run.smithi110.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-19T01:00:33.922 INFO:teuthology.orchestra.run.smithi110.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-19T01:00:33.922 INFO:teuthology.orchestra.run.smithi110.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-19T01:00:33.922 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-19T01:00:33.922 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-19T01:00:33.922 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-19T01:00:33.922 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-19T01:00:33.922 INFO:teuthology.orchestra.run.smithi110.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-19T01:00:33.922 INFO:teuthology.orchestra.run.smithi110.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-19T01:00:33.922 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-19T01:00:33.922 INFO:teuthology.orchestra.run.smithi110.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-19T01:00:33.923 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-19T01:00:33.923 INFO:teuthology.orchestra.run.smithi110.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-19T01:00:33.923 INFO:teuthology.orchestra.run.smithi110.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-19T01:00:33.923 INFO:teuthology.orchestra.run.smithi110.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-19T01:00:33.923 INFO:teuthology.orchestra.run.smithi110.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-19T01:00:33.923 INFO:teuthology.orchestra.run.smithi110.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-19T01:00:33.923 INFO:teuthology.orchestra.run.smithi110.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-19T01:00:33.923 INFO:teuthology.orchestra.run.smithi110.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-19T01:00:34.007 INFO:teuthology.orchestra.run.smithi155.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-19T01:00:34.154 INFO:teuthology.orchestra.run.smithi118.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-3154-g8b4ab94f-1jammy [712 kB] 2024-04-19T01:00:34.173 INFO:teuthology.orchestra.run.smithi155.stdout:Get:14 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-3154-g8b4ab94f-1jammy [112 kB] 2024-04-19T01:00:34.176 INFO:teuthology.orchestra.run.smithi155.stdout:Get:15 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-3154-g8b4ab94f-1jammy [466 kB] 2024-04-19T01:00:34.191 INFO:teuthology.orchestra.run.smithi155.stdout:Get:16 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-3154-g8b4ab94f-1jammy [25.8 MB] 2024-04-19T01:00:34.197 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2024-04-19T01:00:34.213 DEBUG:teuthology.orchestra.run.smithi033:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-3154-g8b4ab94f-1jammy cephadm=19.0.0-3154-g8b4ab94f-1jammy ceph-mds=19.0.0-3154-g8b4ab94f-1jammy ceph-mgr=19.0.0-3154-g8b4ab94f-1jammy ceph-common=19.0.0-3154-g8b4ab94f-1jammy ceph-fuse=19.0.0-3154-g8b4ab94f-1jammy ceph-test=19.0.0-3154-g8b4ab94f-1jammy ceph-volume=19.0.0-3154-g8b4ab94f-1jammy radosgw=19.0.0-3154-g8b4ab94f-1jammy python3-rados=19.0.0-3154-g8b4ab94f-1jammy python3-rgw=19.0.0-3154-g8b4ab94f-1jammy python3-cephfs=19.0.0-3154-g8b4ab94f-1jammy python3-rbd=19.0.0-3154-g8b4ab94f-1jammy libcephfs2=19.0.0-3154-g8b4ab94f-1jammy libcephfs-dev=19.0.0-3154-g8b4ab94f-1jammy librados2=19.0.0-3154-g8b4ab94f-1jammy librbd1=19.0.0-3154-g8b4ab94f-1jammy rbd-fuse=19.0.0-3154-g8b4ab94f-1jammy 2024-04-19T01:00:34.214 INFO:teuthology.orchestra.run.smithi118.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-3154-g8b4ab94f-1jammy [356 kB] 2024-04-19T01:00:34.255 INFO:teuthology.orchestra.run.smithi110.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-19T01:00:34.255 INFO:teuthology.orchestra.run.smithi110.stdout:Need to get 181 MB of archives. 2024-04-19T01:00:34.255 INFO:teuthology.orchestra.run.smithi110.stdout:After this operation, 829 MB of additional disk space will be used. 2024-04-19T01:00:34.255 INFO:teuthology.orchestra.run.smithi110.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-3154-g8b4ab94f-1jammy [3,555 kB] 2024-04-19T01:00:34.265 INFO:teuthology.orchestra.run.smithi118.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-3154-g8b4ab94f-1jammy [32.8 kB] 2024-04-19T01:00:34.266 INFO:teuthology.orchestra.run.smithi118.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-3154-g8b4ab94f-1jammy [180 kB] 2024-04-19T01:00:34.272 INFO:teuthology.orchestra.run.smithi118.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-3154-g8b4ab94f-1jammy [66.9 kB] 2024-04-19T01:00:34.276 INFO:teuthology.orchestra.run.smithi118.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-3154-g8b4ab94f-1jammy [330 kB] 2024-04-19T01:00:34.276 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2024-04-19T01:00:34.278 INFO:teuthology.orchestra.run.smithi155.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-19T01:00:34.286 INFO:teuthology.orchestra.run.smithi118.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-3154-g8b4ab94f-1jammy [6,894 kB] 2024-04-19T01:00:34.288 INFO:teuthology.orchestra.run.smithi155.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-19T01:00:34.318 INFO:teuthology.orchestra.run.smithi155.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-19T01:00:34.329 INFO:teuthology.orchestra.run.smithi155.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-19T01:00:34.330 INFO:teuthology.orchestra.run.smithi155.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-19T01:00:34.332 INFO:teuthology.orchestra.run.smithi155.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-19T01:00:34.352 INFO:teuthology.orchestra.run.smithi155.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-19T01:00:34.480 INFO:teuthology.orchestra.run.smithi033.stdout:Building dependency tree... 2024-04-19T01:00:34.480 INFO:teuthology.orchestra.run.smithi033.stdout:Reading state information... 2024-04-19T01:00:34.537 INFO:teuthology.orchestra.run.smithi155.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-19T01:00:34.683 INFO:teuthology.orchestra.run.smithi033.stdout:The following packages were automatically installed and are no longer required: 2024-04-19T01:00:34.683 INFO:teuthology.orchestra.run.smithi033.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-19T01:00:34.683 INFO:teuthology.orchestra.run.smithi033.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-19T01:00:34.684 INFO:teuthology.orchestra.run.smithi033.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-19T01:00:34.684 INFO:teuthology.orchestra.run.smithi033.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-19T01:00:34.684 INFO:teuthology.orchestra.run.smithi033.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-19T01:00:34.684 INFO:teuthology.orchestra.run.smithi033.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-19T01:00:34.684 INFO:teuthology.orchestra.run.smithi033.stdout:The following additional packages will be installed: 2024-04-19T01:00:34.685 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-19T01:00:34.686 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-19T01:00:34.686 INFO:teuthology.orchestra.run.smithi033.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-19T01:00:34.686 INFO:teuthology.orchestra.run.smithi033.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-19T01:00:34.687 INFO:teuthology.orchestra.run.smithi033.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-19T01:00:34.687 INFO:teuthology.orchestra.run.smithi033.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-19T01:00:34.687 INFO:teuthology.orchestra.run.smithi033.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-19T01:00:34.687 INFO:teuthology.orchestra.run.smithi033.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-19T01:00:34.687 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-19T01:00:34.687 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-19T01:00:34.687 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-19T01:00:34.687 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-19T01:00:34.687 INFO:teuthology.orchestra.run.smithi033.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-19T01:00:34.687 INFO:teuthology.orchestra.run.smithi033.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-19T01:00:34.687 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-19T01:00:34.687 INFO:teuthology.orchestra.run.smithi033.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-19T01:00:34.687 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-19T01:00:34.687 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-19T01:00:34.688 INFO:teuthology.orchestra.run.smithi033.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-19T01:00:34.688 INFO:teuthology.orchestra.run.smithi033.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-19T01:00:34.688 INFO:teuthology.orchestra.run.smithi033.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-19T01:00:34.688 INFO:teuthology.orchestra.run.smithi033.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-19T01:00:34.688 INFO:teuthology.orchestra.run.smithi033.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-19T01:00:34.688 INFO:teuthology.orchestra.run.smithi033.stdout: xmlstarlet zip 2024-04-19T01:00:34.688 INFO:teuthology.orchestra.run.smithi033.stdout:Suggested packages: 2024-04-19T01:00:34.688 INFO:teuthology.orchestra.run.smithi033.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-19T01:00:34.688 INFO:teuthology.orchestra.run.smithi033.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-19T01:00:34.688 INFO:teuthology.orchestra.run.smithi033.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-19T01:00:34.689 INFO:teuthology.orchestra.run.smithi033.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-19T01:00:34.689 INFO:teuthology.orchestra.run.smithi033.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-19T01:00:34.689 INFO:teuthology.orchestra.run.smithi033.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-19T01:00:34.689 INFO:teuthology.orchestra.run.smithi033.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-19T01:00:34.689 INFO:teuthology.orchestra.run.smithi033.stdout:Recommended packages: 2024-04-19T01:00:34.689 INFO:teuthology.orchestra.run.smithi033.stdout: btrfs-tools 2024-04-19T01:00:34.794 INFO:teuthology.orchestra.run.smithi033.stdout:The following NEW packages will be installed: 2024-04-19T01:00:34.794 INFO:teuthology.orchestra.run.smithi033.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-19T01:00:34.794 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-19T01:00:34.794 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-19T01:00:34.794 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-19T01:00:34.794 INFO:teuthology.orchestra.run.smithi033.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-19T01:00:34.794 INFO:teuthology.orchestra.run.smithi033.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-19T01:00:34.796 INFO:teuthology.orchestra.run.smithi033.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-19T01:00:34.796 INFO:teuthology.orchestra.run.smithi033.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-19T01:00:34.796 INFO:teuthology.orchestra.run.smithi033.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-19T01:00:34.796 INFO:teuthology.orchestra.run.smithi033.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-19T01:00:34.796 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-19T01:00:34.796 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-19T01:00:34.796 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-19T01:00:34.796 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-19T01:00:34.796 INFO:teuthology.orchestra.run.smithi033.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-19T01:00:34.796 INFO:teuthology.orchestra.run.smithi033.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-19T01:00:34.796 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-19T01:00:34.797 INFO:teuthology.orchestra.run.smithi033.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-19T01:00:34.797 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-19T01:00:34.797 INFO:teuthology.orchestra.run.smithi033.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-19T01:00:34.797 INFO:teuthology.orchestra.run.smithi033.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-19T01:00:34.797 INFO:teuthology.orchestra.run.smithi033.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-19T01:00:34.797 INFO:teuthology.orchestra.run.smithi033.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-19T01:00:34.797 INFO:teuthology.orchestra.run.smithi033.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-19T01:00:34.797 INFO:teuthology.orchestra.run.smithi033.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-19T01:00:34.797 INFO:teuthology.orchestra.run.smithi033.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-19T01:00:34.889 INFO:teuthology.orchestra.run.smithi118.stdout:Get:10 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-3154-g8b4ab94f-1jammy [112 kB] 2024-04-19T01:00:34.894 INFO:teuthology.orchestra.run.smithi118.stdout:Get:11 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-3154-g8b4ab94f-1jammy [466 kB] 2024-04-19T01:00:34.943 INFO:teuthology.orchestra.run.smithi118.stdout:Get:12 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-3154-g8b4ab94f-1jammy [25.8 MB] 2024-04-19T01:00:34.980 INFO:teuthology.orchestra.run.smithi155.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-19T01:00:35.016 INFO:teuthology.orchestra.run.smithi155.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-19T01:00:35.018 INFO:teuthology.orchestra.run.smithi155.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-19T01:00:35.110 INFO:teuthology.orchestra.run.smithi155.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-19T01:00:35.146 INFO:teuthology.orchestra.run.smithi033.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-19T01:00:35.146 INFO:teuthology.orchestra.run.smithi033.stdout:Need to get 181 MB of archives. 2024-04-19T01:00:35.146 INFO:teuthology.orchestra.run.smithi033.stdout:After this operation, 829 MB of additional disk space will be used. 2024-04-19T01:00:35.146 INFO:teuthology.orchestra.run.smithi033.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-3154-g8b4ab94f-1jammy [3,555 kB] 2024-04-19T01:00:35.496 INFO:teuthology.orchestra.run.smithi155.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-19T01:00:35.903 INFO:teuthology.orchestra.run.smithi155.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-19T01:00:35.905 INFO:teuthology.orchestra.run.smithi155.stdout:Get:31 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-3154-g8b4ab94f-1jammy [5,095 kB] 2024-04-19T01:00:35.950 INFO:teuthology.orchestra.run.smithi118.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-19T01:00:35.988 INFO:teuthology.orchestra.run.smithi155.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-19T01:00:35.993 INFO:teuthology.orchestra.run.smithi155.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-19T01:00:36.071 INFO:teuthology.orchestra.run.smithi118.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-19T01:00:36.075 INFO:teuthology.orchestra.run.smithi118.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-19T01:00:36.101 INFO:teuthology.orchestra.run.smithi118.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-19T01:00:36.102 INFO:teuthology.orchestra.run.smithi033.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-3154-g8b4ab94f-1jammy [3,224 kB] 2024-04-19T01:00:36.121 INFO:teuthology.orchestra.run.smithi155.stdout:Get:34 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-3154-g8b4ab94f-1jammy [244 kB] 2024-04-19T01:00:36.127 INFO:teuthology.orchestra.run.smithi155.stdout:Get:35 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-3154-g8b4ab94f-1jammy [124 kB] 2024-04-19T01:00:36.134 INFO:teuthology.orchestra.run.smithi155.stdout:Get:36 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-3154-g8b4ab94f-1jammy [1,526 kB] 2024-04-19T01:00:36.176 INFO:teuthology.orchestra.run.smithi155.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-19T01:00:36.181 INFO:teuthology.orchestra.run.smithi155.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-19T01:00:36.193 INFO:teuthology.orchestra.run.smithi118.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-19T01:00:36.198 INFO:teuthology.orchestra.run.smithi118.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-19T01:00:36.214 INFO:teuthology.orchestra.run.smithi118.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-19T01:00:36.219 INFO:teuthology.orchestra.run.smithi118.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-19T01:00:36.219 INFO:teuthology.orchestra.run.smithi118.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-19T01:00:36.220 INFO:teuthology.orchestra.run.smithi118.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-19T01:00:36.230 INFO:teuthology.orchestra.run.smithi118.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-19T01:00:36.233 INFO:teuthology.orchestra.run.smithi118.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-19T01:00:36.235 INFO:teuthology.orchestra.run.smithi118.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-19T01:00:36.237 INFO:teuthology.orchestra.run.smithi118.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-19T01:00:36.246 INFO:teuthology.orchestra.run.smithi155.stdout:Get:39 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-3154-g8b4ab94f-1jammy [6,260 kB] 2024-04-19T01:00:36.260 INFO:teuthology.orchestra.run.smithi118.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-19T01:00:36.266 INFO:teuthology.orchestra.run.smithi118.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-19T01:00:36.269 INFO:teuthology.orchestra.run.smithi118.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-19T01:00:36.272 INFO:teuthology.orchestra.run.smithi118.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-19T01:00:36.273 INFO:teuthology.orchestra.run.smithi118.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-19T01:00:36.273 INFO:teuthology.orchestra.run.smithi118.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-19T01:00:36.287 INFO:teuthology.orchestra.run.smithi118.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-19T01:00:36.288 INFO:teuthology.orchestra.run.smithi118.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-19T01:00:36.288 INFO:teuthology.orchestra.run.smithi118.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-19T01:00:36.289 INFO:teuthology.orchestra.run.smithi118.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-19T01:00:36.308 INFO:teuthology.orchestra.run.smithi118.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-19T01:00:36.309 INFO:teuthology.orchestra.run.smithi118.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-19T01:00:36.310 INFO:teuthology.orchestra.run.smithi118.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-19T01:00:36.314 INFO:teuthology.orchestra.run.smithi118.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-19T01:00:36.315 INFO:teuthology.orchestra.run.smithi118.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-19T01:00:36.315 INFO:teuthology.orchestra.run.smithi118.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-19T01:00:36.329 INFO:teuthology.orchestra.run.smithi118.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-19T01:00:36.331 INFO:teuthology.orchestra.run.smithi118.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-19T01:00:36.331 INFO:teuthology.orchestra.run.smithi118.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-19T01:00:36.331 INFO:teuthology.orchestra.run.smithi118.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-19T01:00:36.353 INFO:teuthology.orchestra.run.smithi155.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-19T01:00:36.356 INFO:teuthology.orchestra.run.smithi118.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-19T01:00:36.357 INFO:teuthology.orchestra.run.smithi118.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-19T01:00:36.359 INFO:teuthology.orchestra.run.smithi118.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-19T01:00:36.366 INFO:teuthology.orchestra.run.smithi118.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-19T01:00:36.366 INFO:teuthology.orchestra.run.smithi118.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-19T01:00:36.367 INFO:teuthology.orchestra.run.smithi118.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-19T01:00:36.388 INFO:teuthology.orchestra.run.smithi118.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-19T01:00:36.390 INFO:teuthology.orchestra.run.smithi118.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-19T01:00:36.393 INFO:teuthology.orchestra.run.smithi118.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-19T01:00:36.395 INFO:teuthology.orchestra.run.smithi118.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-19T01:00:36.418 INFO:teuthology.orchestra.run.smithi118.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-19T01:00:36.418 INFO:teuthology.orchestra.run.smithi118.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-19T01:00:36.421 INFO:teuthology.orchestra.run.smithi118.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-19T01:00:36.422 INFO:teuthology.orchestra.run.smithi118.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-19T01:00:36.423 INFO:teuthology.orchestra.run.smithi118.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-19T01:00:36.427 INFO:teuthology.orchestra.run.smithi118.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-19T01:00:36.438 INFO:teuthology.orchestra.run.smithi155.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-19T01:00:36.467 INFO:teuthology.orchestra.run.smithi155.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-19T01:00:36.477 INFO:teuthology.orchestra.run.smithi110.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-19T01:00:36.497 INFO:teuthology.orchestra.run.smithi155.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-19T01:00:36.509 INFO:teuthology.orchestra.run.smithi118.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-19T01:00:36.510 INFO:teuthology.orchestra.run.smithi118.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-19T01:00:36.512 INFO:teuthology.orchestra.run.smithi118.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-19T01:00:36.513 INFO:teuthology.orchestra.run.smithi118.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-19T01:00:36.515 INFO:teuthology.orchestra.run.smithi118.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-19T01:00:36.531 INFO:teuthology.orchestra.run.smithi155.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-19T01:00:36.543 INFO:teuthology.orchestra.run.smithi118.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-19T01:00:36.549 INFO:teuthology.orchestra.run.smithi118.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-19T01:00:36.599 INFO:teuthology.orchestra.run.smithi118.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-19T01:00:36.623 INFO:teuthology.orchestra.run.smithi118.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-19T01:00:36.623 INFO:teuthology.orchestra.run.smithi118.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-19T01:00:36.624 INFO:teuthology.orchestra.run.smithi118.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-19T01:00:36.625 INFO:teuthology.orchestra.run.smithi118.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-19T01:00:36.625 INFO:teuthology.orchestra.run.smithi118.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-19T01:00:36.626 INFO:teuthology.orchestra.run.smithi118.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-19T01:00:36.627 INFO:teuthology.orchestra.run.smithi118.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-19T01:00:36.627 INFO:teuthology.orchestra.run.smithi118.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-19T01:00:36.637 INFO:teuthology.orchestra.run.smithi118.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-19T01:00:36.664 INFO:teuthology.orchestra.run.smithi118.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-19T01:00:36.666 INFO:teuthology.orchestra.run.smithi118.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-19T01:00:36.668 INFO:teuthology.orchestra.run.smithi118.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-19T01:00:36.673 INFO:teuthology.orchestra.run.smithi118.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-19T01:00:36.678 INFO:teuthology.orchestra.run.smithi118.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-19T01:00:36.680 INFO:teuthology.orchestra.run.smithi118.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-19T01:00:36.680 INFO:teuthology.orchestra.run.smithi118.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-19T01:00:36.688 INFO:teuthology.orchestra.run.smithi118.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-19T01:00:36.689 INFO:teuthology.orchestra.run.smithi118.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-19T01:00:36.704 INFO:teuthology.orchestra.run.smithi033.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-3154-g8b4ab94f-1jammy [712 kB] 2024-04-19T01:00:36.712 INFO:teuthology.orchestra.run.smithi110.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-19T01:00:36.713 INFO:teuthology.orchestra.run.smithi118.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-19T01:00:36.714 INFO:teuthology.orchestra.run.smithi118.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-19T01:00:36.715 INFO:teuthology.orchestra.run.smithi118.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-19T01:00:36.715 INFO:teuthology.orchestra.run.smithi118.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-19T01:00:36.717 INFO:teuthology.orchestra.run.smithi155.stdout:Get:45 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-3154-g8b4ab94f-1jammy [22.8 MB] 2024-04-19T01:00:36.719 INFO:teuthology.orchestra.run.smithi110.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-19T01:00:36.745 INFO:teuthology.orchestra.run.smithi118.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-19T01:00:36.746 INFO:teuthology.orchestra.run.smithi118.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-19T01:00:36.758 INFO:teuthology.orchestra.run.smithi118.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-19T01:00:36.758 INFO:teuthology.orchestra.run.smithi118.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-19T01:00:36.759 INFO:teuthology.orchestra.run.smithi118.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-19T01:00:36.762 INFO:teuthology.orchestra.run.smithi118.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-19T01:00:36.773 INFO:teuthology.orchestra.run.smithi110.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-19T01:00:36.779 INFO:teuthology.orchestra.run.smithi118.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-19T01:00:36.828 INFO:teuthology.orchestra.run.smithi033.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-3154-g8b4ab94f-1jammy [356 kB] 2024-04-19T01:00:36.859 INFO:teuthology.orchestra.run.smithi110.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-3154-g8b4ab94f-1jammy [3,224 kB] 2024-04-19T01:00:36.865 INFO:teuthology.orchestra.run.smithi155.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-19T01:00:36.882 INFO:teuthology.orchestra.run.smithi155.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-19T01:00:36.893 INFO:teuthology.orchestra.run.smithi033.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-3154-g8b4ab94f-1jammy [32.8 kB] 2024-04-19T01:00:36.895 INFO:teuthology.orchestra.run.smithi033.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-3154-g8b4ab94f-1jammy [180 kB] 2024-04-19T01:00:36.899 INFO:teuthology.orchestra.run.smithi033.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-3154-g8b4ab94f-1jammy [66.9 kB] 2024-04-19T01:00:36.899 INFO:teuthology.orchestra.run.smithi155.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-19T01:00:36.906 INFO:teuthology.orchestra.run.smithi110.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-19T01:00:36.914 INFO:teuthology.orchestra.run.smithi110.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-19T01:00:36.933 INFO:teuthology.orchestra.run.smithi155.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-19T01:00:36.934 INFO:teuthology.orchestra.run.smithi110.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-19T01:00:36.940 INFO:teuthology.orchestra.run.smithi110.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-19T01:00:36.941 INFO:teuthology.orchestra.run.smithi110.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-19T01:00:36.942 INFO:teuthology.orchestra.run.smithi110.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-19T01:00:36.952 INFO:teuthology.orchestra.run.smithi033.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-3154-g8b4ab94f-1jammy [330 kB] 2024-04-19T01:00:36.954 INFO:teuthology.orchestra.run.smithi110.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-19T01:00:36.957 INFO:teuthology.orchestra.run.smithi110.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-19T01:00:36.960 INFO:teuthology.orchestra.run.smithi110.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-19T01:00:36.962 INFO:teuthology.orchestra.run.smithi110.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-19T01:00:36.970 INFO:teuthology.orchestra.run.smithi110.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-19T01:00:36.972 INFO:teuthology.orchestra.run.smithi110.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-19T01:00:36.975 INFO:teuthology.orchestra.run.smithi110.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-19T01:00:36.978 INFO:teuthology.orchestra.run.smithi110.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-19T01:00:36.979 INFO:teuthology.orchestra.run.smithi110.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-19T01:00:36.992 INFO:teuthology.orchestra.run.smithi110.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-19T01:00:36.993 INFO:teuthology.orchestra.run.smithi110.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-19T01:00:36.993 INFO:teuthology.orchestra.run.smithi110.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-19T01:00:36.994 INFO:teuthology.orchestra.run.smithi110.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-19T01:00:36.994 INFO:teuthology.orchestra.run.smithi110.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-19T01:00:37.011 INFO:teuthology.orchestra.run.smithi110.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-19T01:00:37.012 INFO:teuthology.orchestra.run.smithi110.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-19T01:00:37.012 INFO:teuthology.orchestra.run.smithi110.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-19T01:00:37.015 INFO:teuthology.orchestra.run.smithi033.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-3154-g8b4ab94f-1jammy [6,894 kB] 2024-04-19T01:00:37.016 INFO:teuthology.orchestra.run.smithi110.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-19T01:00:37.016 INFO:teuthology.orchestra.run.smithi110.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-19T01:00:37.033 INFO:teuthology.orchestra.run.smithi110.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-19T01:00:37.033 INFO:teuthology.orchestra.run.smithi110.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-19T01:00:37.034 INFO:teuthology.orchestra.run.smithi110.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-19T01:00:37.035 INFO:teuthology.orchestra.run.smithi110.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-19T01:00:37.035 INFO:teuthology.orchestra.run.smithi110.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-19T01:00:37.041 INFO:teuthology.orchestra.run.smithi155.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-19T01:00:37.053 INFO:teuthology.orchestra.run.smithi110.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-19T01:00:37.054 INFO:teuthology.orchestra.run.smithi110.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-19T01:00:37.054 INFO:teuthology.orchestra.run.smithi110.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-19T01:00:37.061 INFO:teuthology.orchestra.run.smithi110.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-19T01:00:37.062 INFO:teuthology.orchestra.run.smithi110.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-19T01:00:37.075 INFO:teuthology.orchestra.run.smithi110.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-19T01:00:37.076 INFO:teuthology.orchestra.run.smithi110.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-19T01:00:37.078 INFO:teuthology.orchestra.run.smithi155.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-19T01:00:37.079 INFO:teuthology.orchestra.run.smithi110.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-19T01:00:37.081 INFO:teuthology.orchestra.run.smithi110.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-19T01:00:37.082 INFO:teuthology.orchestra.run.smithi110.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-19T01:00:37.099 INFO:teuthology.orchestra.run.smithi110.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-19T01:00:37.100 INFO:teuthology.orchestra.run.smithi110.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-19T01:00:37.103 INFO:teuthology.orchestra.run.smithi110.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-19T01:00:37.103 INFO:teuthology.orchestra.run.smithi110.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-19T01:00:37.104 INFO:teuthology.orchestra.run.smithi110.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-19T01:00:37.122 INFO:teuthology.orchestra.run.smithi110.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-19T01:00:37.235 INFO:teuthology.orchestra.run.smithi110.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-19T01:00:37.236 INFO:teuthology.orchestra.run.smithi110.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-19T01:00:37.238 INFO:teuthology.orchestra.run.smithi110.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-19T01:00:37.239 INFO:teuthology.orchestra.run.smithi110.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-19T01:00:37.240 INFO:teuthology.orchestra.run.smithi110.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-19T01:00:37.246 INFO:teuthology.orchestra.run.smithi155.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-19T01:00:37.265 INFO:teuthology.orchestra.run.smithi110.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-19T01:00:37.267 INFO:teuthology.orchestra.run.smithi110.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-19T01:00:37.268 INFO:teuthology.orchestra.run.smithi110.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-19T01:00:37.296 INFO:teuthology.orchestra.run.smithi110.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-19T01:00:37.296 INFO:teuthology.orchestra.run.smithi110.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-19T01:00:37.303 INFO:teuthology.orchestra.run.smithi110.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-19T01:00:37.304 INFO:teuthology.orchestra.run.smithi110.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-19T01:00:37.305 INFO:teuthology.orchestra.run.smithi110.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-19T01:00:37.306 INFO:teuthology.orchestra.run.smithi110.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-19T01:00:37.307 INFO:teuthology.orchestra.run.smithi110.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-19T01:00:37.308 INFO:teuthology.orchestra.run.smithi110.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-19T01:00:37.313 INFO:teuthology.orchestra.run.smithi110.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-19T01:00:37.330 INFO:teuthology.orchestra.run.smithi110.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-19T01:00:37.332 INFO:teuthology.orchestra.run.smithi110.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-19T01:00:37.333 INFO:teuthology.orchestra.run.smithi110.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-19T01:00:37.352 INFO:teuthology.orchestra.run.smithi110.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-19T01:00:37.356 INFO:teuthology.orchestra.run.smithi110.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-19T01:00:37.357 INFO:teuthology.orchestra.run.smithi110.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-19T01:00:37.358 INFO:teuthology.orchestra.run.smithi110.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-19T01:00:37.365 INFO:teuthology.orchestra.run.smithi110.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-19T01:00:37.365 INFO:teuthology.orchestra.run.smithi110.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-19T01:00:37.370 INFO:teuthology.orchestra.run.smithi110.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-19T01:00:37.381 INFO:teuthology.orchestra.run.smithi110.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-19T01:00:37.382 INFO:teuthology.orchestra.run.smithi110.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-19T01:00:37.382 INFO:teuthology.orchestra.run.smithi110.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-19T01:00:37.407 INFO:teuthology.orchestra.run.smithi110.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-19T01:00:37.408 INFO:teuthology.orchestra.run.smithi110.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-19T01:00:37.418 INFO:teuthology.orchestra.run.smithi110.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-19T01:00:37.419 INFO:teuthology.orchestra.run.smithi110.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-19T01:00:37.420 INFO:teuthology.orchestra.run.smithi110.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-19T01:00:37.423 INFO:teuthology.orchestra.run.smithi110.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-19T01:00:37.423 INFO:teuthology.orchestra.run.smithi110.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-19T01:00:37.555 INFO:teuthology.orchestra.run.smithi118.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-3154-g8b4ab94f-1jammy [5,095 kB] 2024-04-19T01:00:37.580 INFO:teuthology.orchestra.run.smithi155.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-19T01:00:37.612 INFO:teuthology.orchestra.run.smithi155.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-19T01:00:37.622 INFO:teuthology.orchestra.run.smithi155.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-19T01:00:37.658 INFO:teuthology.orchestra.run.smithi033.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-19T01:00:37.785 INFO:teuthology.orchestra.run.smithi033.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-19T01:00:37.788 INFO:teuthology.orchestra.run.smithi033.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-19T01:00:37.812 INFO:teuthology.orchestra.run.smithi033.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-19T01:00:37.828 INFO:teuthology.orchestra.run.smithi155.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-19T01:00:37.829 INFO:teuthology.orchestra.run.smithi155.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-19T01:00:37.836 INFO:teuthology.orchestra.run.smithi155.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-19T01:00:37.842 INFO:teuthology.orchestra.run.smithi155.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-19T01:00:37.856 INFO:teuthology.orchestra.run.smithi155.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-19T01:00:37.878 INFO:teuthology.orchestra.run.smithi155.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-19T01:00:37.884 INFO:teuthology.orchestra.run.smithi155.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-19T01:00:37.889 INFO:teuthology.orchestra.run.smithi033.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-19T01:00:37.894 INFO:teuthology.orchestra.run.smithi033.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-19T01:00:37.908 INFO:teuthology.orchestra.run.smithi033.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-19T01:00:37.913 INFO:teuthology.orchestra.run.smithi033.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-19T01:00:37.914 INFO:teuthology.orchestra.run.smithi033.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-19T01:00:37.915 INFO:teuthology.orchestra.run.smithi033.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-19T01:00:37.925 INFO:teuthology.orchestra.run.smithi033.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-19T01:00:37.927 INFO:teuthology.orchestra.run.smithi033.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-19T01:00:37.930 INFO:teuthology.orchestra.run.smithi033.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-19T01:00:37.931 INFO:teuthology.orchestra.run.smithi033.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-19T01:00:37.947 INFO:teuthology.orchestra.run.smithi033.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-19T01:00:37.949 INFO:teuthology.orchestra.run.smithi033.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-19T01:00:37.952 INFO:teuthology.orchestra.run.smithi033.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-19T01:00:37.954 INFO:teuthology.orchestra.run.smithi033.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-19T01:00:37.955 INFO:teuthology.orchestra.run.smithi033.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-19T01:00:37.968 INFO:teuthology.orchestra.run.smithi033.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-19T01:00:37.969 INFO:teuthology.orchestra.run.smithi033.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-19T01:00:37.970 INFO:teuthology.orchestra.run.smithi033.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-19T01:00:37.970 INFO:teuthology.orchestra.run.smithi033.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-19T01:00:37.971 INFO:teuthology.orchestra.run.smithi033.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-19T01:00:37.988 INFO:teuthology.orchestra.run.smithi033.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-19T01:00:37.988 INFO:teuthology.orchestra.run.smithi033.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-19T01:00:37.989 INFO:teuthology.orchestra.run.smithi033.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-19T01:00:37.992 INFO:teuthology.orchestra.run.smithi033.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-19T01:00:37.992 INFO:teuthology.orchestra.run.smithi033.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-19T01:00:38.008 INFO:teuthology.orchestra.run.smithi033.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-19T01:00:38.009 INFO:teuthology.orchestra.run.smithi033.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-19T01:00:38.010 INFO:teuthology.orchestra.run.smithi033.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-19T01:00:38.010 INFO:teuthology.orchestra.run.smithi033.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-19T01:00:38.011 INFO:teuthology.orchestra.run.smithi033.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-19T01:00:38.028 INFO:teuthology.orchestra.run.smithi033.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-19T01:00:38.030 INFO:teuthology.orchestra.run.smithi033.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-19T01:00:38.030 INFO:teuthology.orchestra.run.smithi033.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-19T01:00:38.036 INFO:teuthology.orchestra.run.smithi033.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-19T01:00:38.036 INFO:teuthology.orchestra.run.smithi033.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-19T01:00:38.050 INFO:teuthology.orchestra.run.smithi033.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-19T01:00:38.050 INFO:teuthology.orchestra.run.smithi033.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-19T01:00:38.052 INFO:teuthology.orchestra.run.smithi033.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-19T01:00:38.053 INFO:teuthology.orchestra.run.smithi118.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-3154-g8b4ab94f-1jammy [244 kB] 2024-04-19T01:00:38.054 INFO:teuthology.orchestra.run.smithi033.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-19T01:00:38.057 INFO:teuthology.orchestra.run.smithi033.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-19T01:00:38.060 INFO:teuthology.orchestra.run.smithi155.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-19T01:00:38.068 INFO:teuthology.orchestra.run.smithi118.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-3154-g8b4ab94f-1jammy [124 kB] 2024-04-19T01:00:38.070 INFO:teuthology.orchestra.run.smithi033.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-19T01:00:38.071 INFO:teuthology.orchestra.run.smithi033.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-19T01:00:38.072 INFO:teuthology.orchestra.run.smithi118.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-3154-g8b4ab94f-1jammy [1,526 kB] 2024-04-19T01:00:38.074 INFO:teuthology.orchestra.run.smithi033.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-19T01:00:38.074 INFO:teuthology.orchestra.run.smithi033.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-19T01:00:38.075 INFO:teuthology.orchestra.run.smithi033.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-19T01:00:38.091 INFO:teuthology.orchestra.run.smithi033.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-19T01:00:38.163 INFO:teuthology.orchestra.run.smithi033.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-19T01:00:38.164 INFO:teuthology.orchestra.run.smithi033.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-19T01:00:38.166 INFO:teuthology.orchestra.run.smithi033.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-19T01:00:38.166 INFO:teuthology.orchestra.run.smithi033.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-19T01:00:38.169 INFO:teuthology.orchestra.run.smithi033.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-19T01:00:38.191 INFO:teuthology.orchestra.run.smithi033.stdout:Get:65 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-3154-g8b4ab94f-1jammy [112 kB] 2024-04-19T01:00:38.199 INFO:teuthology.orchestra.run.smithi033.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-19T01:00:38.202 INFO:teuthology.orchestra.run.smithi033.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-19T01:00:38.202 INFO:teuthology.orchestra.run.smithi033.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-19T01:00:38.216 INFO:teuthology.orchestra.run.smithi118.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-3154-g8b4ab94f-1jammy [6,260 kB] 2024-04-19T01:00:38.228 INFO:teuthology.orchestra.run.smithi033.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-19T01:00:38.228 INFO:teuthology.orchestra.run.smithi033.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-19T01:00:38.235 INFO:teuthology.orchestra.run.smithi033.stdout:Get:71 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-3154-g8b4ab94f-1jammy [466 kB] 2024-04-19T01:00:38.242 INFO:teuthology.orchestra.run.smithi033.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-19T01:00:38.242 INFO:teuthology.orchestra.run.smithi033.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-19T01:00:38.243 INFO:teuthology.orchestra.run.smithi033.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-19T01:00:38.244 INFO:teuthology.orchestra.run.smithi033.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-19T01:00:38.244 INFO:teuthology.orchestra.run.smithi033.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-19T01:00:38.245 INFO:teuthology.orchestra.run.smithi033.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-19T01:00:38.249 INFO:teuthology.orchestra.run.smithi033.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-19T01:00:38.264 INFO:teuthology.orchestra.run.smithi033.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-19T01:00:38.266 INFO:teuthology.orchestra.run.smithi033.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-19T01:00:38.267 INFO:teuthology.orchestra.run.smithi033.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-19T01:00:38.288 INFO:teuthology.orchestra.run.smithi033.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-19T01:00:38.292 INFO:teuthology.orchestra.run.smithi033.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-19T01:00:38.293 INFO:teuthology.orchestra.run.smithi033.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-19T01:00:38.293 INFO:teuthology.orchestra.run.smithi033.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-19T01:00:38.299 INFO:teuthology.orchestra.run.smithi033.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-19T01:00:38.300 INFO:teuthology.orchestra.run.smithi033.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-19T01:00:38.304 INFO:teuthology.orchestra.run.smithi033.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-19T01:00:38.314 INFO:teuthology.orchestra.run.smithi033.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-19T01:00:38.314 INFO:teuthology.orchestra.run.smithi033.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-19T01:00:38.315 INFO:teuthology.orchestra.run.smithi033.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-19T01:00:38.319 INFO:teuthology.orchestra.run.smithi033.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-3154-g8b4ab94f-1jammy [25.8 MB] 2024-04-19T01:00:38.324 INFO:teuthology.orchestra.run.smithi155.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-19T01:00:38.335 INFO:teuthology.orchestra.run.smithi033.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-19T01:00:38.337 INFO:teuthology.orchestra.run.smithi033.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-19T01:00:38.347 INFO:teuthology.orchestra.run.smithi033.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-19T01:00:38.348 INFO:teuthology.orchestra.run.smithi033.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-19T01:00:38.348 INFO:teuthology.orchestra.run.smithi033.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-19T01:00:38.351 INFO:teuthology.orchestra.run.smithi033.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-19T01:00:38.351 INFO:teuthology.orchestra.run.smithi033.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-19T01:00:38.437 INFO:teuthology.orchestra.run.smithi155.stdout:Get:65 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-3154-g8b4ab94f-1jammy [14.1 kB] 2024-04-19T01:00:38.438 INFO:teuthology.orchestra.run.smithi155.stdout:Get:66 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-3154-g8b4ab94f-1jammy [870 kB] 2024-04-19T01:00:38.454 INFO:teuthology.orchestra.run.smithi155.stdout:Get:67 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-3154-g8b4ab94f-1jammy [2,437 kB] 2024-04-19T01:00:38.503 INFO:teuthology.orchestra.run.smithi155.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-19T01:00:38.517 INFO:teuthology.orchestra.run.smithi155.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-19T01:00:38.537 INFO:teuthology.orchestra.run.smithi155.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-19T01:00:38.602 INFO:teuthology.orchestra.run.smithi155.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-19T01:00:38.605 INFO:teuthology.orchestra.run.smithi155.stdout:Get:72 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-3154-g8b4ab94f-1jammy [791 kB] 2024-04-19T01:00:38.670 INFO:teuthology.orchestra.run.smithi155.stdout:Get:73 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-3154-g8b4ab94f-1jammy [148 kB] 2024-04-19T01:00:38.673 INFO:teuthology.orchestra.run.smithi155.stdout:Get:74 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-3154-g8b4ab94f-1jammy [2,297 kB] 2024-04-19T01:00:38.786 INFO:teuthology.orchestra.run.smithi110.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-3154-g8b4ab94f-1jammy [712 kB] 2024-04-19T01:00:38.852 INFO:teuthology.orchestra.run.smithi155.stdout:Get:75 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-3154-g8b4ab94f-1jammy [8,625 kB] 2024-04-19T01:00:38.890 INFO:teuthology.orchestra.run.smithi155.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-19T01:00:38.893 INFO:teuthology.orchestra.run.smithi155.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-19T01:00:38.895 INFO:teuthology.orchestra.run.smithi155.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-19T01:00:38.896 INFO:teuthology.orchestra.run.smithi155.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-19T01:00:38.900 INFO:teuthology.orchestra.run.smithi155.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-19T01:00:39.068 INFO:teuthology.orchestra.run.smithi118.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-3154-g8b4ab94f-1jammy [22.8 MB] 2024-04-19T01:00:39.123 INFO:teuthology.orchestra.run.smithi155.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-19T01:00:39.186 INFO:teuthology.orchestra.run.smithi110.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-3154-g8b4ab94f-1jammy [356 kB] 2024-04-19T01:00:39.398 INFO:teuthology.orchestra.run.smithi110.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-3154-g8b4ab94f-1jammy [32.8 kB] 2024-04-19T01:00:39.441 INFO:teuthology.orchestra.run.smithi110.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-3154-g8b4ab94f-1jammy [180 kB] 2024-04-19T01:00:39.468 INFO:teuthology.orchestra.run.smithi155.stdout:Get:82 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-3154-g8b4ab94f-1jammy [14.2 kB] 2024-04-19T01:00:39.468 INFO:teuthology.orchestra.run.smithi155.stdout:Get:83 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-3154-g8b4ab94f-1jammy [51.6 MB] 2024-04-19T01:00:39.533 INFO:teuthology.orchestra.run.smithi110.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-3154-g8b4ab94f-1jammy [66.9 kB] 2024-04-19T01:00:39.542 INFO:teuthology.orchestra.run.smithi110.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-3154-g8b4ab94f-1jammy [330 kB] 2024-04-19T01:00:39.561 INFO:teuthology.orchestra.run.smithi155.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-19T01:00:39.567 INFO:teuthology.orchestra.run.smithi155.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-19T01:00:39.748 INFO:teuthology.orchestra.run.smithi110.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-3154-g8b4ab94f-1jammy [6,894 kB] 2024-04-19T01:00:39.859 INFO:teuthology.orchestra.run.smithi155.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-19T01:00:39.859 INFO:teuthology.orchestra.run.smithi155.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-19T01:00:39.861 INFO:teuthology.orchestra.run.smithi155.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-19T01:00:39.863 INFO:teuthology.orchestra.run.smithi155.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-19T01:00:39.865 INFO:teuthology.orchestra.run.smithi155.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-19T01:00:39.867 INFO:teuthology.orchestra.run.smithi155.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-19T01:00:39.868 INFO:teuthology.orchestra.run.smithi155.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-19T01:00:39.870 INFO:teuthology.orchestra.run.smithi155.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-19T01:00:40.054 INFO:teuthology.orchestra.run.smithi155.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-19T01:00:40.484 INFO:teuthology.orchestra.run.smithi155.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-19T01:00:40.537 INFO:teuthology.orchestra.run.smithi155.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-19T01:00:40.559 INFO:teuthology.orchestra.run.smithi155.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-19T01:00:40.666 INFO:teuthology.orchestra.run.smithi155.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-19T01:00:40.710 INFO:teuthology.orchestra.run.smithi155.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-19T01:00:40.722 INFO:teuthology.orchestra.run.smithi155.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-19T01:00:40.728 INFO:teuthology.orchestra.run.smithi155.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-19T01:00:40.773 INFO:teuthology.orchestra.run.smithi155.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-19T01:00:40.776 INFO:teuthology.orchestra.run.smithi155.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-19T01:00:40.798 INFO:teuthology.orchestra.run.smithi155.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-19T01:00:40.962 INFO:teuthology.orchestra.run.smithi155.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-19T01:00:41.242 INFO:teuthology.orchestra.run.smithi155.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-19T01:00:41.259 INFO:teuthology.orchestra.run.smithi155.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-19T01:00:41.414 INFO:teuthology.orchestra.run.smithi155.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-19T01:00:41.444 INFO:teuthology.orchestra.run.smithi155.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-19T01:00:41.476 INFO:teuthology.orchestra.run.smithi118.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-3154-g8b4ab94f-1jammy [14.1 kB] 2024-04-19T01:00:41.477 INFO:teuthology.orchestra.run.smithi118.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-3154-g8b4ab94f-1jammy [870 kB] 2024-04-19T01:00:41.504 INFO:teuthology.orchestra.run.smithi118.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-3154-g8b4ab94f-1jammy [2,437 kB] 2024-04-19T01:00:41.602 INFO:teuthology.orchestra.run.smithi155.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-19T01:00:41.605 INFO:teuthology.orchestra.run.smithi155.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-19T01:00:41.607 INFO:teuthology.orchestra.run.smithi155.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-19T01:00:41.630 INFO:teuthology.orchestra.run.smithi155.stdout:Get:113 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-19T01:00:41.634 INFO:teuthology.orchestra.run.smithi155.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-19T01:00:41.766 INFO:teuthology.orchestra.run.smithi118.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-3154-g8b4ab94f-1jammy [791 kB] 2024-04-19T01:00:41.843 INFO:teuthology.orchestra.run.smithi118.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-3154-g8b4ab94f-1jammy [148 kB] 2024-04-19T01:00:41.870 INFO:teuthology.orchestra.run.smithi118.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-3154-g8b4ab94f-1jammy [2,297 kB] 2024-04-19T01:00:42.099 INFO:teuthology.orchestra.run.smithi118.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-3154-g8b4ab94f-1jammy [8,625 kB] 2024-04-19T01:00:42.852 INFO:teuthology.orchestra.run.smithi033.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-3154-g8b4ab94f-1jammy [5,095 kB] 2024-04-19T01:00:43.041 INFO:teuthology.orchestra.run.smithi118.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-3154-g8b4ab94f-1jammy [14.2 kB] 2024-04-19T01:00:43.041 INFO:teuthology.orchestra.run.smithi118.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-3154-g8b4ab94f-1jammy [51.6 MB] 2024-04-19T01:00:43.222 INFO:teuthology.orchestra.run.smithi110.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-3154-g8b4ab94f-1jammy [112 kB] 2024-04-19T01:00:43.255 INFO:teuthology.orchestra.run.smithi110.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-3154-g8b4ab94f-1jammy [466 kB] 2024-04-19T01:00:43.456 INFO:teuthology.orchestra.run.smithi110.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-3154-g8b4ab94f-1jammy [25.8 MB] 2024-04-19T01:00:43.556 INFO:teuthology.orchestra.run.smithi155.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-3154-g8b4ab94f-1jammy [124 kB] 2024-04-19T01:00:43.558 INFO:teuthology.orchestra.run.smithi155.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-3154-g8b4ab94f-1jammy [38.9 kB] 2024-04-19T01:00:43.559 INFO:teuthology.orchestra.run.smithi155.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-3154-g8b4ab94f-1jammy [13.6 MB] 2024-04-19T01:00:43.626 INFO:teuthology.orchestra.run.smithi033.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-3154-g8b4ab94f-1jammy [244 kB] 2024-04-19T01:00:43.690 INFO:teuthology.orchestra.run.smithi033.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-3154-g8b4ab94f-1jammy [124 kB] 2024-04-19T01:00:43.697 INFO:teuthology.orchestra.run.smithi033.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-3154-g8b4ab94f-1jammy [1,526 kB] 2024-04-19T01:00:43.987 INFO:teuthology.orchestra.run.smithi033.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-3154-g8b4ab94f-1jammy [6,260 kB] 2024-04-19T01:00:44.427 INFO:teuthology.orchestra.run.smithi155.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-3154-g8b4ab94f-1jammy [91.4 kB] 2024-04-19T01:00:44.787 INFO:teuthology.orchestra.run.smithi033.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-3154-g8b4ab94f-1jammy [22.8 MB] 2024-04-19T01:00:44.984 INFO:teuthology.orchestra.run.smithi155.stdout:Fetched 181 MB in 11s (16.1 MB/s) 2024-04-19T01:00:45.148 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-19T01:00:45.249 INFO:teuthology.orchestra.run.smithi155.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-19T01:00:45.255 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-19T01:00:45.347 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-19T01:00:45.642 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-19T01:00:45.668 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-19T01:00:45.700 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-19T01:00:46.011 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-19T01:00:46.038 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-19T01:00:46.078 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-19T01:00:46.781 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-19T01:00:46.808 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-19T01:00:46.858 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T01:00:47.226 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-19T01:00:47.253 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-19T01:00:47.284 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T01:00:47.300 INFO:teuthology.orchestra.run.smithi033.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-3154-g8b4ab94f-1jammy [14.1 kB] 2024-04-19T01:00:47.301 INFO:teuthology.orchestra.run.smithi033.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-3154-g8b4ab94f-1jammy [870 kB] 2024-04-19T01:00:47.318 INFO:teuthology.orchestra.run.smithi033.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-3154-g8b4ab94f-1jammy [2,437 kB] 2024-04-19T01:00:47.366 INFO:teuthology.orchestra.run.smithi118.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-3154-g8b4ab94f-1jammy [124 kB] 2024-04-19T01:00:47.368 INFO:teuthology.orchestra.run.smithi118.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-3154-g8b4ab94f-1jammy [38.9 kB] 2024-04-19T01:00:47.370 INFO:teuthology.orchestra.run.smithi118.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-3154-g8b4ab94f-1jammy [13.6 MB] 2024-04-19T01:00:47.562 INFO:teuthology.orchestra.run.smithi033.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-3154-g8b4ab94f-1jammy [791 kB] 2024-04-19T01:00:47.595 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-19T01:00:47.622 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-19T01:00:47.662 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T01:00:47.670 INFO:teuthology.orchestra.run.smithi033.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-3154-g8b4ab94f-1jammy [148 kB] 2024-04-19T01:00:47.674 INFO:teuthology.orchestra.run.smithi033.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-3154-g8b4ab94f-1jammy [2,297 kB] 2024-04-19T01:00:47.909 INFO:teuthology.orchestra.run.smithi033.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-3154-g8b4ab94f-1jammy [8,625 kB] 2024-04-19T01:00:47.948 INFO:teuthology.orchestra.run.smithi110.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-3154-g8b4ab94f-1jammy [5,095 kB] 2024-04-19T01:00:48.048 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-19T01:00:48.074 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-19T01:00:48.114 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-19T01:00:48.163 INFO:teuthology.orchestra.run.smithi118.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-3154-g8b4ab94f-1jammy [91.4 kB] 2024-04-19T01:00:48.272 INFO:teuthology.orchestra.run.smithi110.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-3154-g8b4ab94f-1jammy [244 kB] 2024-04-19T01:00:48.279 INFO:teuthology.orchestra.run.smithi110.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-3154-g8b4ab94f-1jammy [124 kB] 2024-04-19T01:00:48.301 INFO:teuthology.orchestra.run.smithi110.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-3154-g8b4ab94f-1jammy [1,526 kB] 2024-04-19T01:00:48.388 INFO:teuthology.orchestra.run.smithi110.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-3154-g8b4ab94f-1jammy [6,260 kB] 2024-04-19T01:00:48.442 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package librados2. 2024-04-19T01:00:48.469 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../007-librados2_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:00:48.509 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking librados2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:00:48.600 INFO:teuthology.orchestra.run.smithi033.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-3154-g8b4ab94f-1jammy [14.2 kB] 2024-04-19T01:00:48.601 INFO:teuthology.orchestra.run.smithi033.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-3154-g8b4ab94f-1jammy [51.6 MB] 2024-04-19T01:00:48.716 INFO:teuthology.orchestra.run.smithi118.stdout:Fetched 181 MB in 15s (12.3 MB/s) 2024-04-19T01:00:48.783 INFO:teuthology.orchestra.run.smithi110.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-3154-g8b4ab94f-1jammy [22.8 MB] 2024-04-19T01:00:48.879 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-19T01:00:48.912 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package librbd1. 2024-04-19T01:00:48.938 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../008-librbd1_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:00:48.970 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking librbd1 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:00:48.977 INFO:teuthology.orchestra.run.smithi118.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-19T01:00:48.982 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-19T01:00:49.069 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-19T01:00:49.373 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-19T01:00:49.397 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package libcephfs2. 2024-04-19T01:00:49.399 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-19T01:00:49.423 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:00:49.431 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-19T01:00:49.464 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking libcephfs2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:00:49.758 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-rados. 2024-04-19T01:00:49.776 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-19T01:00:49.784 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../010-python3-rados_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:00:49.802 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-19T01:00:49.824 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-rados (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:00:49.842 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-19T01:00:50.202 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-19T01:00:50.221 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-19T01:00:50.228 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:00:50.248 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-19T01:00:50.268 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-ceph-argparse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:00:50.288 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T01:00:50.321 INFO:teuthology.orchestra.run.smithi110.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-3154-g8b4ab94f-1jammy [14.1 kB] 2024-04-19T01:00:50.322 INFO:teuthology.orchestra.run.smithi110.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-3154-g8b4ab94f-1jammy [870 kB] 2024-04-19T01:00:50.338 INFO:teuthology.orchestra.run.smithi110.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-3154-g8b4ab94f-1jammy [2,437 kB] 2024-04-19T01:00:50.380 INFO:teuthology.orchestra.run.smithi110.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-3154-g8b4ab94f-1jammy [791 kB] 2024-04-19T01:00:50.423 INFO:teuthology.orchestra.run.smithi110.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-3154-g8b4ab94f-1jammy [148 kB] 2024-04-19T01:00:50.428 INFO:teuthology.orchestra.run.smithi110.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-3154-g8b4ab94f-1jammy [2,297 kB] 2024-04-19T01:00:50.521 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-cephfs. 2024-04-19T01:00:50.547 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:00:50.567 INFO:teuthology.orchestra.run.smithi110.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-3154-g8b4ab94f-1jammy [8,625 kB] 2024-04-19T01:00:50.587 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-cephfs (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:00:50.666 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-19T01:00:50.693 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-19T01:00:50.733 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T01:00:50.898 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-19T01:00:50.925 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:00:50.966 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-ceph-common (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:00:51.057 INFO:teuthology.orchestra.run.smithi110.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-3154-g8b4ab94f-1jammy [14.2 kB] 2024-04-19T01:00:51.057 INFO:teuthology.orchestra.run.smithi110.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-3154-g8b4ab94f-1jammy [51.6 MB] 2024-04-19T01:00:51.102 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-19T01:00:51.129 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-19T01:00:51.169 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T01:00:51.335 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-19T01:00:51.361 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-19T01:00:51.410 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-19T01:00:51.464 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-19T01:00:51.479 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-19T01:00:51.522 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-19T01:00:51.834 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package librados2. 2024-04-19T01:00:51.860 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../007-librados2_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:00:51.879 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-prettytable. 2024-04-19T01:00:51.900 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking librados2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:00:51.905 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-19T01:00:51.954 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-19T01:00:52.223 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-rbd. 2024-04-19T01:00:52.239 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:00:52.281 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-rbd (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:00:52.353 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package librbd1. 2024-04-19T01:00:52.379 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../008-librbd1_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:00:52.420 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking librbd1 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:00:52.642 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-19T01:00:52.669 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-19T01:00:52.709 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-19T01:00:52.743 INFO:teuthology.orchestra.run.smithi033.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-3154-g8b4ab94f-1jammy [124 kB] 2024-04-19T01:00:52.746 INFO:teuthology.orchestra.run.smithi033.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-3154-g8b4ab94f-1jammy [38.9 kB] 2024-04-19T01:00:52.747 INFO:teuthology.orchestra.run.smithi033.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-3154-g8b4ab94f-1jammy [13.6 MB] 2024-04-19T01:00:53.011 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-19T01:00:53.038 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-19T01:00:53.073 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libcephfs2. 2024-04-19T01:00:53.086 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-19T01:00:53.099 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:00:53.139 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libcephfs2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:00:53.434 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-rados. 2024-04-19T01:00:53.455 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-19T01:00:53.460 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../010-python3-rados_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:00:53.482 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-19T01:00:53.500 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-rados (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:00:53.530 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-19T01:00:53.748 INFO:teuthology.orchestra.run.smithi033.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-3154-g8b4ab94f-1jammy [91.4 kB] 2024-04-19T01:00:53.795 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-19T01:00:53.821 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:00:53.853 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-ceph-argparse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:00:53.867 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package lua5.1. 2024-04-19T01:00:53.893 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-19T01:00:53.933 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-19T01:00:54.054 INFO:teuthology.orchestra.run.smithi110.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-3154-g8b4ab94f-1jammy [124 kB] 2024-04-19T01:00:54.057 INFO:teuthology.orchestra.run.smithi110.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-3154-g8b4ab94f-1jammy [38.9 kB] 2024-04-19T01:00:54.058 INFO:teuthology.orchestra.run.smithi110.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-3154-g8b4ab94f-1jammy [13.6 MB] 2024-04-19T01:00:54.156 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-cephfs. 2024-04-19T01:00:54.182 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:00:54.222 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-cephfs (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:00:54.253 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package lua-any. 2024-04-19T01:00:54.279 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-19T01:00:54.320 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-19T01:00:54.334 INFO:teuthology.orchestra.run.smithi033.stdout:Fetched 181 MB in 19s (9,560 kB/s) 2024-04-19T01:00:54.392 INFO:teuthology.orchestra.run.smithi110.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-3154-g8b4ab94f-1jammy [91.4 kB] 2024-04-19T01:00:54.500 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-19T01:00:54.534 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-19T01:00:54.560 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:00:54.603 INFO:teuthology.orchestra.run.smithi033.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-19T01:00:54.605 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package zip. 2024-04-19T01:00:54.608 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-19T01:00:54.609 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-ceph-common (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:00:54.632 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-19T01:00:54.680 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking zip (3.0-12build2) ... 2024-04-19T01:00:54.690 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-19T01:00:54.987 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-19T01:00:55.012 INFO:teuthology.orchestra.run.smithi110.stdout:Fetched 181 MB in 21s (8,821 kB/s) 2024-04-19T01:00:55.012 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-19T01:00:55.062 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-19T01:00:55.067 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package unzip. 2024-04-19T01:00:55.069 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-19T01:00:55.093 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-19T01:00:55.095 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-19T01:00:55.135 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-19T01:00:55.141 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-19T01:00:55.175 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-19T01:00:55.275 INFO:teuthology.orchestra.run.smithi110.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-19T01:00:55.281 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-19T01:00:55.365 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-19T01:00:55.423 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-prettytable. 2024-04-19T01:00:55.449 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-19T01:00:55.464 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-19T01:00:55.490 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-19T01:00:55.490 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-19T01:00:55.530 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-19T01:00:55.545 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package luarocks. 2024-04-19T01:00:55.572 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-19T01:00:55.612 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-19T01:00:55.676 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-19T01:00:55.703 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-19T01:00:55.743 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-19T01:00:55.751 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-rbd. 2024-04-19T01:00:55.777 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:00:55.817 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-rbd (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:00:55.901 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-19T01:00:55.927 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-19T01:00:55.976 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T01:00:55.994 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package librgw2. 2024-04-19T01:00:56.021 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../025-librgw2_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:00:56.060 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking librgw2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:00:56.074 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-19T01:00:56.098 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-19T01:00:56.138 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-19T01:00:56.212 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-19T01:00:56.238 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-19T01:00:56.287 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-19T01:00:56.354 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-19T01:00:56.381 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-19T01:00:56.420 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T01:00:56.623 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-19T01:00:56.629 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-rgw. 2024-04-19T01:00:56.638 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-19T01:00:56.645 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:00:56.683 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-19T01:00:56.688 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-rgw (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:00:56.689 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-19T01:00:56.710 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-19T01:00:56.758 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T01:00:56.899 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-19T01:00:56.926 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-19T01:00:56.965 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T01:00:57.059 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-19T01:00:57.074 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-19T01:00:57.126 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-19T01:00:57.186 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-19T01:00:57.213 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-19T01:00:57.215 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-19T01:00:57.242 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-19T01:00:57.253 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T01:00:57.282 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-19T01:00:57.302 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-19T01:00:57.329 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-19T01:00:57.368 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-19T01:00:57.496 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package lua5.1. 2024-04-19T01:00:57.522 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-19T01:00:57.539 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-19T01:00:57.552 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package libradosstriper1. 2024-04-19T01:00:57.562 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-19T01:00:57.566 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-19T01:00:57.567 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:00:57.601 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking libradosstriper1 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:00:57.606 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T01:00:57.663 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package librados2. 2024-04-19T01:00:57.679 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../007-librados2_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:00:57.730 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking librados2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:00:57.899 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package lua-any. 2024-04-19T01:00:57.925 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-19T01:00:57.929 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-19T01:00:57.952 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-19T01:00:57.992 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-19T01:00:58.030 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package ceph-common. 2024-04-19T01:00:58.056 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../029-ceph-common_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:00:58.058 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-19T01:00:58.096 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking ceph-common (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:00:58.150 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package librbd1. 2024-04-19T01:00:58.176 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../008-librbd1_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:00:58.216 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking librbd1 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:00:58.312 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package librados2. 2024-04-19T01:00:58.338 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../007-librados2_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:00:58.352 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package zip. 2024-04-19T01:00:58.378 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-19T01:00:58.379 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking librados2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:00:58.427 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking zip (3.0-12build2) ... 2024-04-19T01:00:58.669 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libcephfs2. 2024-04-19T01:00:58.696 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:00:58.735 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libcephfs2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:00:58.764 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package unzip. 2024-04-19T01:00:58.790 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-19T01:00:58.831 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package librbd1. 2024-04-19T01:00:58.838 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-19T01:00:58.858 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../008-librbd1_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:00:58.898 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking librbd1 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:00:59.021 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-rados. 2024-04-19T01:00:59.048 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../010-python3-rados_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:00:59.088 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-rados (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:00:59.226 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package luarocks. 2024-04-19T01:00:59.252 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-19T01:00:59.292 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libcephfs2. 2024-04-19T01:00:59.293 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-19T01:00:59.319 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:00:59.359 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libcephfs2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:00:59.450 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-19T01:00:59.476 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package ceph-base. 2024-04-19T01:00:59.477 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:00:59.503 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../030-ceph-base_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:00:59.524 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-ceph-argparse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:00:59.552 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking ceph-base (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:00:59.636 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-rados. 2024-04-19T01:00:59.652 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../010-python3-rados_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:00:59.659 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package librgw2. 2024-04-19T01:00:59.685 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../025-librgw2_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:00:59.695 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-rados (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:00:59.726 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking librgw2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:00:59.777 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-cephfs. 2024-04-19T01:00:59.793 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:00:59.835 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-cephfs (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:00:59.997 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-19T01:01:00.024 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:01:00.064 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-ceph-argparse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:00.138 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-19T01:01:00.155 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-19T01:01:00.165 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:01:00.182 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-19T01:01:00.204 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-ceph-common (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:00.221 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-19T01:01:00.295 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-rgw. 2024-04-19T01:01:00.317 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-cephfs. 2024-04-19T01:01:00.321 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:00.343 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:00.370 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-rgw (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:00.383 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-cephfs (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:00.499 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-19T01:01:00.508 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-cheroot. 2024-04-19T01:01:00.526 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-19T01:01:00.534 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-19T01:01:00.557 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-19T01:01:00.566 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-19T01:01:00.644 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-19T01:01:00.659 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:01:00.703 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-ceph-common (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:00.723 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-19T01:01:00.749 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-19T01:01:00.798 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-19T01:01:00.869 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-prettytable. 2024-04-19T01:01:00.877 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-19T01:01:00.893 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-19T01:01:00.895 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-19T01:01:00.927 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-19T01:01:00.935 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-19T01:01:01.014 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-19T01:01:01.040 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-19T01:01:01.081 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-19T01:01:01.109 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libradosstriper1. 2024-04-19T01:01:01.135 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:01.176 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libradosstriper1 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:01.205 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-rbd. 2024-04-19T01:01:01.213 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-19T01:01:01.220 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:01.240 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-19T01:01:01.263 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-rbd (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:01.279 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-19T01:01:01.400 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-prettytable. 2024-04-19T01:01:01.427 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-19T01:01:01.514 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-19T01:01:01.582 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-19T01:01:01.608 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-19T01:01:01.609 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-19T01:01:01.635 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-19T01:01:01.649 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-19T01:01:01.674 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-19T01:01:01.696 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-common. 2024-04-19T01:01:01.722 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../029-ceph-common_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:01.771 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-common (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:01.811 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-rbd. 2024-04-19T01:01:01.826 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:01.861 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-rbd (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:01.919 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-19T01:01:01.926 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-tempora. 2024-04-19T01:01:01.934 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-19T01:01:01.953 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-19T01:01:01.977 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-19T01:01:01.993 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-19T01:01:02.214 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-19T01:01:02.240 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-19T01:01:02.245 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-portend. 2024-04-19T01:01:02.271 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-19T01:01:02.272 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-19T01:01:02.280 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-19T01:01:02.298 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-19T01:01:02.312 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-19T01:01:02.338 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-19T01:01:02.533 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-19T01:01:02.549 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-19T01:01:02.591 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-19T01:01:02.615 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-19T01:01:02.642 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-19T01:01:02.681 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-19T01:01:02.683 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package lua5.1. 2024-04-19T01:01:02.710 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-19T01:01:02.750 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-19T01:01:02.902 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-19T01:01:02.918 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-19T01:01:02.961 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-19T01:01:02.976 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-19T01:01:03.003 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-19T01:01:03.036 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package lua-any. 2024-04-19T01:01:03.042 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-19T01:01:03.052 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-19T01:01:03.086 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-19T01:01:03.256 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package lua5.1. 2024-04-19T01:01:03.283 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-19T01:01:03.312 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-base. 2024-04-19T01:01:03.314 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-19T01:01:03.339 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../030-ceph-base_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:03.339 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package zip. 2024-04-19T01:01:03.355 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-19T01:01:03.370 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-natsort. 2024-04-19T01:01:03.379 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-base (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:03.397 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-19T01:01:03.397 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking zip (3.0-12build2) ... 2024-04-19T01:01:03.437 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-19T01:01:03.626 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package lua-any. 2024-04-19T01:01:03.652 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-19T01:01:03.692 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-19T01:01:03.748 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-logutils. 2024-04-19T01:01:03.775 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-19T01:01:03.776 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package unzip. 2024-04-19T01:01:03.803 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-19T01:01:03.815 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-19T01:01:03.842 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-19T01:01:03.919 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package zip. 2024-04-19T01:01:03.925 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-19T01:01:03.935 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-19T01:01:03.952 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-19T01:01:03.986 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking zip (3.0-12build2) ... 2024-04-19T01:01:03.991 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-19T01:01:04.101 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-19T01:01:04.127 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-19T01:01:04.167 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-19T01:01:04.246 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package luarocks. 2024-04-19T01:01:04.273 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package unzip. 2024-04-19T01:01:04.273 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-19T01:01:04.288 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-19T01:01:04.313 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-19T01:01:04.320 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-cheroot. 2024-04-19T01:01:04.323 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-19T01:01:04.347 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-19T01:01:04.386 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-19T01:01:04.462 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-mako. 2024-04-19T01:01:04.488 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-19T01:01:04.528 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-19T01:01:04.677 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package luarocks. 2024-04-19T01:01:04.703 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-19T01:01:04.723 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-19T01:01:04.743 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-19T01:01:04.746 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package librgw2. 2024-04-19T01:01:04.750 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-19T01:01:04.773 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../025-librgw2_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:04.789 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-19T01:01:04.821 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking librgw2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:04.856 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-19T01:01:04.882 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-19T01:01:04.923 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-19T01:01:05.051 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-19T01:01:05.066 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-19T01:01:05.101 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-19T01:01:05.118 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package librgw2. 2024-04-19T01:01:05.144 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../025-librgw2_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:05.184 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking librgw2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:05.242 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-19T01:01:05.268 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-19T01:01:05.309 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-19T01:01:05.387 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-19T01:01:05.414 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-19T01:01:05.415 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-rgw. 2024-04-19T01:01:05.431 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:05.454 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-19T01:01:05.473 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-rgw (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:05.611 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-webob. 2024-04-19T01:01:05.638 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-19T01:01:05.678 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-19T01:01:05.745 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-rgw. 2024-04-19T01:01:05.757 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-tempora. 2024-04-19T01:01:05.772 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:05.784 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-19T01:01:05.810 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-19T01:01:05.812 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-rgw (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:05.823 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-19T01:01:05.837 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-19T01:01:05.876 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-19T01:01:06.005 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-waitress. 2024-04-19T01:01:06.028 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-19T01:01:06.067 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-19T01:01:06.082 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-19T01:01:06.108 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-19T01:01:06.110 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-portend. 2024-04-19T01:01:06.137 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-19T01:01:06.139 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-19T01:01:06.168 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-19T01:01:06.221 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libradosstriper1. 2024-04-19T01:01:06.248 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:06.296 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libradosstriper1 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:06.350 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-tempita. 2024-04-19T01:01:06.377 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-19T01:01:06.408 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-19T01:01:06.429 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-19T01:01:06.456 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-19T01:01:06.459 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libradosstriper1. 2024-04-19T01:01:06.485 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:06.496 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-19T01:01:06.526 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libradosstriper1 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:06.711 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-paste. 2024-04-19T01:01:06.737 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-19T01:01:06.777 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-19T01:01:06.799 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-19T01:01:06.816 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-common. 2024-04-19T01:01:06.826 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-19T01:01:06.843 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../029-ceph-common_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:06.866 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-19T01:01:06.891 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-common (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:06.963 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-common. 2024-04-19T01:01:06.989 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../029-ceph-common_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:07.029 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-common (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:07.088 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-19T01:01:07.105 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-19T01:01:07.138 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-19T01:01:07.212 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-natsort. 2024-04-19T01:01:07.238 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-19T01:01:07.278 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-19T01:01:07.590 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-logutils. 2024-04-19T01:01:07.617 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-19T01:01:07.656 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-19T01:01:07.968 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-19T01:01:07.995 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-19T01:01:08.034 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-19T01:01:08.296 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-mako. 2024-04-19T01:01:08.323 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-19T01:01:08.338 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-base. 2024-04-19T01:01:08.354 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-19T01:01:08.365 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../030-ceph-base_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:08.392 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-19T01:01:08.415 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-base (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:08.419 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-19T01:01:08.458 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-19T01:01:08.657 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-19T01:01:08.684 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-19T01:01:08.724 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-19T01:01:08.744 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-webtest. 2024-04-19T01:01:08.771 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-19T01:01:08.811 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-19T01:01:09.027 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-19T01:01:09.054 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-19T01:01:09.094 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-19T01:01:09.122 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-pecan. 2024-04-19T01:01:09.148 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-19T01:01:09.168 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-base. 2024-04-19T01:01:09.188 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-19T01:01:09.195 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../030-ceph-base_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:09.244 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-base (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:09.380 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-webob. 2024-04-19T01:01:09.406 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-19T01:01:09.438 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-19T01:01:09.560 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-certifi. 2024-04-19T01:01:09.586 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-19T01:01:09.626 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-19T01:01:09.643 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-19T01:01:09.671 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-19T01:01:09.718 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-19T01:01:09.783 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-waitress. 2024-04-19T01:01:09.789 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-19T01:01:09.804 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-19T01:01:09.809 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-19T01:01:09.839 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-19T01:01:09.861 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-19T01:01:09.929 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-idna. 2024-04-19T01:01:09.955 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-19T01:01:09.987 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-19T01:01:10.071 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-cheroot. 2024-04-19T01:01:10.075 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-cheroot. 2024-04-19T01:01:10.099 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-19T01:01:10.101 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-19T01:01:10.136 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-tempita. 2024-04-19T01:01:10.141 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-19T01:01:10.146 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-19T01:01:10.163 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-19T01:01:10.203 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-19T01:01:10.290 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-urllib3. 2024-04-19T01:01:10.316 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-19T01:01:10.356 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-19T01:01:10.411 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-19T01:01:10.437 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-19T01:01:10.456 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-paste. 2024-04-19T01:01:10.477 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-19T01:01:10.482 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-19T01:01:10.491 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-19T01:01:10.518 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-19T01:01:10.522 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-19T01:01:10.558 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-19T01:01:10.676 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-requests. 2024-04-19T01:01:10.703 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-19T01:01:10.743 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-19T01:01:10.781 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-19T01:01:10.810 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-19T01:01:10.847 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-19T01:01:10.860 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-19T01:01:10.868 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-19T01:01:10.883 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-19T01:01:10.887 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-19T01:01:10.918 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-19T01:01:10.927 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-19T01:01:11.029 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-19T01:01:11.055 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-19T01:01:11.095 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-19T01:01:11.200 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-19T01:01:11.212 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-19T01:01:11.226 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-19T01:01:11.239 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-19T01:01:11.267 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-19T01:01:11.271 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-19T01:01:11.279 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-19T01:01:11.299 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-19T01:01:11.347 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-19T01:01:11.415 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-19T01:01:11.441 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:01:11.481 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking ceph-mgr-modules-core (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:11.528 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-tempora. 2024-04-19T01:01:11.554 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-19T01:01:11.582 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-webtest. 2024-04-19T01:01:11.586 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-19T01:01:11.609 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-19T01:01:11.649 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-19T01:01:11.660 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-tempora. 2024-04-19T01:01:11.687 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-19T01:01:11.726 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-19T01:01:11.836 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-19T01:01:11.863 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:11.894 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:11.952 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-pecan. 2024-04-19T01:01:11.978 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-19T01:01:12.027 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-19T01:01:12.081 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-portend. 2024-04-19T01:01:12.108 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-19T01:01:12.147 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-19T01:01:12.159 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-portend. 2024-04-19T01:01:12.180 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-19T01:01:12.186 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-19T01:01:12.207 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-19T01:01:12.246 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-19T01:01:12.287 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-19T01:01:12.399 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-certifi. 2024-04-19T01:01:12.425 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-19T01:01:12.433 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-19T01:01:12.460 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-19T01:01:12.465 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-19T01:01:12.500 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-19T01:01:12.549 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-openssl. 2024-04-19T01:01:12.576 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-19T01:01:12.619 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-19T01:01:12.657 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-19T01:01:12.684 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-19T01:01:12.732 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-19T01:01:12.777 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-idna. 2024-04-19T01:01:12.778 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-19T01:01:12.794 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-19T01:01:12.803 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-19T01:01:12.828 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-19T01:01:12.835 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-19T01:01:13.052 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package ceph-mgr. 2024-04-19T01:01:13.077 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-19T01:01:13.079 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:13.104 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-19T01:01:13.119 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking ceph-mgr (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:13.143 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-19T01:01:13.155 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-urllib3. 2024-04-19T01:01:13.165 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-natsort. 2024-04-19T01:01:13.170 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-19T01:01:13.192 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-19T01:01:13.213 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-19T01:01:13.231 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-19T01:01:13.472 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-natsort. 2024-04-19T01:01:13.499 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-19T01:01:13.530 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-19T01:01:13.543 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-logutils. 2024-04-19T01:01:13.546 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package ceph-mon. 2024-04-19T01:01:13.570 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-19T01:01:13.573 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:13.609 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-19T01:01:13.613 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking ceph-mon (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:13.646 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-requests. 2024-04-19T01:01:13.673 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-19T01:01:13.741 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-19T01:01:13.921 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-19T01:01:13.948 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-19T01:01:13.949 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-logutils. 2024-04-19T01:01:13.976 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-19T01:01:13.987 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-19T01:01:14.024 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-19T01:01:14.078 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-19T01:01:14.105 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-19T01:01:14.144 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-19T01:01:14.225 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package ceph-osd. 2024-04-19T01:01:14.251 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:14.257 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-mako. 2024-04-19T01:01:14.272 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-19T01:01:14.291 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking ceph-osd (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:14.307 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-19T01:01:14.327 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-19T01:01:14.354 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-19T01:01:14.394 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-19T01:01:14.431 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-19T01:01:14.458 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:01:14.498 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-mgr-modules-core (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:14.577 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-19T01:01:14.603 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-19T01:01:14.643 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-19T01:01:14.696 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-mako. 2024-04-19T01:01:14.724 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-19T01:01:14.763 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-19T01:01:14.895 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-19T01:01:14.922 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:14.946 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-19T01:01:14.961 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:14.973 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-19T01:01:15.013 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-19T01:01:15.091 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-19T01:01:15.118 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-19T01:01:15.166 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-19T01:01:15.214 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-19T01:01:15.241 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-19T01:01:15.272 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-19T01:01:15.316 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-webob. 2024-04-19T01:01:15.342 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-19T01:01:15.382 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-19T01:01:15.403 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package ceph. 2024-04-19T01:01:15.429 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../066-ceph_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:15.444 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-19T01:01:15.471 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-19T01:01:15.477 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking ceph (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:15.511 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-19T01:01:15.584 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-openssl. 2024-04-19T01:01:15.611 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-19T01:01:15.651 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-19T01:01:15.702 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-waitress. 2024-04-19T01:01:15.728 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-19T01:01:15.763 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-19T01:01:15.805 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-webob. 2024-04-19T01:01:15.832 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-19T01:01:15.871 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-19T01:01:15.880 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package ceph-fuse. 2024-04-19T01:01:15.907 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:15.955 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking ceph-fuse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:15.979 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-mgr. 2024-04-19T01:01:16.005 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:16.040 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-tempita. 2024-04-19T01:01:16.045 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-mgr (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:16.066 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-19T01:01:16.107 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-19T01:01:16.224 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-waitress. 2024-04-19T01:01:16.254 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-19T01:01:16.294 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-19T01:01:16.299 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package ceph-mds. 2024-04-19T01:01:16.326 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:16.366 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking ceph-mds (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:16.418 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-paste. 2024-04-19T01:01:16.444 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-19T01:01:16.449 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-mon. 2024-04-19T01:01:16.475 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:16.484 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-19T01:01:16.515 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-mon (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:16.611 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-tempita. 2024-04-19T01:01:16.638 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-19T01:01:16.685 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-19T01:01:16.813 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-19T01:01:16.821 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package cephadm. 2024-04-19T01:01:16.839 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-19T01:01:16.847 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../069-cephadm_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:16.879 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-19T01:01:16.895 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking cephadm (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:16.988 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-paste. 2024-04-19T01:01:17.015 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-19T01:01:17.055 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-19T01:01:17.136 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-osd. 2024-04-19T01:01:17.163 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:17.194 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-osd (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:17.199 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-19T01:01:17.226 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-19T01:01:17.266 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-19T01:01:17.273 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-19T01:01:17.300 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-19T01:01:17.348 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-19T01:01:17.374 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-19T01:01:17.401 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-19T01:01:17.450 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-19T01:01:17.527 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-webtest. 2024-04-19T01:01:17.553 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-19T01:01:17.593 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-19T01:01:17.692 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-19T01:01:17.707 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-19T01:01:17.752 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-19T01:01:17.758 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-19T01:01:17.779 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-19T01:01:17.819 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-19T01:01:17.846 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pecan. 2024-04-19T01:01:17.862 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-19T01:01:17.904 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-19T01:01:18.080 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-webtest. 2024-04-19T01:01:18.107 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-19T01:01:18.138 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-19T01:01:18.256 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph. 2024-04-19T01:01:18.259 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-certifi. 2024-04-19T01:01:18.271 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../066-ceph_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:18.287 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-19T01:01:18.306 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:18.326 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-19T01:01:18.408 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pecan. 2024-04-19T01:01:18.435 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-19T01:01:18.475 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-19T01:01:18.609 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-fuse. 2024-04-19T01:01:18.612 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-idna. 2024-04-19T01:01:18.636 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:18.639 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-19T01:01:18.670 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-19T01:01:18.684 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-fuse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:18.822 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-babel. 2024-04-19T01:01:18.829 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-certifi. 2024-04-19T01:01:18.849 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-19T01:01:18.856 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-19T01:01:18.888 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-19T01:01:18.895 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-19T01:01:18.990 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-urllib3. 2024-04-19T01:01:19.017 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-19T01:01:19.057 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-19T01:01:19.087 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-mds. 2024-04-19T01:01:19.114 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:19.154 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-mds (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:19.208 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-jinja2. 2024-04-19T01:01:19.224 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-idna. 2024-04-19T01:01:19.235 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-19T01:01:19.250 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-19T01:01:19.274 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-19T01:01:19.290 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-19T01:01:19.360 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-requests. 2024-04-19T01:01:19.386 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-19T01:01:19.427 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-19T01:01:19.524 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package cephadm. 2024-04-19T01:01:19.539 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../069-cephadm_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:19.573 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking cephadm (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:19.601 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-urllib3. 2024-04-19T01:01:19.611 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-19T01:01:19.628 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-19T01:01:19.637 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:01:19.668 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-19T01:01:19.677 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking ceph-mgr-cephadm (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:19.688 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-19T01:01:19.714 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-19T01:01:19.746 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-19T01:01:19.910 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-19T01:01:19.937 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-19T01:01:19.993 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-19T01:01:19.996 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-requests. 2024-04-19T01:01:20.005 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-19T01:01:20.023 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-19T01:01:20.024 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-19T01:01:20.032 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-19T01:01:20.040 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:01:20.062 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-19T01:01:20.071 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-19T01:01:20.074 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr-modules-core (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:20.279 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-19T01:01:20.295 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-19T01:01:20.338 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-19T01:01:20.357 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-19T01:01:20.374 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-routes. 2024-04-19T01:01:20.384 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-19T01:01:20.401 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-19T01:01:20.425 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-19T01:01:20.440 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-19T01:01:20.479 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-19T01:01:20.506 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:20.546 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:20.727 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-19T01:01:20.754 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:01:20.793 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-mgr-modules-core (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:20.794 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-19T01:01:20.821 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:01:20.840 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-19T01:01:20.860 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking ceph-mgr-dashboard (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:20.867 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-19T01:01:20.898 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-19T01:01:21.168 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-openssl. 2024-04-19T01:01:21.189 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-19T01:01:21.194 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-19T01:01:21.216 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:21.234 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-19T01:01:21.256 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:21.418 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-babel. 2024-04-19T01:01:21.446 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-19T01:01:21.485 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-19T01:01:21.571 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr. 2024-04-19T01:01:21.576 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-19T01:01:21.597 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:21.602 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-19T01:01:21.637 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:21.642 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-19T01:01:21.771 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-jinja2. 2024-04-19T01:01:21.798 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-19T01:01:21.829 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-19T01:01:21.945 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-openssl. 2024-04-19T01:01:21.972 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-19T01:01:22.011 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-19T01:01:22.058 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mon. 2024-04-19T01:01:22.084 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:22.124 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mon (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:22.132 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-19T01:01:22.149 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-19T01:01:22.159 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-19T01:01:22.176 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:01:22.198 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-19T01:01:22.216 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-mgr-cephadm (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:22.314 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-mgr. 2024-04-19T01:01:22.341 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:22.372 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-mgr (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:22.536 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-19T01:01:22.563 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-19T01:01:22.594 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-19T01:01:22.643 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-joblib. 2024-04-19T01:01:22.670 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-19T01:01:22.701 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-19T01:01:22.759 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-mon. 2024-04-19T01:01:22.786 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:22.826 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-mon (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:22.882 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-routes. 2024-04-19T01:01:22.894 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-osd. 2024-04-19T01:01:22.908 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-19T01:01:22.920 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:22.947 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-19T01:01:22.960 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-osd (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:23.284 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-19T01:01:23.311 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:01:23.405 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-19T01:01:23.409 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-mgr-dashboard (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:23.431 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-19T01:01:23.471 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-19T01:01:23.529 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-osd. 2024-04-19T01:01:23.556 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:23.595 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-osd (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:23.790 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-sklearn. 2024-04-19T01:01:23.817 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-19T01:01:23.857 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-19T01:01:23.989 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph. 2024-04-19T01:01:24.007 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../066-ceph_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:24.047 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:24.375 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-fuse. 2024-04-19T01:01:24.401 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:24.441 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-fuse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:24.523 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-19T01:01:24.551 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:01:24.675 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:24.828 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mds. 2024-04-19T01:01:24.840 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph. 2024-04-19T01:01:24.854 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:24.856 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-19T01:01:24.867 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../066-ceph_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:24.872 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-19T01:01:24.894 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mds (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:24.898 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:24.915 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-19T01:01:25.243 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-fuse. 2024-04-19T01:01:25.264 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package cephadm. 2024-04-19T01:01:25.270 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:25.291 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../069-cephadm_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:25.309 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-fuse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:25.331 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking cephadm (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:25.435 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-joblib. 2024-04-19T01:01:25.462 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-19T01:01:25.501 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-19T01:01:25.620 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-cachetools. 2024-04-19T01:01:25.636 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-19T01:01:25.650 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-19T01:01:25.663 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-mds. 2024-04-19T01:01:25.669 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-19T01:01:25.677 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-19T01:01:25.690 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:25.717 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-19T01:01:25.729 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-mds (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:25.847 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-19T01:01:25.874 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-19T01:01:25.913 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-19T01:01:25.998 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-19T01:01:26.019 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-19T01:01:26.025 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-19T01:01:26.035 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-19T01:01:26.064 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-19T01:01:26.077 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-19T01:01:26.107 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package cephadm. 2024-04-19T01:01:26.123 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../069-cephadm_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:26.158 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking cephadm (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:26.216 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-sklearn. 2024-04-19T01:01:26.244 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-19T01:01:26.283 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-19T01:01:26.375 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-19T01:01:26.402 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-19T01:01:26.442 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-19T01:01:26.451 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-19T01:01:26.478 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-19T01:01:26.517 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-19T01:01:26.745 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-rsa. 2024-04-19T01:01:26.772 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-19T01:01:26.811 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-19T01:01:26.845 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-19T01:01:26.872 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-19T01:01:26.912 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-19T01:01:26.950 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-19T01:01:26.966 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:01:27.066 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-babel. 2024-04-19T01:01:27.086 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:27.093 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-19T01:01:27.132 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-google-auth. 2024-04-19T01:01:27.133 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-19T01:01:27.159 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-19T01:01:27.198 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-19T01:01:27.419 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-jinja2. 2024-04-19T01:01:27.446 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-19T01:01:27.477 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-19T01:01:27.518 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-19T01:01:27.545 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-19T01:01:27.586 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-19T01:01:27.822 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-19T01:01:27.849 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:01:27.880 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr-cephadm (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:27.887 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-websocket. 2024-04-19T01:01:27.914 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-19T01:01:27.953 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-19T01:01:28.090 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-cachetools. 2024-04-19T01:01:28.117 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-19T01:01:28.133 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-babel. 2024-04-19T01:01:28.157 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-19T01:01:28.161 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-19T01:01:28.200 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-19T01:01:28.200 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-19T01:01:28.216 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-19T01:01:28.250 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-19T01:01:28.256 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-19T01:01:28.272 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-19T01:01:28.306 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-19T01:01:28.443 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-19T01:01:28.471 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-19T01:01:28.510 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-19T01:01:28.511 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-routes. 2024-04-19T01:01:28.519 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-jinja2. 2024-04-19T01:01:28.538 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-19T01:01:28.546 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-19T01:01:28.578 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-19T01:01:28.586 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-19T01:01:28.813 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-19T01:01:28.841 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-19T01:01:28.864 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-19T01:01:28.871 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-19T01:01:28.891 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:01:28.905 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-19T01:01:28.920 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-19T01:01:28.922 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr-dashboard (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:28.933 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:01:28.936 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:01:28.972 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-mgr-cephadm (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:28.978 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:29.116 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-rsa. 2024-04-19T01:01:29.140 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-19T01:01:29.174 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-19T01:01:29.309 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-19T01:01:29.331 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-19T01:01:29.336 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-19T01:01:29.358 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-19T01:01:29.375 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-19T01:01:29.397 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-19T01:01:29.512 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-google-auth. 2024-04-19T01:01:29.539 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-19T01:01:29.578 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-19T01:01:29.675 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-19T01:01:29.702 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-19T01:01:29.703 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-routes. 2024-04-19T01:01:29.730 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-19T01:01:29.741 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-19T01:01:29.769 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-19T01:01:29.898 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-19T01:01:29.926 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-19T01:01:29.965 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-19T01:01:30.019 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package jq. 2024-04-19T01:01:30.035 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-19T01:01:30.077 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-19T01:01:30.098 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-19T01:01:30.125 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:01:30.164 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-mgr-dashboard (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:30.211 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-19T01:01:30.227 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-19T01:01:30.269 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-19T01:01:30.285 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-websocket. 2024-04-19T01:01:30.312 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-19T01:01:30.330 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package socat. 2024-04-19T01:01:30.345 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-19T01:01:30.351 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-19T01:01:30.388 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-19T01:01:30.639 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-19T01:01:30.665 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-19T01:01:30.699 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package xmlstarlet. 2024-04-19T01:01:30.713 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-19T01:01:30.726 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-19T01:01:30.766 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-19T01:01:30.773 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-joblib. 2024-04-19T01:01:30.800 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-19T01:01:30.839 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-19T01:01:31.112 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package ceph-test. 2024-04-19T01:01:31.139 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../097-ceph-test_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:31.178 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking ceph-test (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:31.201 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-19T01:01:31.228 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-19T01:01:31.268 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-19T01:01:31.344 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-19T01:01:31.360 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:01:31.394 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-19T01:01:31.402 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:31.422 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-19T01:01:31.460 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-19T01:01:31.570 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-sklearn. 2024-04-19T01:01:31.597 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-19T01:01:31.637 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-19T01:01:31.705 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-19T01:01:31.733 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-19T01:01:31.772 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-19T01:01:31.939 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-joblib. 2024-04-19T01:01:31.955 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-19T01:01:31.989 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-19T01:01:32.142 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-19T01:01:32.169 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-19T01:01:32.208 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-19T01:01:32.321 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-19T01:01:32.337 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:01:32.342 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-19T01:01:32.370 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-19T01:01:32.409 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-19T01:01:32.428 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package jq. 2024-04-19T01:01:32.453 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-19T01:01:32.464 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:32.486 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-19T01:01:32.713 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-sklearn. 2024-04-19T01:01:32.739 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-19T01:01:32.739 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package socat. 2024-04-19T01:01:32.767 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-19T01:01:32.778 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-19T01:01:32.805 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-19T01:01:33.126 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package xmlstarlet. 2024-04-19T01:01:33.153 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-19T01:01:33.200 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-19T01:01:33.418 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-cachetools. 2024-04-19T01:01:33.445 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-19T01:01:33.485 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-19T01:01:33.486 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-19T01:01:33.503 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:01:33.564 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-test. 2024-04-19T01:01:33.580 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../097-ceph-test_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:33.616 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package ceph-volume. 2024-04-19T01:01:33.622 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-test (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:33.630 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:33.643 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:01:33.682 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking ceph-volume (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:33.746 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-19T01:01:33.773 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-19T01:01:33.804 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-19T01:01:34.003 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-19T01:01:34.030 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:34.069 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking libcephfs-dev (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:34.116 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-19T01:01:34.143 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-19T01:01:34.174 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-19T01:01:34.397 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-19T01:01:34.411 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-rsa. 2024-04-19T01:01:34.424 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-19T01:01:34.438 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-19T01:01:34.463 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-19T01:01:34.469 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-19T01:01:34.634 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-cachetools. 2024-04-19T01:01:34.661 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-19T01:01:34.708 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-19T01:01:34.758 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-google-auth. 2024-04-19T01:01:34.783 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-19T01:01:34.792 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-19T01:01:34.820 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-19T01:01:34.822 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-19T01:01:34.858 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-19T01:01:35.003 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-19T01:01:35.030 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-19T01:01:35.069 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-19T01:01:35.151 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-19T01:01:35.178 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-19T01:01:35.212 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package nvme-cli. 2024-04-19T01:01:35.217 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-19T01:01:35.240 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-19T01:01:35.279 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-19T01:01:35.364 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-19T01:01:35.389 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-19T01:01:35.422 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-19T01:01:35.512 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-websocket. 2024-04-19T01:01:35.539 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-19T01:01:35.578 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-19T01:01:35.675 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package pkg-config. 2024-04-19T01:01:35.701 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-rsa. 2024-04-19T01:01:35.703 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-19T01:01:35.728 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-19T01:01:35.741 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-19T01:01:35.758 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-19T01:01:35.890 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-19T01:01:35.917 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-19T01:01:35.956 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-19T01:01:35.970 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-19T01:01:35.986 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-19T01:01:36.019 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-19T01:01:36.079 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-google-auth. 2024-04-19T01:01:36.107 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-19T01:01:36.145 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-19T01:01:36.194 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-volume. 2024-04-19T01:01:36.221 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:01:36.260 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-volume (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:36.449 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-19T01:01:36.474 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-19T01:01:36.476 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-19T01:01:36.501 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-19T01:01:36.514 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-19T01:01:36.540 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-19T01:01:36.547 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-19T01:01:36.563 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:36.597 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-19T01:01:36.598 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libcephfs-dev (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:36.612 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:01:36.654 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:36.775 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-pastescript. 2024-04-19T01:01:36.803 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-19T01:01:36.841 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-19T01:01:36.893 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-websocket. 2024-04-19T01:01:36.917 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-19T01:01:36.921 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-19T01:01:36.944 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-19T01:01:36.960 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-19T01:01:36.965 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-19T01:01:36.983 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-19T01:01:36.992 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-19T01:01:37.031 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-19T01:01:37.169 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-pluggy. 2024-04-19T01:01:37.197 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-19T01:01:37.236 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-19T01:01:37.263 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-19T01:01:37.290 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-19T01:01:37.297 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-19T01:01:37.325 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-19T01:01:37.329 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-19T01:01:37.363 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-19T01:01:37.393 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-19T01:01:37.421 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-19T01:01:37.459 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-19T01:01:37.547 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-psutil. 2024-04-19T01:01:37.574 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-19T01:01:37.613 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-19T01:01:37.709 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package nvme-cli. 2024-04-19T01:01:37.737 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-19T01:01:37.767 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-19T01:01:37.937 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package jq. 2024-04-19T01:01:37.949 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-py. 2024-04-19T01:01:37.953 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-19T01:01:37.974 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-19T01:01:37.987 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-19T01:01:38.016 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-19T01:01:38.052 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-19T01:01:38.080 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:01:38.118 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:38.181 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package socat. 2024-04-19T01:01:38.197 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-19T01:01:38.239 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-19T01:01:38.290 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package pkg-config. 2024-04-19T01:01:38.318 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-19T01:01:38.344 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-pygments. 2024-04-19T01:01:38.365 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-19T01:01:38.371 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-19T01:01:38.410 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-19T01:01:38.437 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-19T01:01:38.465 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-19T01:01:38.504 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-19T01:01:38.551 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package xmlstarlet. 2024-04-19T01:01:38.579 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-19T01:01:38.646 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-19T01:01:38.684 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-19T01:01:38.700 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-19T01:01:38.743 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-19T01:01:38.832 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-19T01:01:38.856 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-19T01:01:38.860 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-19T01:01:38.872 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-19T01:01:38.890 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-19T01:01:38.905 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-19T01:01:39.005 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-test. 2024-04-19T01:01:39.021 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../097-ceph-test_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:39.063 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-test (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:39.151 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package jq. 2024-04-19T01:01:39.166 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-toml. 2024-04-19T01:01:39.167 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-19T01:01:39.194 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-19T01:01:39.209 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-19T01:01:39.233 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-19T01:01:39.246 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-19T01:01:39.274 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-19T01:01:39.321 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-19T01:01:39.462 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package socat. 2024-04-19T01:01:39.478 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-19T01:01:39.519 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-pytest. 2024-04-19T01:01:39.520 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-19T01:01:39.547 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-19T01:01:39.585 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-19T01:01:39.616 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-pastescript. 2024-04-19T01:01:39.643 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-19T01:01:39.682 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-19T01:01:39.848 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package xmlstarlet. 2024-04-19T01:01:39.876 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-19T01:01:39.880 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-simplejson. 2024-04-19T01:01:39.896 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-19T01:01:39.915 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-19T01:01:39.930 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-19T01:01:39.988 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-pluggy. 2024-04-19T01:01:40.016 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-19T01:01:40.054 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-19T01:01:40.199 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-19T01:01:40.215 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-19T01:01:40.257 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-19T01:01:40.277 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-test. 2024-04-19T01:01:40.294 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../097-ceph-test_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:40.335 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-test (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:40.366 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-psutil. 2024-04-19T01:01:40.393 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-19T01:01:40.432 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-19T01:01:40.760 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-py. 2024-04-19T01:01:40.787 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-19T01:01:40.826 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-19T01:01:40.953 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package radosgw. 2024-04-19T01:01:40.981 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../116-radosgw_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:41.036 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking radosgw (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:41.163 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-pygments. 2024-04-19T01:01:41.190 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-19T01:01:41.229 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-19T01:01:41.251 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-volume. 2024-04-19T01:01:41.278 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:01:41.318 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-volume (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:41.605 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-19T01:01:41.632 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:41.663 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libcephfs-dev (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:41.700 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-19T01:01:41.728 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-19T01:01:41.766 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-19T01:01:41.856 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package rbd-fuse. 2024-04-19T01:01:41.872 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:41.914 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking rbd-fuse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:41.966 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-19T01:01:41.993 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-19T01:01:42.032 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-19T01:01:42.061 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-toml. 2024-04-19T01:01:42.088 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-19T01:01:42.127 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-19T01:01:42.226 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-19T01:01:42.379 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-19T01:01:42.406 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-19T01:01:42.438 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-pytest. 2024-04-19T01:01:42.445 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-19T01:01:42.466 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-19T01:01:42.476 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-19T01:01:42.505 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-19T01:01:42.586 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-19T01:01:42.615 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-volume. 2024-04-19T01:01:42.642 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:01:42.681 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-volume (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:42.800 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-simplejson. 2024-04-19T01:01:42.816 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package nvme-cli. 2024-04-19T01:01:42.821 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-19T01:01:42.827 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-19T01:01:42.843 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-19T01:01:42.866 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-19T01:01:42.882 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-19T01:01:43.003 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-19T01:01:43.030 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:43.076 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libcephfs-dev (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:43.110 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-19T01:01:43.126 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-19T01:01:43.168 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-19T01:01:43.177 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-19T01:01:43.321 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package pkg-config. 2024-04-19T01:01:43.349 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-19T01:01:43.388 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-19T01:01:43.633 INFO:teuthology.orchestra.run.smithi155.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-19T01:01:43.646 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-19T01:01:43.657 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-19T01:01:43.674 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-19T01:01:43.685 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-19T01:01:43.704 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up cephadm (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:43.712 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-19T01:01:43.723 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-19T01:01:43.922 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package radosgw. 2024-04-19T01:01:43.950 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../116-radosgw_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:43.989 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking radosgw (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:44.025 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-19T01:01:44.053 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-19T01:01:44.100 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-19T01:01:44.102 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-19T01:01:44.129 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-19T01:01:44.160 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-19T01:01:44.221 INFO:teuthology.orchestra.run.smithi155.stdout:Adding system user cephadm....done 2024-04-19T01:01:44.384 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-19T01:01:44.447 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pastescript. 2024-04-19T01:01:44.474 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-19T01:01:44.479 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package nvme-cli. 2024-04-19T01:01:44.507 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-19T01:01:44.513 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-19T01:01:44.546 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-19T01:01:44.671 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-19T01:01:44.817 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package rbd-fuse. 2024-04-19T01:01:44.833 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:44.833 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pluggy. 2024-04-19T01:01:44.861 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-19T01:01:44.875 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking rbd-fuse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:44.899 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-19T01:01:44.937 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-19T01:01:44.976 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package pkg-config. 2024-04-19T01:01:45.003 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-19T01:01:45.042 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-19T01:01:45.046 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-19T01:01:45.178 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-psutil. 2024-04-19T01:01:45.205 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-19T01:01:45.236 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-19T01:01:45.303 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-19T01:01:45.306 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-19T01:01:45.320 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-19T01:01:45.361 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-19T01:01:45.412 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-19T01:01:45.557 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-19T01:01:45.597 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-py. 2024-04-19T01:01:45.627 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-19T01:01:45.663 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-19T01:01:45.671 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-19T01:01:45.683 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-19T01:01:45.789 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-19T01:01:45.790 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-19T01:01:45.817 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-19T01:01:45.856 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-19T01:01:45.983 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pygments. 2024-04-19T01:01:46.009 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-19T01:01:46.011 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-19T01:01:46.050 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-19T01:01:46.050 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-19T01:01:46.134 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pastescript. 2024-04-19T01:01:46.162 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-19T01:01:46.200 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-19T01:01:46.370 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-19T01:01:46.373 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-19T01:01:46.462 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pluggy. 2024-04-19T01:01:46.471 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-19T01:01:46.490 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-19T01:01:46.495 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-19T01:01:46.520 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-19T01:01:46.529 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-19T01:01:46.621 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-19T01:01:46.781 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-psutil. 2024-04-19T01:01:46.809 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-19T01:01:46.815 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-toml. 2024-04-19T01:01:46.815 INFO:teuthology.orchestra.run.smithi118.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-19T01:01:46.839 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-19T01:01:46.842 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-19T01:01:46.881 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-19T01:01:46.888 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-19T01:01:46.916 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up cephadm (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:47.123 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-19T01:01:47.176 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-py. 2024-04-19T01:01:47.193 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pytest. 2024-04-19T01:01:47.204 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-19T01:01:47.220 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-19T01:01:47.242 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-19T01:01:47.259 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-19T01:01:47.408 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-19T01:01:47.525 INFO:teuthology.orchestra.run.smithi118.stdout:Adding system user cephadm....done 2024-04-19T01:01:47.554 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pygments. 2024-04-19T01:01:47.571 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-simplejson. 2024-04-19T01:01:47.582 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-19T01:01:47.599 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-19T01:01:47.612 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-19T01:01:47.637 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-19T01:01:47.667 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-19T01:01:47.706 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-19T01:01:47.926 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-ceph-argparse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:47.931 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-19T01:01:47.947 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-19T01:01:47.990 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-19T01:01:48.041 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-19T01:01:48.069 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-19T01:01:48.107 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-19T01:01:48.175 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-19T01:01:48.335 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-19T01:01:48.435 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-toml. 2024-04-19T01:01:48.442 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-19T01:01:48.445 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-19T01:01:48.463 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-19T01:01:48.502 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-19T01:01:48.554 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-19T01:01:48.568 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-19T01:01:48.680 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-19T01:01:48.702 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package radosgw. 2024-04-19T01:01:48.730 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../116-radosgw_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:48.769 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking radosgw (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:48.813 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pytest. 2024-04-19T01:01:48.828 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-19T01:01:48.841 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-19T01:01:48.879 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-19T01:01:49.130 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-19T01:01:49.164 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-19T01:01:49.224 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-simplejson. 2024-04-19T01:01:49.252 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-19T01:01:49.255 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-19T01:01:49.290 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-19T01:01:49.557 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-19T01:01:49.566 INFO:teuthology.orchestra.run.smithi155.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-19T01:01:49.571 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package rbd-fuse. 2024-04-19T01:01:49.577 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-19T01:01:49.599 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:49.602 INFO:teuthology.orchestra.run.smithi155.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-19T01:01:49.605 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-19T01:01:49.638 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking rbd-fuse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:49.643 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-19T01:01:49.667 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-19T01:01:49.793 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-19T01:01:49.917 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-19T01:01:49.934 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-19T01:01:50.161 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-19T01:01:50.176 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-19T01:01:50.193 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-19T01:01:50.262 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-19T01:01:50.311 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-19T01:01:50.406 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package radosgw. 2024-04-19T01:01:50.434 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../116-radosgw_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:50.469 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-19T01:01:50.471 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking radosgw (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:50.521 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-19T01:01:50.565 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-19T01:01:50.713 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-19T01:01:50.814 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-19T01:01:50.893 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-19T01:01:50.958 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-19T01:01:51.115 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up zip (3.0-12build2) ... 2024-04-19T01:01:51.257 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-19T01:01:51.274 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package rbd-fuse. 2024-04-19T01:01:51.291 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:01:51.315 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-19T01:01:51.324 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking rbd-fuse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:51.330 INFO:teuthology.orchestra.run.smithi110.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-19T01:01:51.405 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up cephadm (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:51.574 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-ceph-argparse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:51.623 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-19T01:01:51.833 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-19T01:01:51.930 INFO:teuthology.orchestra.run.smithi110.stdout:Adding system user cephadm....done 2024-04-19T01:01:51.931 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-19T01:01:51.942 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-19T01:01:51.959 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-19T01:01:52.057 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-19T01:01:52.085 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-19T01:01:52.086 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-19T01:01:52.202 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-19T01:01:52.211 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-19T01:01:52.326 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-19T01:01:52.336 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-19T01:01:52.337 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-19T01:01:52.462 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-19T01:01:52.479 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-19T01:01:52.622 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-19T01:01:52.649 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-19T01:01:52.748 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-19T01:01:52.754 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-19T01:01:52.874 INFO:teuthology.orchestra.run.smithi118.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-19T01:01:52.901 INFO:teuthology.orchestra.run.smithi118.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-19T01:01:52.965 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-19T01:01:53.007 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-19T01:01:53.070 INFO:teuthology.orchestra.run.smithi033.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-19T01:01:53.092 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-19T01:01:53.139 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-19T01:01:53.142 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up cephadm (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:53.275 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-19T01:01:53.381 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-19T01:01:53.384 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-19T01:01:53.467 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-19T01:01:53.593 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-19T01:01:53.650 INFO:teuthology.orchestra.run.smithi033.stdout:Adding system user cephadm....done 2024-04-19T01:01:53.670 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-19T01:01:53.766 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-19T01:01:53.798 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-19T01:01:53.861 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-19T01:01:54.005 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-19T01:01:54.059 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-19T01:01:54.074 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-19T01:01:54.129 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-19T01:01:54.256 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-19T01:01:54.317 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-19T01:01:54.427 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-19T01:01:54.430 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up zip (3.0-12build2) ... 2024-04-19T01:01:54.443 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-19T01:01:54.541 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-19T01:01:54.556 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-19T01:01:54.670 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-19T01:01:54.703 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-19T01:01:54.792 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-19T01:01:54.796 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-ceph-common (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:54.962 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-19T01:01:55.044 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-19T01:01:55.088 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-19T01:01:55.089 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-idna (3.3-1) ... 2024-04-19T01:01:55.216 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-19T01:01:55.295 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-19T01:01:55.398 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-19T01:01:55.407 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-19T01:01:55.492 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-19T01:01:55.545 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-ceph-argparse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:55.618 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-19T01:01:55.775 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-19T01:01:55.805 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-19T01:01:55.811 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-19T01:01:55.931 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-19T01:01:56.018 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-19T01:01:56.057 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-19T01:01:56.095 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-19T01:01:56.183 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-19T01:01:56.320 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-19T01:01:56.444 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-19T01:01:56.463 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-19T01:01:56.570 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-19T01:01:56.571 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-19T01:01:56.706 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-19T01:01:56.780 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T01:01:56.815 INFO:teuthology.orchestra.run.smithi110.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-19T01:01:56.839 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-19T01:01:56.850 INFO:teuthology.orchestra.run.smithi110.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-19T01:01:56.889 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-19T01:01:56.931 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-19T01:01:57.073 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-19T01:01:57.090 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-19T01:01:57.106 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-19T01:01:57.157 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-19T01:01:57.283 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-19T01:01:57.349 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-ceph-argparse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:57.383 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-19T01:01:57.434 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-19T01:01:57.441 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-19T01:01:57.567 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-19T01:01:57.675 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-19T01:01:57.726 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-19T01:01:57.727 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-19T01:01:57.801 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-19T01:01:57.835 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-19T01:01:57.927 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-19T01:01:57.977 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-19T01:01:57.994 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-19T01:01:58.061 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-19T01:01:58.078 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-19T01:01:58.120 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-ceph-common (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:01:58.237 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T01:01:58.320 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-19T01:01:58.354 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-19T01:01:58.371 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up zip (3.0-12build2) ... 2024-04-19T01:01:58.396 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-idna (3.3-1) ... 2024-04-19T01:01:58.497 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-19T01:01:58.639 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-19T01:01:58.696 INFO:teuthology.orchestra.run.smithi033.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-19T01:01:58.705 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-19T01:01:58.728 INFO:teuthology.orchestra.run.smithi033.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-19T01:01:58.783 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-19T01:01:58.890 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-19T01:01:59.087 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-19T01:01:59.158 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-19T01:01:59.198 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-19T01:01:59.417 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-19T01:01:59.458 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-19T01:01:59.584 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-19T01:01:59.592 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-19T01:01:59.668 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-19T01:01:59.709 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-19T01:01:59.718 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-19T01:01:59.911 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-19T01:01:59.936 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-19T01:01:59.995 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-19T01:02:00.011 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-19T01:02:00.045 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-19T01:02:00.129 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T01:02:00.262 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-19T01:02:00.263 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-19T01:02:00.330 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-19T01:02:00.378 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-19T01:02:00.455 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-19T01:02:00.523 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-19T01:02:00.564 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up zip (3.0-12build2) ... 2024-04-19T01:02:00.613 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-19T01:02:00.674 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-19T01:02:00.690 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-19T01:02:00.748 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-19T01:02:00.800 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-19T01:02:00.981 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-19T01:02:01.049 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-19T01:02:01.109 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-19T01:02:01.282 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-19T01:02:01.300 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-19T01:02:01.366 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-19T01:02:01.368 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-19T01:02:01.426 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-19T01:02:01.577 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-19T01:02:01.609 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-19T01:02:01.636 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T01:02:01.659 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-19T01:02:01.666 INFO:teuthology.orchestra.run.smithi155.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-19T01:02:01.735 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-19T01:02:01.754 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-19T01:02:01.870 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-19T01:02:01.895 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-19T01:02:01.909 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-19T01:02:02.019 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-ceph-common (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:02.021 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-19T01:02:02.056 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-19T01:02:02.154 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-19T01:02:02.298 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-19T01:02:02.303 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-idna (3.3-1) ... 2024-04-19T01:02:02.406 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-19T01:02:02.563 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-19T01:02:02.566 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-19T01:02:02.613 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-19T01:02:02.673 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-19T01:02:02.806 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-19T01:02:02.812 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-19T01:02:02.825 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-19T01:02:03.067 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T01:02:03.076 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-19T01:02:03.183 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-19T01:02:03.193 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-19T01:02:03.344 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-19T01:02:03.453 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-19T01:02:03.476 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-19T01:02:03.504 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-19T01:02:03.762 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-19T01:02:03.805 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-19T01:02:03.827 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-19T01:02:03.835 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-19T01:02:03.889 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-19T01:02:04.053 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T01:02:04.081 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-19T01:02:04.087 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-19T01:02:04.163 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-19T01:02:04.173 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-19T01:02:04.207 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-19T01:02:04.213 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-ceph-common (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:04.422 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-19T01:02:04.490 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-19T01:02:04.514 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-idna (3.3-1) ... 2024-04-19T01:02:04.548 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-19T01:02:04.592 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-19T01:02:04.674 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-19T01:02:04.775 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-19T01:02:04.832 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-19T01:02:04.868 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-19T01:02:04.901 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-19T01:02:04.967 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-19T01:02:05.027 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-19T01:02:05.124 INFO:teuthology.orchestra.run.smithi118.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-19T01:02:05.201 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-19T01:02:05.229 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-19T01:02:05.362 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-19T01:02:05.444 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T01:02:05.463 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-19T01:02:05.570 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-19T01:02:05.614 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-19T01:02:05.831 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-19T01:02:05.880 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-19T01:02:05.899 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-19T01:02:06.062 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-19T01:02:06.090 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-19T01:02:06.139 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-19T01:02:06.216 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-19T01:02:06.251 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-19T01:02:06.331 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T01:02:06.352 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up librados2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:06.398 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-19T01:02:06.456 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-19T01:02:06.470 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up libsqlite3-mod-ceph (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:06.475 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T01:02:06.596 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-19T01:02:06.601 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-19T01:02:06.649 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-19T01:02:06.707 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-19T01:02:06.834 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-19T01:02:06.900 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-19T01:02:06.919 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-19T01:02:06.932 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up libcephfs2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:06.960 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-19T01:02:07.058 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up libradosstriper1 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:07.153 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-19T01:02:07.184 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-19T01:02:07.229 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-19T01:02:07.261 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-19T01:02:07.278 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-19T01:02:07.444 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up librbd1 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:07.480 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-19T01:02:07.520 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-19T01:02:07.545 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-19T01:02:07.553 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up ceph-mgr-modules-core (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:07.606 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-19T01:02:07.671 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-19T01:02:07.688 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up ceph-fuse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:07.788 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T01:02:07.865 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-19T01:02:07.900 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-19T01:02:07.900 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-19T01:02:07.906 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-19T01:02:07.956 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-19T01:02:08.133 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-19T01:02:08.207 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-19T01:02:08.257 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-19T01:02:08.381 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up libcephfs-dev (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:08.450 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-19T01:02:08.486 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-19T01:02:08.507 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-rados (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:08.524 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-19T01:02:08.633 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up librgw2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:08.718 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-19T01:02:08.776 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-19T01:02:08.871 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-19T01:02:08.977 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-19T01:02:09.050 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-19T01:02:09.156 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-19T01:02:09.176 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-19T01:02:09.220 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-19T01:02:09.266 INFO:teuthology.orchestra.run.smithi110.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-19T01:02:09.471 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-19T01:02:09.511 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-19T01:02:09.533 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-19T01:02:09.597 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-19T01:02:09.637 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-19T01:02:09.659 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up librados2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:09.777 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libsqlite3-mod-ceph (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:09.878 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-19T01:02:09.890 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-19T01:02:09.988 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-rbd (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:09.991 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-19T01:02:10.030 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-19T01:02:10.114 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up rbd-fuse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:10.189 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libcephfs2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:10.240 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-rgw (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:10.275 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-19T01:02:10.298 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-19T01:02:10.315 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libradosstriper1 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:10.365 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-cephfs (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:10.424 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-19T01:02:10.441 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-19T01:02:10.492 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up ceph-common (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:10.576 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-19T01:02:10.683 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T01:02:10.717 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up librbd1 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:10.809 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-19T01:02:10.843 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-mgr-modules-core (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:10.844 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-19T01:02:10.970 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-19T01:02:10.983 INFO:teuthology.orchestra.run.smithi155.stdout:Adding group ceph....done 2024-04-19T01:02:11.096 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-19T01:02:11.152 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-19T01:02:11.185 INFO:teuthology.orchestra.run.smithi033.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-19T01:02:11.278 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-fuse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:11.385 INFO:teuthology.orchestra.run.smithi155.stdout:Adding system user ceph....done 2024-04-19T01:02:11.423 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-19T01:02:11.462 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-19T01:02:11.498 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-19T01:02:11.498 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-19T01:02:11.524 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-19T01:02:11.554 INFO:teuthology.orchestra.run.smithi155.stdout:Setting system user ceph properties....done 2024-04-19T01:02:11.566 INFO:teuthology.orchestra.run.smithi155.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-19T01:02:11.702 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-19T01:02:11.738 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-19T01:02:11.865 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-19T01:02:11.917 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-19T01:02:11.987 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libcephfs-dev (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:12.045 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-19T01:02:12.088 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-rados (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:12.131 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-19T01:02:12.177 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-19T01:02:12.189 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up librgw2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:12.294 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-19T01:02:12.315 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-19T01:02:12.399 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-19T01:02:12.525 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up ceph-test (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:12.537 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T01:02:12.652 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up radosgw (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:12.663 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-19T01:02:12.744 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-19T01:02:12.981 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-19T01:02:13.099 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-19T01:02:13.099 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-19T01:02:13.120 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-19T01:02:13.315 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-19T01:02:13.397 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-19T01:02:13.527 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-rbd (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:13.561 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up ceph-base (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:13.591 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-19T01:02:13.653 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up rbd-fuse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:13.717 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-19T01:02:13.732 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-19T01:02:13.779 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-rgw (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:13.833 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up librados2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:13.906 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-cephfs (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:13.924 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-19T01:02:13.960 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libsqlite3-mod-ceph (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:13.993 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-19T01:02:14.039 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-common (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:14.069 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-19T01:02:14.270 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-19T01:02:14.364 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libcephfs2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:14.439 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up ceph-mds (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:14.489 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libradosstriper1 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:14.615 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-19T01:02:14.631 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-19T01:02:14.648 INFO:teuthology.orchestra.run.smithi118.stdout:Adding group ceph....done 2024-04-19T01:02:14.669 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-19T01:02:14.669 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-19T01:02:14.892 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up librbd1 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:14.999 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-19T01:02:15.035 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr-modules-core (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:15.058 INFO:teuthology.orchestra.run.smithi118.stdout:Adding system user ceph....done 2024-04-19T01:02:15.161 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-fuse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:15.174 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up ceph-mgr (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:15.227 INFO:teuthology.orchestra.run.smithi118.stdout:Setting system user ceph properties....done 2024-04-19T01:02:15.240 INFO:teuthology.orchestra.run.smithi118.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-19T01:02:15.284 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-19T01:02:15.373 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-19T01:02:15.385 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-19T01:02:15.385 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-19T01:02:15.426 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-19T01:02:15.426 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-19T01:02:15.654 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-19T01:02:15.708 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-19T01:02:15.788 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up librados2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:15.870 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libcephfs-dev (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:15.914 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libsqlite3-mod-ceph (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:15.949 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up ceph-osd (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:15.988 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-rados (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:16.031 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-19T01:02:16.114 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up librgw2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:16.240 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-19T01:02:16.241 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-test (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:16.293 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-19T01:02:16.293 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-19T01:02:16.301 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libcephfs2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:16.391 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up radosgw (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:16.427 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libradosstriper1 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:16.536 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-19T01:02:16.761 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up ceph-mgr-k8sevents (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:16.822 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up librbd1 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:16.861 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:16.907 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-19T01:02:16.907 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-19T01:02:16.947 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-mgr-modules-core (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:17.013 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up ceph-mon (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:17.073 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-fuse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:17.223 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-19T01:02:17.223 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-19T01:02:17.292 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-19T01:02:17.293 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-19T01:02:17.393 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-base (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:17.502 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-rbd (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:17.612 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up rbd-fuse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:17.712 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-rgw (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:17.714 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up ceph-mgr-cephadm (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:17.746 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-19T01:02:17.766 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libcephfs-dev (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:17.830 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-cephfs (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:17.857 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up ceph (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:17.917 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-rados (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:17.956 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-common (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:17.983 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up ceph-mgr-dashboard (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:18.043 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up librgw2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:18.125 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up ceph-volume (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:18.169 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-19T01:02:18.237 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-mds (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:18.473 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-19T01:02:18.473 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-19T01:02:18.473 INFO:teuthology.orchestra.run.smithi110.stdout:Adding group ceph....done 2024-04-19T01:02:18.587 INFO:teuthology.orchestra.run.smithi155.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-19T01:02:18.679 INFO:teuthology.orchestra.run.smithi155.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-19T01:02:18.826 INFO:teuthology.orchestra.run.smithi110.stdout:Adding system user ceph....done 2024-04-19T01:02:18.831 INFO:teuthology.orchestra.run.smithi155.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-19T01:02:18.980 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-mgr (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:18.985 INFO:teuthology.orchestra.run.smithi110.stdout:Setting system user ceph properties....done 2024-04-19T01:02:19.004 INFO:teuthology.orchestra.run.smithi110.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-19T01:02:19.138 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-19T01:02:19.198 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-19T01:02:19.198 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-19T01:02:19.468 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-19T01:02:19.472 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-rbd (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:19.573 INFO:teuthology.orchestra.run.smithi155.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-19T01:02:19.574 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up rbd-fuse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:19.656 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-osd (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:19.691 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-rgw (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:19.801 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-cephfs (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:19.936 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-common (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:19.948 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-test (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:19.967 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-19T01:02:19.967 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-19T01:02:20.049 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up radosgw (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:20.227 INFO:teuthology.orchestra.run.smithi155.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-19T01:02:20.233 DEBUG:teuthology.parallel:result is None 2024-04-19T01:02:20.453 INFO:teuthology.orchestra.run.smithi033.stdout:Adding group ceph....done 2024-04-19T01:02:20.491 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-mgr-k8sevents (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:20.562 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-19T01:02:20.563 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-19T01:02:20.651 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:20.827 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-mon (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:20.846 INFO:teuthology.orchestra.run.smithi033.stdout:Adding system user ceph....done 2024-04-19T01:02:21.000 INFO:teuthology.orchestra.run.smithi033.stdout:Setting system user ceph properties....done 2024-04-19T01:02:21.014 INFO:teuthology.orchestra.run.smithi033.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-19T01:02:21.033 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-19T01:02:21.034 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-19T01:02:21.042 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-base (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:21.161 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-19T01:02:21.377 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-19T01:02:21.538 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-19T01:02:21.604 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-mgr-cephadm (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:21.884 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:21.886 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mds (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:22.037 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-test (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:22.113 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-19T01:02:22.113 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-19T01:02:22.132 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-mgr-dashboard (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:22.205 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up radosgw (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:22.501 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-volume (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:22.571 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:22.686 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-19T01:02:22.686 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-19T01:02:22.783 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-19T01:02:22.784 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-19T01:02:22.952 INFO:teuthology.orchestra.run.smithi118.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-19T01:02:23.053 INFO:teuthology.orchestra.run.smithi118.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-19T01:02:23.206 INFO:teuthology.orchestra.run.smithi118.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-19T01:02:23.215 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-base (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:23.281 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-osd (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:23.563 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-19T01:02:23.563 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-19T01:02:23.574 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-19T01:02:23.980 INFO:teuthology.orchestra.run.smithi118.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-19T01:02:23.992 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-mds (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:24.099 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr-k8sevents (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:24.217 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:24.226 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-19T01:02:24.227 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-19T01:02:24.343 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mon (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:24.560 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-19T01:02:24.560 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-19T01:02:24.637 INFO:teuthology.orchestra.run.smithi118.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-19T01:02:24.642 DEBUG:teuthology.parallel:result is None 2024-04-19T01:02:24.694 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-mgr (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:25.038 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr-cephadm (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:25.164 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:25.214 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-19T01:02:25.214 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-19T01:02:25.290 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr-dashboard (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:25.441 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-volume (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:25.671 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-osd (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:25.828 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-19T01:02:25.954 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-19T01:02:25.958 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-19T01:02:25.958 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-19T01:02:26.106 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-19T01:02:26.456 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-mgr-k8sevents (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:26.582 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:26.708 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-mon (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:26.924 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-19T01:02:26.924 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-19T01:02:27.223 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-19T01:02:27.417 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-mgr-cephadm (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:27.569 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:27.695 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-mgr-dashboard (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:27.853 INFO:teuthology.orchestra.run.smithi110.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-19T01:02:27.857 DEBUG:teuthology.parallel:result is None 2024-04-19T01:02:27.887 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-volume (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:02:28.249 INFO:teuthology.orchestra.run.smithi033.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-19T01:02:28.366 INFO:teuthology.orchestra.run.smithi033.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-19T01:02:28.510 INFO:teuthology.orchestra.run.smithi033.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-19T01:02:29.569 INFO:teuthology.orchestra.run.smithi033.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-19T01:02:30.220 INFO:teuthology.orchestra.run.smithi033.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-19T01:02:30.224 DEBUG:teuthology.parallel:result is None 2024-04-19T01:02:30.225 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8b4ab94f9c30da63487fa2f241ce12f483378860 2024-04-19T01:02:30.444 DEBUG:teuthology.orchestra.run.smithi033:> dpkg-query -W -f '${Version}' ceph 2024-04-19T01:02:30.464 INFO:teuthology.orchestra.run.smithi033.stdout:19.0.0-3154-g8b4ab94f-1jammy 2024-04-19T01:02:30.464 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3154-g8b4ab94f-1jammy 2024-04-19T01:02:30.464 INFO:teuthology.task.install:The correct ceph version 19.0.0-3154-g8b4ab94f-1jammy is installed. 2024-04-19T01:02:30.466 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8b4ab94f9c30da63487fa2f241ce12f483378860 2024-04-19T01:02:30.681 DEBUG:teuthology.orchestra.run.smithi110:> dpkg-query -W -f '${Version}' ceph 2024-04-19T01:02:30.701 INFO:teuthology.orchestra.run.smithi110.stdout:19.0.0-3154-g8b4ab94f-1jammy 2024-04-19T01:02:30.701 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3154-g8b4ab94f-1jammy 2024-04-19T01:02:30.701 INFO:teuthology.task.install:The correct ceph version 19.0.0-3154-g8b4ab94f-1jammy is installed. 2024-04-19T01:02:30.703 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8b4ab94f9c30da63487fa2f241ce12f483378860 2024-04-19T01:02:30.846 DEBUG:teuthology.orchestra.run.smithi118:> dpkg-query -W -f '${Version}' ceph 2024-04-19T01:02:30.866 INFO:teuthology.orchestra.run.smithi118.stdout:19.0.0-3154-g8b4ab94f-1jammy 2024-04-19T01:02:30.866 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3154-g8b4ab94f-1jammy 2024-04-19T01:02:30.866 INFO:teuthology.task.install:The correct ceph version 19.0.0-3154-g8b4ab94f-1jammy is installed. 2024-04-19T01:02:30.868 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8b4ab94f9c30da63487fa2f241ce12f483378860 2024-04-19T01:02:31.010 DEBUG:teuthology.orchestra.run.smithi155:> dpkg-query -W -f '${Version}' ceph 2024-04-19T01:02:31.030 INFO:teuthology.orchestra.run.smithi155.stdout:19.0.0-3154-g8b4ab94f-1jammy 2024-04-19T01:02:31.030 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3154-g8b4ab94f-1jammy 2024-04-19T01:02:31.030 INFO:teuthology.task.install:The correct ceph version 19.0.0-3154-g8b4ab94f-1jammy is installed. 2024-04-19T01:02:31.033 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-19T01:02:31.033 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-19T01:02:31.033 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-19T01:02:31.048 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-19T01:02:31.048 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-19T01:02:31.063 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-04-19T01:02:31.063 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-19T01:02:31.077 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-19T01:02:31.078 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-19T01:02:31.091 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-19T01:02:31.091 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-19T01:02:31.091 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/daemon-helper 2024-04-19T01:02:31.104 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-19T01:02:31.159 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-19T01:02:31.160 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/daemon-helper 2024-04-19T01:02:31.173 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-19T01:02:31.229 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-04-19T01:02:31.229 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/daemon-helper 2024-04-19T01:02:31.242 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-19T01:02:31.298 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-19T01:02:31.298 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/daemon-helper 2024-04-19T01:02:31.314 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-19T01:02:31.367 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-19T01:02:31.367 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-19T01:02:31.367 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-19T01:02:31.381 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-19T01:02:31.436 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-19T01:02:31.436 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-19T01:02:31.449 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-19T01:02:31.504 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-04-19T01:02:31.504 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-19T01:02:31.518 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-19T01:02:31.574 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-19T01:02:31.575 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-19T01:02:31.588 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-19T01:02:31.643 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-19T01:02:31.643 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-19T01:02:31.643 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/stdin-killer 2024-04-19T01:02:31.657 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-19T01:02:31.711 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-19T01:02:31.711 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/stdin-killer 2024-04-19T01:02:31.725 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-19T01:02:31.781 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-04-19T01:02:31.781 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/stdin-killer 2024-04-19T01:02:31.796 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-19T01:02:31.850 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-19T01:02:31.850 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/stdin-killer 2024-04-19T01:02:31.863 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-19T01:02:31.919 INFO:teuthology.run_tasks:Running task ceph... 2024-04-19T01:02:32.007 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-19T01:02:32.008 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 777 /var/log/ceph 2024-04-19T01:02:32.010 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 777 /var/log/ceph 2024-04-19T01:02:32.012 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 777 /var/log/ceph 2024-04-19T01:02:32.014 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 777 /var/log/ceph 2024-04-19T01:02:32.028 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-19T01:02:32.028 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-19T01:02:32.065 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-19T01:02:32.067 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-19T01:02:32.072 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-19T01:02:32.083 INFO:tasks.ceph:Creating extra log directories... 2024-04-19T01:02:32.084 DEBUG:teuthology.orchestra.run.smithi033:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-19T01:02:32.121 DEBUG:teuthology.orchestra.run.smithi110:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-19T01:02:32.123 DEBUG:teuthology.orchestra.run.smithi118:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-19T01:02:32.125 DEBUG:teuthology.orchestra.run.smithi155:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-19T01:02:32.151 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-19T01:02:32.152 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-19T01:02:32.152 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663223', 'branch': 'wip-yuri5-testing-2024-04-17-1400', 'description': 'rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7663223', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri5-testing-2024-04-17-1400'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': '8b4ab94f9c30da63487fa2f241ce12f483378860'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8b4ab94f9c30da63487fa2f241ce12f483378860'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri5-testing-2024-04-17-1400', 'sha1': '8b4ab94f9c30da63487fa2f241ce12f483378860'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'], ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'], ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'], ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0']], 'seed': 1937, 'sha1': '8b4ab94f9c30da63487fa2f241ce12f483378860', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri5-testing-2024-04-17-1400', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_8b4ab94f9c30da63487fa2f241ce12f483378860/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '8b4ab94f9c30da63487fa2f241ce12f483378860', 'targets': {'smithi033.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLpj9Qk/UG0JRgamdP9/Iob2qWzoJv6O9SKRCTlLstPpnJX2lcDw5VMd2tdADQnRV0XFb8kkOmy/hZOWz91kKvA=', 'smithi110.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMpT4SWuPk3vehHOSyYehjC5rmtrs9bqSDnqYxND2vcHtiEpikXMu814iob8v7tQQX3zntntxYotfbUXKXYd85A=', 'smithi118.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFDPX1v+vTbcLofsXdEKMfDHoJAKbxRJZMaJtKb4xIkhtkXyQQxepVG8bmh95EH8s6MV4IwVR2BmzHuyM+6UE8A=', 'smithi155.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMQRDx7ifdlfs6P34mYZ0RVkeh/us0Z15Iz4nF0HRtCT+ftN1LRVRF4gmhst0dx6WW4tHqOwvAgRJtl/cPZWoR8='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': '8b4ab94f9c30da63487fa2f241ce12f483378860', 'cluster': 'ceph'}}, {'thrashosds': {'aggressive_pg_num_changes': False, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200}}, {'rados': {'clients': ['client.0'], 'ec_pool': True, 'erasure_code_profile': {'c': 2, 'crush-failure-domain': 'osd', 'k': 4, 'm': 3, 'name': 'shecprofile', 'plugin': 'shec'}, 'objects': 50, 'op_weights': {'append': 100, 'copy_from': 50, 'delete': 50, 'read': 100, 'rmattr': 25, 'rollback': 50, 'setattr': 25, 'snap_create': 50, 'snap_remove': 50, 'write': 0}, 'ops': 400, 'write_append_excl': False}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-18_22:49:05', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-19T01:02:32.152 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-19T01:02:32.181 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-19T01:02:32.183 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-19T01:02:32.186 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-19T01:02:32.192 DEBUG:teuthology.orchestra.run.smithi033:> sudo install -d -m0777 -- /var/run/ceph 2024-04-19T01:02:32.229 DEBUG:teuthology.orchestra.run.smithi110:> sudo install -d -m0777 -- /var/run/ceph 2024-04-19T01:02:32.232 DEBUG:teuthology.orchestra.run.smithi118:> sudo install -d -m0777 -- /var/run/ceph 2024-04-19T01:02:32.236 DEBUG:teuthology.orchestra.run.smithi155:> sudo install -d -m0777 -- /var/run/ceph 2024-04-19T01:02:32.250 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-19T01:02:32.250 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/scratch_devs of=/dev/stdout 2024-04-19T01:02:32.288 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-19T01:02:32.288 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_1 2024-04-19T01:02:32.338 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-19T01:02:32.339 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T01:02:32.339 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 869 Links: 1 2024-04-19T01:02:32.339 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T01:02:32.339 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-04-19 00:56:55.446847643 +0000 2024-04-19T01:02:32.339 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-04-19 00:56:55.170850095 +0000 2024-04-19T01:02:32.339 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-04-19 00:56:55.170850095 +0000 2024-04-19T01:02:32.339 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: - 2024-04-19T01:02:32.339 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-19T01:02:32.395 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-04-19T01:02:32.395 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-04-19T01:02:32.395 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000469233 s, 1.1 MB/s 2024-04-19T01:02:32.396 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-19T01:02:32.445 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_2 2024-04-19T01:02:32.492 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-19T01:02:32.492 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T01:02:32.493 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 886 Links: 1 2024-04-19T01:02:32.493 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T01:02:32.493 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-04-19 00:56:55.954843130 +0000 2024-04-19T01:02:32.493 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-04-19 00:56:55.682845546 +0000 2024-04-19T01:02:32.493 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-04-19 00:56:55.682845546 +0000 2024-04-19T01:02:32.493 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: - 2024-04-19T01:02:32.493 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-19T01:02:32.547 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-04-19T01:02:32.547 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-04-19T01:02:32.547 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000401901 s, 1.3 MB/s 2024-04-19T01:02:32.548 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-19T01:02:32.597 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_3 2024-04-19T01:02:32.644 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-19T01:02:32.644 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T01:02:32.644 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 911 Links: 1 2024-04-19T01:02:32.644 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T01:02:32.644 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-04-19 00:56:56.438838830 +0000 2024-04-19T01:02:32.644 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-04-19 00:56:56.190841034 +0000 2024-04-19T01:02:32.644 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-04-19 00:56:56.190841034 +0000 2024-04-19T01:02:32.644 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: - 2024-04-19T01:02:32.644 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-19T01:02:32.698 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-04-19T01:02:32.698 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-04-19T01:02:32.699 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000401388 s, 1.3 MB/s 2024-04-19T01:02:32.700 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-19T01:02:32.749 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_4 2024-04-19T01:02:32.796 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-19T01:02:32.796 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T01:02:32.796 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 921 Links: 1 2024-04-19T01:02:32.796 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T01:02:32.796 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-04-19 00:56:56.914834602 +0000 2024-04-19T01:02:32.796 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-04-19 00:56:56.650836947 +0000 2024-04-19T01:02:32.796 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-04-19 00:56:56.650836947 +0000 2024-04-19T01:02:32.796 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: - 2024-04-19T01:02:32.797 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-19T01:02:32.855 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-04-19T01:02:32.855 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-04-19T01:02:32.855 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000380598 s, 1.3 MB/s 2024-04-19T01:02:32.856 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-19T01:02:32.906 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-04-19T01:02:32.906 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-19T01:02:32.906 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/scratch_devs of=/dev/stdout 2024-04-19T01:02:32.912 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-19T01:02:32.912 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_1 2024-04-19T01:02:32.958 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-19T01:02:32.958 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T01:02:32.958 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 884 Links: 1 2024-04-19T01:02:32.958 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T01:02:32.958 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-04-19 00:56:59.569123618 +0000 2024-04-19T01:02:32.959 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-04-19 00:56:59.565123652 +0000 2024-04-19T01:02:32.959 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-04-19 00:56:59.565123652 +0000 2024-04-19T01:02:32.959 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2024-04-19T01:02:32.959 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-19T01:02:33.012 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-04-19T01:02:33.012 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-04-19T01:02:33.012 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000250336 s, 2.0 MB/s 2024-04-19T01:02:33.014 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-19T01:02:33.063 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_2 2024-04-19T01:02:33.109 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-19T01:02:33.109 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T01:02:33.109 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 900 Links: 1 2024-04-19T01:02:33.109 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T01:02:33.110 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-04-19 00:57:00.325116863 +0000 2024-04-19T01:02:33.110 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-04-19 00:57:00.077119079 +0000 2024-04-19T01:02:33.110 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-04-19 00:57:00.077119079 +0000 2024-04-19T01:02:33.110 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2024-04-19T01:02:33.110 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-19T01:02:33.165 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-04-19T01:02:33.165 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-04-19T01:02:33.165 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.0003063 s, 1.7 MB/s 2024-04-19T01:02:33.166 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-19T01:02:33.214 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_3 2024-04-19T01:02:33.261 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-19T01:02:33.261 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T01:02:33.261 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 918 Links: 1 2024-04-19T01:02:33.261 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T01:02:33.262 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-04-19 00:57:00.825112400 +0000 2024-04-19T01:02:33.262 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-04-19 00:57:00.561114755 +0000 2024-04-19T01:02:33.262 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-04-19 00:57:00.561114755 +0000 2024-04-19T01:02:33.262 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2024-04-19T01:02:33.262 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-19T01:02:33.316 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-04-19T01:02:33.316 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-04-19T01:02:33.316 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000314754 s, 1.6 MB/s 2024-04-19T01:02:33.317 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-19T01:02:33.367 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_4 2024-04-19T01:02:33.414 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-19T01:02:33.414 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T01:02:33.414 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 930 Links: 1 2024-04-19T01:02:33.414 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T01:02:33.414 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-04-19 00:57:01.065110257 +0000 2024-04-19T01:02:33.414 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-04-19 00:57:01.057110328 +0000 2024-04-19T01:02:33.414 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-04-19 00:57:01.057110328 +0000 2024-04-19T01:02:33.414 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2024-04-19T01:02:33.415 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-19T01:02:33.467 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-04-19T01:02:33.467 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-04-19T01:02:33.467 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000247577 s, 2.1 MB/s 2024-04-19T01:02:33.469 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-19T01:02:33.519 INFO:tasks.ceph:osd dev map: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-04-19T01:02:33.519 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-04-19T01:02:33.519 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/scratch_devs of=/dev/stdout 2024-04-19T01:02:33.525 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-19T01:02:33.525 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_1 2024-04-19T01:02:33.571 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-19T01:02:33.571 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T01:02:33.571 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 5h/5d Inode: 879 Links: 1 2024-04-19T01:02:33.571 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T01:02:33.571 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-04-19 00:56:48.947007540 +0000 2024-04-19T01:02:33.572 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-04-19 00:56:48.671010005 +0000 2024-04-19T01:02:33.572 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-04-19 00:56:48.671010005 +0000 2024-04-19T01:02:33.572 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: - 2024-04-19T01:02:33.572 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-19T01:02:33.625 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-04-19T01:02:33.625 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-04-19T01:02:33.625 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.000318768 s, 1.6 MB/s 2024-04-19T01:02:33.627 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-19T01:02:33.676 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_2 2024-04-19T01:02:33.723 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-19T01:02:33.723 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T01:02:33.723 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 5h/5d Inode: 896 Links: 1 2024-04-19T01:02:33.723 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T01:02:33.723 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-04-19 00:56:49.427003253 +0000 2024-04-19T01:02:33.723 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-04-19 00:56:49.163005611 +0000 2024-04-19T01:02:33.723 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-04-19 00:56:49.163005611 +0000 2024-04-19T01:02:33.724 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: - 2024-04-19T01:02:33.724 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-19T01:02:33.777 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-04-19T01:02:33.777 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-04-19T01:02:33.777 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.000296174 s, 1.7 MB/s 2024-04-19T01:02:33.779 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-19T01:02:33.828 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_3 2024-04-19T01:02:33.875 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-19T01:02:33.875 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T01:02:33.875 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 5h/5d Inode: 915 Links: 1 2024-04-19T01:02:33.875 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T01:02:33.875 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-04-19 00:56:49.942998643 +0000 2024-04-19T01:02:33.875 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-04-19 00:56:49.691000894 +0000 2024-04-19T01:02:33.875 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-04-19 00:56:49.691000894 +0000 2024-04-19T01:02:33.875 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: - 2024-04-19T01:02:33.876 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-19T01:02:33.928 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-04-19T01:02:33.929 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-04-19T01:02:33.929 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.000301742 s, 1.7 MB/s 2024-04-19T01:02:33.930 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-19T01:02:33.981 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_4 2024-04-19T01:02:34.027 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-19T01:02:34.027 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T01:02:34.027 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 5h/5d Inode: 931 Links: 1 2024-04-19T01:02:34.027 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T01:02:34.027 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-04-19 00:56:50.494993712 +0000 2024-04-19T01:02:34.027 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-04-19 00:56:50.234996036 +0000 2024-04-19T01:02:34.027 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-04-19 00:56:50.234996036 +0000 2024-04-19T01:02:34.027 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: - 2024-04-19T01:02:34.027 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-19T01:02:34.080 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-04-19T01:02:34.081 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-04-19T01:02:34.081 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.000301634 s, 1.7 MB/s 2024-04-19T01:02:34.082 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-19T01:02:34.132 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-04-19T01:02:34.132 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-19T01:02:34.132 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/scratch_devs of=/dev/stdout 2024-04-19T01:02:34.138 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-19T01:02:34.138 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_1 2024-04-19T01:02:34.186 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-19T01:02:34.187 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T01:02:34.187 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 877 Links: 1 2024-04-19T01:02:34.187 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T01:02:34.187 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-04-19 00:56:52.158933222 +0000 2024-04-19T01:02:34.187 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-04-19 00:56:51.878935723 +0000 2024-04-19T01:02:34.187 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-04-19 00:56:51.878935723 +0000 2024-04-19T01:02:34.187 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: - 2024-04-19T01:02:34.187 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-19T01:02:34.239 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-04-19T01:02:34.239 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-04-19T01:02:34.239 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000444923 s, 1.2 MB/s 2024-04-19T01:02:34.240 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-19T01:02:34.290 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_2 2024-04-19T01:02:34.336 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-19T01:02:34.336 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T01:02:34.336 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 889 Links: 1 2024-04-19T01:02:34.337 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T01:02:34.337 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-04-19 00:56:52.682928541 +0000 2024-04-19T01:02:34.337 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-04-19 00:56:52.418930900 +0000 2024-04-19T01:02:34.337 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-04-19 00:56:52.418930900 +0000 2024-04-19T01:02:34.337 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: - 2024-04-19T01:02:34.337 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-19T01:02:34.390 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-04-19T01:02:34.390 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-04-19T01:02:34.390 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.00032302 s, 1.6 MB/s 2024-04-19T01:02:34.391 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-19T01:02:34.441 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_3 2024-04-19T01:02:34.488 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-19T01:02:34.489 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T01:02:34.489 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 909 Links: 1 2024-04-19T01:02:34.489 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T01:02:34.489 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-04-19 00:56:53.250923467 +0000 2024-04-19T01:02:34.489 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-04-19 00:56:52.962926041 +0000 2024-04-19T01:02:34.489 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-04-19 00:56:52.962926041 +0000 2024-04-19T01:02:34.489 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: - 2024-04-19T01:02:34.489 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-19T01:02:34.543 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-04-19T01:02:34.543 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-04-19T01:02:34.543 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000419861 s, 1.2 MB/s 2024-04-19T01:02:34.544 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-19T01:02:34.594 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_4 2024-04-19T01:02:34.640 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-19T01:02:34.641 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T01:02:34.641 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 928 Links: 1 2024-04-19T01:02:34.641 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T01:02:34.641 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-04-19 00:56:53.746919037 +0000 2024-04-19T01:02:34.641 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-04-19 00:56:53.486921361 +0000 2024-04-19T01:02:34.641 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-04-19 00:56:53.486921361 +0000 2024-04-19T01:02:34.641 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: - 2024-04-19T01:02:34.641 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-19T01:02:34.694 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-04-19T01:02:34.695 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-04-19T01:02:34.695 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000325543 s, 1.6 MB/s 2024-04-19T01:02:34.696 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-19T01:02:34.749 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-04-19T01:02:34.749 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi033.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi110.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi118.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi155.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2024-04-19T01:02:34.749 INFO:tasks.ceph:Generating config... 2024-04-19T01:02:34.751 INFO:tasks.ceph:[osd] debug monc = 20 2024-04-19T01:02:34.751 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-19T01:02:34.751 INFO:tasks.ceph:[osd] bluestore compression algorithm = zlib 2024-04-19T01:02:34.751 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-04-19T01:02:34.751 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-19T01:02:34.751 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-04-19T01:02:34.751 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-19T01:02:34.751 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-19T01:02:34.751 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-19T01:02:34.751 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-19T01:02:34.751 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-19T01:02:34.751 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-19T01:02:34.752 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-19T01:02:34.752 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-19T01:02:34.752 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-04-19T01:02:34.752 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.3 2024-04-19T01:02:34.752 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-04-19T01:02:34.752 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-04-19T01:02:34.752 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-19T01:02:34.752 INFO:tasks.ceph:[osd] osd max backfills = 3 2024-04-19T01:02:34.752 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-04-19T01:02:34.752 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-04-19T01:02:34.752 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-04-19T01:02:34.752 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-19T01:02:34.752 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-04-19T01:02:34.752 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-04-19T01:02:34.752 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2024-04-19T01:02:34.753 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2024-04-19T01:02:34.753 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-04-19T01:02:34.753 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-04-19T01:02:34.753 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-04-19T01:02:34.753 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2024-04-19T01:02:34.753 INFO:tasks.ceph:[global] ms tcp read timeout = 5 2024-04-19T01:02:34.753 INFO:tasks.ceph:[global] osd_async_recovery_min_cost = 1 2024-04-19T01:02:34.753 INFO:tasks.ceph:[global] osd_object_clean_region_max_num_intervals = 1000 2024-04-19T01:02:34.753 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-19T01:02:34.753 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-19T01:02:34.753 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-19T01:02:34.753 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-19T01:02:34.753 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-19T01:02:34.753 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-04-19T01:02:34.754 INFO:tasks.ceph:Setting up mon.a... 2024-04-19T01:02:34.754 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-19T01:02:34.789 INFO:teuthology.orchestra.run.smithi033.stdout:creating /etc/ceph/ceph.keyring 2024-04-19T01:02:34.803 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-19T01:02:34.883 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-19T01:02:34.940 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.33'), ('mon.b', '172.21.15.110'), ('mon.c', '172.21.15.118')] 2024-04-19T01:02:34.940 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.33,172.21.15.110,172.21.15.118', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2024-04-19T01:02:34.941 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-19T01:02:34.941 DEBUG:teuthology.orchestra.run.smithi033:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-19T01:02:34.992 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.33 --add b 172.21.15.110 --add c 172.21.15.118 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-19T01:02:35.078 INFO:teuthology.orchestra.run.smithi033.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-19T01:02:35.078 INFO:teuthology.orchestra.run.smithi033.stdout:monmaptool: generated fsid f11bd2d5-4456-4ffa-83c0-722dddd22ad3 2024-04-19T01:02:35.078 INFO:teuthology.orchestra.run.smithi033.stdout:setting min_mon_release = quincy 2024-04-19T01:02:35.078 INFO:teuthology.orchestra.run.smithi033.stdout:epoch 0 2024-04-19T01:02:35.079 INFO:teuthology.orchestra.run.smithi033.stdout:fsid f11bd2d5-4456-4ffa-83c0-722dddd22ad3 2024-04-19T01:02:35.079 INFO:teuthology.orchestra.run.smithi033.stdout:last_changed 2024-04-19T01:02:35.082348+0000 2024-04-19T01:02:35.079 INFO:teuthology.orchestra.run.smithi033.stdout:created 2024-04-19T01:02:35.082348+0000 2024-04-19T01:02:35.079 INFO:teuthology.orchestra.run.smithi033.stdout:min_mon_release 17 (quincy) 2024-04-19T01:02:35.079 INFO:teuthology.orchestra.run.smithi033.stdout:election_strategy: 1 2024-04-19T01:02:35.079 INFO:teuthology.orchestra.run.smithi033.stdout:0: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.a 2024-04-19T01:02:35.079 INFO:teuthology.orchestra.run.smithi033.stdout:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2024-04-19T01:02:35.079 INFO:teuthology.orchestra.run.smithi033.stdout:2: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.c 2024-04-19T01:02:35.079 INFO:teuthology.orchestra.run.smithi033.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-19T01:02:35.080 DEBUG:teuthology.orchestra.run.smithi033:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-19T01:02:35.128 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID f11bd2d5-4456-4ffa-83c0-722dddd22ad3... 2024-04-19T01:02:35.132 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-19T01:02:35.172 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-19T01:02:35.175 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-19T01:02:35.177 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-19T01:02:35.202 INFO:teuthology.orchestra.run.smithi033.stdout:[global] 2024-04-19T01:02:35.203 INFO:teuthology.orchestra.run.smithi033.stdout: chdir = "" 2024-04-19T01:02:35.203 INFO:teuthology.orchestra.run.smithi033.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-19T01:02:35.203 INFO:teuthology.orchestra.run.smithi033.stdout: auth supported = cephx 2024-04-19T01:02:35.203 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:02:35.203 INFO:teuthology.orchestra.run.smithi033.stdout: filestore xattr use omap = true 2024-04-19T01:02:35.203 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:02:35.203 INFO:teuthology.orchestra.run.smithi033.stdout: mon clock drift allowed = 1.000 2024-04-19T01:02:35.203 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:02:35.203 INFO:teuthology.orchestra.run.smithi033.stdout: osd crush chooseleaf type = 0 2024-04-19T01:02:35.203 INFO:teuthology.orchestra.run.smithi033.stdout: auth debug = true 2024-04-19T01:02:35.204 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:02:35.204 INFO:teuthology.orchestra.run.smithi033.stdout: ms die on old message = true 2024-04-19T01:02:35.205 INFO:teuthology.orchestra.run.smithi033.stdout: ms die on bug = true 2024-04-19T01:02:35.205 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:02:35.205 INFO:teuthology.orchestra.run.smithi033.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-19T01:02:35.205 INFO:teuthology.orchestra.run.smithi033.stdout: mon pg warn max object skew = 0 2024-04-19T01:02:35.206 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:02:35.206 INFO:teuthology.orchestra.run.smithi033.stdout: # disable pg_autoscaler by default for new pools 2024-04-19T01:02:35.206 INFO:teuthology.orchestra.run.smithi033.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-19T01:02:35.206 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:02:35.206 INFO:teuthology.orchestra.run.smithi033.stdout: osd pool default size = 2 2024-04-19T01:02:35.206 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:02:35.206 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd allow primary affinity = true 2024-04-19T01:02:35.206 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd allow pg remap = true 2024-04-19T01:02:35.206 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on legacy crush tunables = false 2024-04-19T01:02:35.206 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on crush straw calc version zero = false 2024-04-19T01:02:35.206 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on no sortbitwise = false 2024-04-19T01:02:35.206 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on osd down out interval zero = false 2024-04-19T01:02:35.206 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on too few osds = false 2024-04-19T01:02:35.207 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-19T01:02:35.207 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-19T01:02:35.207 INFO:teuthology.orchestra.run.smithi033.stdout: mon_allow_pool_size_one = true 2024-04-19T01:02:35.207 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:02:35.207 INFO:teuthology.orchestra.run.smithi033.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-19T01:02:35.207 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:02:35.207 INFO:teuthology.orchestra.run.smithi033.stdout: osd default data pool replay window = 5 2024-04-19T01:02:35.207 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:02:35.207 INFO:teuthology.orchestra.run.smithi033.stdout: mon allow pool delete = true 2024-04-19T01:02:35.207 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:02:35.207 INFO:teuthology.orchestra.run.smithi033.stdout: mon cluster log file level = debug 2024-04-19T01:02:35.207 INFO:teuthology.orchestra.run.smithi033.stdout: debug asserts on shutdown = true 2024-04-19T01:02:35.207 INFO:teuthology.orchestra.run.smithi033.stdout: mon health detail to clog = false 2024-04-19T01:02:35.208 INFO:teuthology.orchestra.run.smithi033.stdout: mon host = "172.21.15.33,172.21.15.110,172.21.15.118" 2024-04-19T01:02:35.208 INFO:teuthology.orchestra.run.smithi033.stdout: mon client directed command retry = 5 2024-04-19T01:02:35.208 INFO:teuthology.orchestra.run.smithi033.stdout: mon election default strategy = 1 2024-04-19T01:02:35.208 INFO:teuthology.orchestra.run.smithi033.stdout: ms inject socket failures = 2500 2024-04-19T01:02:35.208 INFO:teuthology.orchestra.run.smithi033.stdout: ms tcp read timeout = 5 2024-04-19T01:02:35.208 INFO:teuthology.orchestra.run.smithi033.stdout: osd_async_recovery_min_cost = 1 2024-04-19T01:02:35.208 INFO:teuthology.orchestra.run.smithi033.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-04-19T01:02:35.208 INFO:teuthology.orchestra.run.smithi033.stdout: fsid = f11bd2d5-4456-4ffa-83c0-722dddd22ad3 2024-04-19T01:02:35.208 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:02:35.208 INFO:teuthology.orchestra.run.smithi033.stdout:[osd] 2024-04-19T01:02:35.208 INFO:teuthology.orchestra.run.smithi033.stdout: osd journal size = 100 2024-04-19T01:02:35.208 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:02:35.208 INFO:teuthology.orchestra.run.smithi033.stdout: osd scrub load threshold = 5.0 2024-04-19T01:02:35.209 INFO:teuthology.orchestra.run.smithi033.stdout: osd scrub max interval = 120 2024-04-19T01:02:35.209 INFO:teuthology.orchestra.run.smithi033.stdout: osd mclock profile = high_recovery_ops 2024-04-19T01:02:35.209 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:02:35.209 INFO:teuthology.orchestra.run.smithi033.stdout: osd recover clone overlap = true 2024-04-19T01:02:35.209 INFO:teuthology.orchestra.run.smithi033.stdout: osd recovery max chunk = 1048576 2024-04-19T01:02:35.209 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:02:35.209 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug shutdown = true 2024-04-19T01:02:35.209 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug op order = true 2024-04-19T01:02:35.209 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug verify stray on activate = true 2024-04-19T01:02:35.209 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug trim objects = true 2024-04-19T01:02:35.209 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:02:35.209 INFO:teuthology.orchestra.run.smithi033.stdout: osd open classes on start = true 2024-04-19T01:02:35.209 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug pg log writeout = true 2024-04-19T01:02:35.210 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:02:35.210 INFO:teuthology.orchestra.run.smithi033.stdout: osd deep scrub update digest min age = 30 2024-04-19T01:02:35.210 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:02:35.210 INFO:teuthology.orchestra.run.smithi033.stdout: osd map max advance = 10 2024-04-19T01:02:35.210 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:02:35.210 INFO:teuthology.orchestra.run.smithi033.stdout: journal zero on create = true 2024-04-19T01:02:35.210 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:02:35.210 INFO:teuthology.orchestra.run.smithi033.stdout: filestore ondisk finisher threads = 3 2024-04-19T01:02:35.210 INFO:teuthology.orchestra.run.smithi033.stdout: filestore apply finisher threads = 3 2024-04-19T01:02:35.210 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:02:35.210 INFO:teuthology.orchestra.run.smithi033.stdout: bdev debug aio = true 2024-04-19T01:02:35.210 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug misdirected ops = true 2024-04-19T01:02:35.210 INFO:teuthology.orchestra.run.smithi033.stdout: debug monc = 20 2024-04-19T01:02:35.210 INFO:teuthology.orchestra.run.smithi033.stdout: bluestore block size = 96636764160 2024-04-19T01:02:35.211 INFO:teuthology.orchestra.run.smithi033.stdout: bluestore compression algorithm = zlib 2024-04-19T01:02:35.211 INFO:teuthology.orchestra.run.smithi033.stdout: bluestore compression mode = aggressive 2024-04-19T01:02:35.214 INFO:teuthology.orchestra.run.smithi033.stdout: bluestore fsck on mount = True 2024-04-19T01:02:35.214 INFO:teuthology.orchestra.run.smithi033.stdout: bluestore zero block detection = True 2024-04-19T01:02:35.214 INFO:teuthology.orchestra.run.smithi033.stdout: debug bluefs = 20 2024-04-19T01:02:35.214 INFO:teuthology.orchestra.run.smithi033.stdout: debug bluestore = 20 2024-04-19T01:02:35.214 INFO:teuthology.orchestra.run.smithi033.stdout: debug ms = 1 2024-04-19T01:02:35.214 INFO:teuthology.orchestra.run.smithi033.stdout: debug osd = 20 2024-04-19T01:02:35.214 INFO:teuthology.orchestra.run.smithi033.stdout: debug rocksdb = 10 2024-04-19T01:02:35.214 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd backfillfull_ratio = 0.85 2024-04-19T01:02:35.214 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd full ratio = 0.9 2024-04-19T01:02:35.214 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd nearfull ratio = 0.8 2024-04-19T01:02:35.214 INFO:teuthology.orchestra.run.smithi033.stdout: osd blocked scrub grace period = 3600 2024-04-19T01:02:35.215 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug reject backfill probability = 0.3 2024-04-19T01:02:35.215 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug verify cached snaps = True 2024-04-19T01:02:35.216 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug verify missing on start = True 2024-04-19T01:02:35.216 INFO:teuthology.orchestra.run.smithi033.stdout: osd failsafe full ratio = 0.95 2024-04-19T01:02:35.216 INFO:teuthology.orchestra.run.smithi033.stdout: osd max backfills = 3 2024-04-19T01:02:35.216 INFO:teuthology.orchestra.run.smithi033.stdout: osd max markdown count = 1000 2024-04-19T01:02:35.216 INFO:teuthology.orchestra.run.smithi033.stdout: osd mclock override recovery settings = True 2024-04-19T01:02:35.216 INFO:teuthology.orchestra.run.smithi033.stdout: osd objectstore = bluestore 2024-04-19T01:02:35.216 INFO:teuthology.orchestra.run.smithi033.stdout: osd op queue = debug_random 2024-04-19T01:02:35.216 INFO:teuthology.orchestra.run.smithi033.stdout: osd op queue cut off = debug_random 2024-04-19T01:02:35.216 INFO:teuthology.orchestra.run.smithi033.stdout: osd scrub min interval = 60 2024-04-19T01:02:35.217 INFO:teuthology.orchestra.run.smithi033.stdout: osd shutdown pgref assert = True 2024-04-19T01:02:35.217 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:02:35.217 INFO:teuthology.orchestra.run.smithi033.stdout:[mgr] 2024-04-19T01:02:35.217 INFO:teuthology.orchestra.run.smithi033.stdout: debug ms = 1 2024-04-19T01:02:35.217 INFO:teuthology.orchestra.run.smithi033.stdout: debug mgr = 20 2024-04-19T01:02:35.217 INFO:teuthology.orchestra.run.smithi033.stdout: debug mon = 20 2024-04-19T01:02:35.217 INFO:teuthology.orchestra.run.smithi033.stdout: debug auth = 20 2024-04-19T01:02:35.217 INFO:teuthology.orchestra.run.smithi033.stdout: mon reweight min pgs per osd = 4 2024-04-19T01:02:35.217 INFO:teuthology.orchestra.run.smithi033.stdout: mon reweight min bytes per osd = 10 2024-04-19T01:02:35.217 INFO:teuthology.orchestra.run.smithi033.stdout: mgr/telemetry/nag = false 2024-04-19T01:02:35.218 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:02:35.218 INFO:teuthology.orchestra.run.smithi033.stdout:[mon] 2024-04-19T01:02:35.218 INFO:teuthology.orchestra.run.smithi033.stdout: debug ms = 1 2024-04-19T01:02:35.218 INFO:teuthology.orchestra.run.smithi033.stdout: debug mon = 20 2024-04-19T01:02:35.218 INFO:teuthology.orchestra.run.smithi033.stdout: debug paxos = 20 2024-04-19T01:02:35.218 INFO:teuthology.orchestra.run.smithi033.stdout: debug auth = 20 2024-04-19T01:02:35.218 INFO:teuthology.orchestra.run.smithi033.stdout: mon data avail warn = 5 2024-04-19T01:02:35.218 INFO:teuthology.orchestra.run.smithi033.stdout: mon mgr mkfs grace = 240 2024-04-19T01:02:35.218 INFO:teuthology.orchestra.run.smithi033.stdout: mon reweight min pgs per osd = 4 2024-04-19T01:02:35.219 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd reporter subtree level = osd 2024-04-19T01:02:35.219 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd prime pg temp = true 2024-04-19T01:02:35.219 INFO:teuthology.orchestra.run.smithi033.stdout: mon reweight min bytes per osd = 10 2024-04-19T01:02:35.219 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:02:35.219 INFO:teuthology.orchestra.run.smithi033.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-19T01:02:35.219 INFO:teuthology.orchestra.run.smithi033.stdout: auth mon ticket ttl = 660 # 11m 2024-04-19T01:02:35.219 INFO:teuthology.orchestra.run.smithi033.stdout: auth service ticket ttl = 240 # 4m 2024-04-19T01:02:35.219 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:02:35.219 INFO:teuthology.orchestra.run.smithi033.stdout: # don't complain about insecure global_id in the test suite 2024-04-19T01:02:35.219 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-19T01:02:35.219 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-19T01:02:35.219 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:02:35.219 INFO:teuthology.orchestra.run.smithi033.stdout: # 1m isn't quite enough 2024-04-19T01:02:35.219 INFO:teuthology.orchestra.run.smithi033.stdout: mon_down_mkfs_grace = 2m 2024-04-19T01:02:35.220 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:02:35.220 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_filestore_osds = false 2024-04-19T01:02:35.220 INFO:teuthology.orchestra.run.smithi033.stdout: mon scrub interval = 300 2024-04-19T01:02:35.220 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:02:35.220 INFO:teuthology.orchestra.run.smithi033.stdout:[client] 2024-04-19T01:02:35.220 INFO:teuthology.orchestra.run.smithi033.stdout: rgw cache enabled = true 2024-04-19T01:02:35.220 INFO:teuthology.orchestra.run.smithi033.stdout: rgw enable ops log = true 2024-04-19T01:02:35.220 INFO:teuthology.orchestra.run.smithi033.stdout: rgw enable usage log = true 2024-04-19T01:02:35.220 INFO:teuthology.orchestra.run.smithi033.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-19T01:02:35.220 INFO:teuthology.orchestra.run.smithi033.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-19T01:02:35.221 INFO:teuthology.orchestra.run.smithi033.stdout:[mon.a] 2024-04-19T01:02:35.221 INFO:teuthology.orchestra.run.smithi033.stdout:[mon.b] 2024-04-19T01:02:35.221 INFO:teuthology.orchestra.run.smithi033.stdout:[mon.c] 2024-04-19T01:02:35.224 INFO:teuthology.orchestra.run.smithi155.stdout:[global] 2024-04-19T01:02:35.224 INFO:teuthology.orchestra.run.smithi155.stdout: chdir = "" 2024-04-19T01:02:35.224 INFO:teuthology.orchestra.run.smithi155.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-19T01:02:35.224 INFO:teuthology.orchestra.run.smithi155.stdout: auth supported = cephx 2024-04-19T01:02:35.224 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-19T01:02:35.224 INFO:teuthology.orchestra.run.smithi155.stdout: filestore xattr use omap = true 2024-04-19T01:02:35.224 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-19T01:02:35.225 INFO:teuthology.orchestra.run.smithi155.stdout: mon clock drift allowed = 1.000 2024-04-19T01:02:35.225 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-19T01:02:35.225 INFO:teuthology.orchestra.run.smithi155.stdout: osd crush chooseleaf type = 0 2024-04-19T01:02:35.225 INFO:teuthology.orchestra.run.smithi155.stdout: auth debug = true 2024-04-19T01:02:35.225 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-19T01:02:35.225 INFO:teuthology.orchestra.run.smithi155.stdout: ms die on old message = true 2024-04-19T01:02:35.225 INFO:teuthology.orchestra.run.smithi155.stdout: ms die on bug = true 2024-04-19T01:02:35.225 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-19T01:02:35.225 INFO:teuthology.orchestra.run.smithi155.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-19T01:02:35.225 INFO:teuthology.orchestra.run.smithi155.stdout: mon pg warn max object skew = 0 2024-04-19T01:02:35.225 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-19T01:02:35.225 INFO:teuthology.orchestra.run.smithi155.stdout: # disable pg_autoscaler by default for new pools 2024-04-19T01:02:35.225 INFO:teuthology.orchestra.run.smithi155.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-19T01:02:35.225 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-19T01:02:35.225 INFO:teuthology.orchestra.run.smithi155.stdout: osd pool default size = 2 2024-04-19T01:02:35.226 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-19T01:02:35.226 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd allow primary affinity = true 2024-04-19T01:02:35.226 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd allow pg remap = true 2024-04-19T01:02:35.226 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on legacy crush tunables = false 2024-04-19T01:02:35.226 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on crush straw calc version zero = false 2024-04-19T01:02:35.226 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on no sortbitwise = false 2024-04-19T01:02:35.226 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on osd down out interval zero = false 2024-04-19T01:02:35.226 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on too few osds = false 2024-04-19T01:02:35.226 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-19T01:02:35.226 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-19T01:02:35.226 INFO:teuthology.orchestra.run.smithi155.stdout: mon_allow_pool_size_one = true 2024-04-19T01:02:35.226 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-19T01:02:35.226 INFO:teuthology.orchestra.run.smithi155.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-19T01:02:35.226 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-19T01:02:35.227 INFO:teuthology.orchestra.run.smithi155.stdout: osd default data pool replay window = 5 2024-04-19T01:02:35.227 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-19T01:02:35.227 INFO:teuthology.orchestra.run.smithi155.stdout: mon allow pool delete = true 2024-04-19T01:02:35.227 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-19T01:02:35.227 INFO:teuthology.orchestra.run.smithi155.stdout: mon cluster log file level = debug 2024-04-19T01:02:35.227 INFO:teuthology.orchestra.run.smithi155.stdout: debug asserts on shutdown = true 2024-04-19T01:02:35.227 INFO:teuthology.orchestra.run.smithi155.stdout: mon health detail to clog = false 2024-04-19T01:02:35.227 INFO:teuthology.orchestra.run.smithi155.stdout: mon host = "172.21.15.33,172.21.15.110,172.21.15.118" 2024-04-19T01:02:35.227 INFO:teuthology.orchestra.run.smithi155.stdout: mon client directed command retry = 5 2024-04-19T01:02:35.227 INFO:teuthology.orchestra.run.smithi155.stdout: mon election default strategy = 1 2024-04-19T01:02:35.227 INFO:teuthology.orchestra.run.smithi155.stdout: ms inject socket failures = 2500 2024-04-19T01:02:35.227 INFO:teuthology.orchestra.run.smithi155.stdout: ms tcp read timeout = 5 2024-04-19T01:02:35.227 INFO:teuthology.orchestra.run.smithi155.stdout: osd_async_recovery_min_cost = 1 2024-04-19T01:02:35.227 INFO:teuthology.orchestra.run.smithi155.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-04-19T01:02:35.227 INFO:teuthology.orchestra.run.smithi155.stdout: fsid = f11bd2d5-4456-4ffa-83c0-722dddd22ad3 2024-04-19T01:02:35.228 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-19T01:02:35.228 INFO:teuthology.orchestra.run.smithi155.stdout:[osd] 2024-04-19T01:02:35.228 INFO:teuthology.orchestra.run.smithi155.stdout: osd journal size = 100 2024-04-19T01:02:35.228 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-19T01:02:35.228 INFO:teuthology.orchestra.run.smithi155.stdout: osd scrub load threshold = 5.0 2024-04-19T01:02:35.228 INFO:teuthology.orchestra.run.smithi155.stdout: osd scrub max interval = 120 2024-04-19T01:02:35.228 INFO:teuthology.orchestra.run.smithi155.stdout: osd mclock profile = high_recovery_ops 2024-04-19T01:02:35.228 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-19T01:02:35.228 INFO:teuthology.orchestra.run.smithi155.stdout: osd recover clone overlap = true 2024-04-19T01:02:35.228 INFO:teuthology.orchestra.run.smithi155.stdout: osd recovery max chunk = 1048576 2024-04-19T01:02:35.228 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-19T01:02:35.228 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug shutdown = true 2024-04-19T01:02:35.228 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug op order = true 2024-04-19T01:02:35.228 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug verify stray on activate = true 2024-04-19T01:02:35.229 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug trim objects = true 2024-04-19T01:02:35.229 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-19T01:02:35.229 INFO:teuthology.orchestra.run.smithi155.stdout: osd open classes on start = true 2024-04-19T01:02:35.229 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug pg log writeout = true 2024-04-19T01:02:35.229 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-19T01:02:35.229 INFO:teuthology.orchestra.run.smithi155.stdout: osd deep scrub update digest min age = 30 2024-04-19T01:02:35.229 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-19T01:02:35.229 INFO:teuthology.orchestra.run.smithi155.stdout: osd map max advance = 10 2024-04-19T01:02:35.229 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-19T01:02:35.229 INFO:teuthology.orchestra.run.smithi155.stdout: journal zero on create = true 2024-04-19T01:02:35.229 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-19T01:02:35.229 INFO:teuthology.orchestra.run.smithi155.stdout: filestore ondisk finisher threads = 3 2024-04-19T01:02:35.229 INFO:teuthology.orchestra.run.smithi155.stdout: filestore apply finisher threads = 3 2024-04-19T01:02:35.229 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-19T01:02:35.229 INFO:teuthology.orchestra.run.smithi155.stdout: bdev debug aio = true 2024-04-19T01:02:35.230 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug misdirected ops = true 2024-04-19T01:02:35.230 INFO:teuthology.orchestra.run.smithi155.stdout: debug monc = 20 2024-04-19T01:02:35.230 INFO:teuthology.orchestra.run.smithi155.stdout: bluestore block size = 96636764160 2024-04-19T01:02:35.230 INFO:teuthology.orchestra.run.smithi155.stdout: bluestore compression algorithm = zlib 2024-04-19T01:02:35.230 INFO:teuthology.orchestra.run.smithi155.stdout: bluestore compression mode = aggressive 2024-04-19T01:02:35.230 INFO:teuthology.orchestra.run.smithi155.stdout: bluestore fsck on mount = True 2024-04-19T01:02:35.230 INFO:teuthology.orchestra.run.smithi155.stdout: bluestore zero block detection = True 2024-04-19T01:02:35.230 INFO:teuthology.orchestra.run.smithi155.stdout: debug bluefs = 20 2024-04-19T01:02:35.230 INFO:teuthology.orchestra.run.smithi155.stdout: debug bluestore = 20 2024-04-19T01:02:35.230 INFO:teuthology.orchestra.run.smithi155.stdout: debug ms = 1 2024-04-19T01:02:35.230 INFO:teuthology.orchestra.run.smithi155.stdout: debug osd = 20 2024-04-19T01:02:35.230 INFO:teuthology.orchestra.run.smithi155.stdout: debug rocksdb = 10 2024-04-19T01:02:35.230 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd backfillfull_ratio = 0.85 2024-04-19T01:02:35.230 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd full ratio = 0.9 2024-04-19T01:02:35.230 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd nearfull ratio = 0.8 2024-04-19T01:02:35.231 INFO:teuthology.orchestra.run.smithi155.stdout: osd blocked scrub grace period = 3600 2024-04-19T01:02:35.231 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug reject backfill probability = 0.3 2024-04-19T01:02:35.231 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug verify cached snaps = True 2024-04-19T01:02:35.231 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug verify missing on start = True 2024-04-19T01:02:35.231 INFO:teuthology.orchestra.run.smithi155.stdout: osd failsafe full ratio = 0.95 2024-04-19T01:02:35.231 INFO:teuthology.orchestra.run.smithi155.stdout: osd max backfills = 3 2024-04-19T01:02:35.231 INFO:teuthology.orchestra.run.smithi155.stdout: osd max markdown count = 1000 2024-04-19T01:02:35.231 INFO:teuthology.orchestra.run.smithi155.stdout: osd mclock override recovery settings = True 2024-04-19T01:02:35.231 INFO:teuthology.orchestra.run.smithi155.stdout: osd objectstore = bluestore 2024-04-19T01:02:35.231 INFO:teuthology.orchestra.run.smithi155.stdout: osd op queue = debug_random 2024-04-19T01:02:35.231 INFO:teuthology.orchestra.run.smithi155.stdout: osd op queue cut off = debug_random 2024-04-19T01:02:35.231 INFO:teuthology.orchestra.run.smithi155.stdout: osd scrub min interval = 60 2024-04-19T01:02:35.231 INFO:teuthology.orchestra.run.smithi155.stdout: osd shutdown pgref assert = True 2024-04-19T01:02:35.231 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-19T01:02:35.231 INFO:teuthology.orchestra.run.smithi155.stdout:[mgr] 2024-04-19T01:02:35.232 INFO:teuthology.orchestra.run.smithi155.stdout: debug ms = 1 2024-04-19T01:02:35.232 INFO:teuthology.orchestra.run.smithi155.stdout: debug mgr = 20 2024-04-19T01:02:35.232 INFO:teuthology.orchestra.run.smithi155.stdout: debug mon = 20 2024-04-19T01:02:35.232 INFO:teuthology.orchestra.run.smithi155.stdout: debug auth = 20 2024-04-19T01:02:35.232 INFO:teuthology.orchestra.run.smithi155.stdout: mon reweight min pgs per osd = 4 2024-04-19T01:02:35.232 INFO:teuthology.orchestra.run.smithi155.stdout: mon reweight min bytes per osd = 10 2024-04-19T01:02:35.232 INFO:teuthology.orchestra.run.smithi155.stdout: mgr/telemetry/nag = false 2024-04-19T01:02:35.232 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-19T01:02:35.232 INFO:teuthology.orchestra.run.smithi155.stdout:[mon] 2024-04-19T01:02:35.232 INFO:teuthology.orchestra.run.smithi155.stdout: debug ms = 1 2024-04-19T01:02:35.232 INFO:teuthology.orchestra.run.smithi155.stdout: debug mon = 20 2024-04-19T01:02:35.232 INFO:teuthology.orchestra.run.smithi155.stdout: debug paxos = 20 2024-04-19T01:02:35.232 INFO:teuthology.orchestra.run.smithi155.stdout: debug auth = 20 2024-04-19T01:02:35.232 INFO:teuthology.orchestra.run.smithi155.stdout: mon data avail warn = 5 2024-04-19T01:02:35.232 INFO:teuthology.orchestra.run.smithi155.stdout: mon mgr mkfs grace = 240 2024-04-19T01:02:35.233 INFO:teuthology.orchestra.run.smithi155.stdout: mon reweight min pgs per osd = 4 2024-04-19T01:02:35.233 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd reporter subtree level = osd 2024-04-19T01:02:35.233 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd prime pg temp = true 2024-04-19T01:02:35.233 INFO:teuthology.orchestra.run.smithi155.stdout: mon reweight min bytes per osd = 10 2024-04-19T01:02:35.233 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-19T01:02:35.233 INFO:teuthology.orchestra.run.smithi155.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-19T01:02:35.233 INFO:teuthology.orchestra.run.smithi155.stdout: auth mon ticket ttl = 660 # 11m 2024-04-19T01:02:35.233 INFO:teuthology.orchestra.run.smithi155.stdout: auth service ticket ttl = 240 # 4m 2024-04-19T01:02:35.233 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-19T01:02:35.233 INFO:teuthology.orchestra.run.smithi155.stdout: # don't complain about insecure global_id in the test suite 2024-04-19T01:02:35.233 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-19T01:02:35.233 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-19T01:02:35.233 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-19T01:02:35.233 INFO:teuthology.orchestra.run.smithi155.stdout: # 1m isn't quite enough 2024-04-19T01:02:35.233 INFO:teuthology.orchestra.run.smithi155.stdout: mon_down_mkfs_grace = 2m 2024-04-19T01:02:35.234 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-19T01:02:35.234 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_filestore_osds = false 2024-04-19T01:02:35.234 INFO:teuthology.orchestra.run.smithi155.stdout: mon scrub interval = 300 2024-04-19T01:02:35.234 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-19T01:02:35.234 INFO:teuthology.orchestra.run.smithi155.stdout:[client] 2024-04-19T01:02:35.234 INFO:teuthology.orchestra.run.smithi155.stdout: rgw cache enabled = true 2024-04-19T01:02:35.234 INFO:teuthology.orchestra.run.smithi155.stdout: rgw enable ops log = true 2024-04-19T01:02:35.234 INFO:teuthology.orchestra.run.smithi155.stdout: rgw enable usage log = true 2024-04-19T01:02:35.234 INFO:teuthology.orchestra.run.smithi155.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-19T01:02:35.234 INFO:teuthology.orchestra.run.smithi155.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-19T01:02:35.234 INFO:teuthology.orchestra.run.smithi155.stdout:[mon.a] 2024-04-19T01:02:35.234 INFO:teuthology.orchestra.run.smithi155.stdout:[mon.b] 2024-04-19T01:02:35.234 INFO:teuthology.orchestra.run.smithi155.stdout:[mon.c] 2024-04-19T01:02:35.235 INFO:teuthology.orchestra.run.smithi110.stdout:[global] 2024-04-19T01:02:35.235 INFO:teuthology.orchestra.run.smithi110.stdout: chdir = "" 2024-04-19T01:02:35.235 INFO:teuthology.orchestra.run.smithi110.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-19T01:02:35.235 INFO:teuthology.orchestra.run.smithi110.stdout: auth supported = cephx 2024-04-19T01:02:35.235 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-19T01:02:35.235 INFO:teuthology.orchestra.run.smithi110.stdout: filestore xattr use omap = true 2024-04-19T01:02:35.235 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-19T01:02:35.235 INFO:teuthology.orchestra.run.smithi110.stdout: mon clock drift allowed = 1.000 2024-04-19T01:02:35.235 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-19T01:02:35.235 INFO:teuthology.orchestra.run.smithi110.stdout: osd crush chooseleaf type = 0 2024-04-19T01:02:35.235 INFO:teuthology.orchestra.run.smithi110.stdout: auth debug = true 2024-04-19T01:02:35.235 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-19T01:02:35.235 INFO:teuthology.orchestra.run.smithi110.stdout: ms die on old message = true 2024-04-19T01:02:35.235 INFO:teuthology.orchestra.run.smithi110.stdout: ms die on bug = true 2024-04-19T01:02:35.236 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-19T01:02:35.236 INFO:teuthology.orchestra.run.smithi110.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-19T01:02:35.236 INFO:teuthology.orchestra.run.smithi110.stdout: mon pg warn max object skew = 0 2024-04-19T01:02:35.236 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-19T01:02:35.236 INFO:teuthology.orchestra.run.smithi110.stdout: # disable pg_autoscaler by default for new pools 2024-04-19T01:02:35.236 INFO:teuthology.orchestra.run.smithi110.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-19T01:02:35.236 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-19T01:02:35.236 INFO:teuthology.orchestra.run.smithi110.stdout: osd pool default size = 2 2024-04-19T01:02:35.236 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-19T01:02:35.236 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd allow primary affinity = true 2024-04-19T01:02:35.236 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd allow pg remap = true 2024-04-19T01:02:35.236 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on legacy crush tunables = false 2024-04-19T01:02:35.236 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on crush straw calc version zero = false 2024-04-19T01:02:35.236 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on no sortbitwise = false 2024-04-19T01:02:35.236 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on osd down out interval zero = false 2024-04-19T01:02:35.237 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on too few osds = false 2024-04-19T01:02:35.237 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-19T01:02:35.237 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-19T01:02:35.237 INFO:teuthology.orchestra.run.smithi110.stdout: mon_allow_pool_size_one = true 2024-04-19T01:02:35.237 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-19T01:02:35.237 INFO:teuthology.orchestra.run.smithi110.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-19T01:02:35.237 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-19T01:02:35.237 INFO:teuthology.orchestra.run.smithi110.stdout: osd default data pool replay window = 5 2024-04-19T01:02:35.237 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-19T01:02:35.237 INFO:teuthology.orchestra.run.smithi110.stdout: mon allow pool delete = true 2024-04-19T01:02:35.237 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-19T01:02:35.237 INFO:teuthology.orchestra.run.smithi110.stdout: mon cluster log file level = debug 2024-04-19T01:02:35.237 INFO:teuthology.orchestra.run.smithi110.stdout: debug asserts on shutdown = true 2024-04-19T01:02:35.237 INFO:teuthology.orchestra.run.smithi110.stdout: mon health detail to clog = false 2024-04-19T01:02:35.237 INFO:teuthology.orchestra.run.smithi110.stdout: mon host = "172.21.15.33,172.21.15.110,172.21.15.118" 2024-04-19T01:02:35.238 INFO:teuthology.orchestra.run.smithi110.stdout: mon client directed command retry = 5 2024-04-19T01:02:35.238 INFO:teuthology.orchestra.run.smithi110.stdout: mon election default strategy = 1 2024-04-19T01:02:35.238 INFO:teuthology.orchestra.run.smithi110.stdout: ms inject socket failures = 2500 2024-04-19T01:02:35.238 INFO:teuthology.orchestra.run.smithi110.stdout: ms tcp read timeout = 5 2024-04-19T01:02:35.238 INFO:teuthology.orchestra.run.smithi110.stdout: osd_async_recovery_min_cost = 1 2024-04-19T01:02:35.238 INFO:teuthology.orchestra.run.smithi110.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-04-19T01:02:35.238 INFO:teuthology.orchestra.run.smithi110.stdout: fsid = f11bd2d5-4456-4ffa-83c0-722dddd22ad3 2024-04-19T01:02:35.238 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-19T01:02:35.238 INFO:teuthology.orchestra.run.smithi110.stdout:[osd] 2024-04-19T01:02:35.238 INFO:teuthology.orchestra.run.smithi110.stdout: osd journal size = 100 2024-04-19T01:02:35.238 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-19T01:02:35.238 INFO:teuthology.orchestra.run.smithi110.stdout: osd scrub load threshold = 5.0 2024-04-19T01:02:35.238 INFO:teuthology.orchestra.run.smithi110.stdout: osd scrub max interval = 120 2024-04-19T01:02:35.238 INFO:teuthology.orchestra.run.smithi110.stdout: osd mclock profile = high_recovery_ops 2024-04-19T01:02:35.238 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-19T01:02:35.239 INFO:teuthology.orchestra.run.smithi110.stdout: osd recover clone overlap = true 2024-04-19T01:02:35.239 INFO:teuthology.orchestra.run.smithi110.stdout: osd recovery max chunk = 1048576 2024-04-19T01:02:35.239 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-19T01:02:35.239 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug shutdown = true 2024-04-19T01:02:35.239 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug op order = true 2024-04-19T01:02:35.239 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug verify stray on activate = true 2024-04-19T01:02:35.239 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug trim objects = true 2024-04-19T01:02:35.239 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-19T01:02:35.239 INFO:teuthology.orchestra.run.smithi110.stdout: osd open classes on start = true 2024-04-19T01:02:35.239 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug pg log writeout = true 2024-04-19T01:02:35.239 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-19T01:02:35.239 INFO:teuthology.orchestra.run.smithi110.stdout: osd deep scrub update digest min age = 30 2024-04-19T01:02:35.239 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-19T01:02:35.239 INFO:teuthology.orchestra.run.smithi110.stdout: osd map max advance = 10 2024-04-19T01:02:35.239 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-19T01:02:35.240 INFO:teuthology.orchestra.run.smithi110.stdout: journal zero on create = true 2024-04-19T01:02:35.240 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-19T01:02:35.240 INFO:teuthology.orchestra.run.smithi110.stdout: filestore ondisk finisher threads = 3 2024-04-19T01:02:35.240 INFO:teuthology.orchestra.run.smithi110.stdout: filestore apply finisher threads = 3 2024-04-19T01:02:35.240 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-19T01:02:35.240 INFO:teuthology.orchestra.run.smithi110.stdout: bdev debug aio = true 2024-04-19T01:02:35.240 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug misdirected ops = true 2024-04-19T01:02:35.240 INFO:teuthology.orchestra.run.smithi110.stdout: debug monc = 20 2024-04-19T01:02:35.240 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore block size = 96636764160 2024-04-19T01:02:35.240 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore compression algorithm = zlib 2024-04-19T01:02:35.240 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore compression mode = aggressive 2024-04-19T01:02:35.240 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore fsck on mount = True 2024-04-19T01:02:35.240 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore zero block detection = True 2024-04-19T01:02:35.240 INFO:teuthology.orchestra.run.smithi110.stdout: debug bluefs = 20 2024-04-19T01:02:35.240 INFO:teuthology.orchestra.run.smithi110.stdout: debug bluestore = 20 2024-04-19T01:02:35.241 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2024-04-19T01:02:35.241 INFO:teuthology.orchestra.run.smithi110.stdout: debug osd = 20 2024-04-19T01:02:35.241 INFO:teuthology.orchestra.run.smithi110.stdout: debug rocksdb = 10 2024-04-19T01:02:35.241 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd backfillfull_ratio = 0.85 2024-04-19T01:02:35.241 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd full ratio = 0.9 2024-04-19T01:02:35.241 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd nearfull ratio = 0.8 2024-04-19T01:02:35.241 INFO:teuthology.orchestra.run.smithi110.stdout: osd blocked scrub grace period = 3600 2024-04-19T01:02:35.241 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug reject backfill probability = 0.3 2024-04-19T01:02:35.241 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug verify cached snaps = True 2024-04-19T01:02:35.241 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug verify missing on start = True 2024-04-19T01:02:35.241 INFO:teuthology.orchestra.run.smithi110.stdout: osd failsafe full ratio = 0.95 2024-04-19T01:02:35.241 INFO:teuthology.orchestra.run.smithi110.stdout: osd max backfills = 3 2024-04-19T01:02:35.241 INFO:teuthology.orchestra.run.smithi110.stdout: osd max markdown count = 1000 2024-04-19T01:02:35.241 INFO:teuthology.orchestra.run.smithi110.stdout: osd mclock override recovery settings = True 2024-04-19T01:02:35.241 INFO:teuthology.orchestra.run.smithi110.stdout: osd objectstore = bluestore 2024-04-19T01:02:35.242 INFO:teuthology.orchestra.run.smithi110.stdout: osd op queue = debug_random 2024-04-19T01:02:35.242 INFO:teuthology.orchestra.run.smithi110.stdout: osd op queue cut off = debug_random 2024-04-19T01:02:35.242 INFO:teuthology.orchestra.run.smithi110.stdout: osd scrub min interval = 60 2024-04-19T01:02:35.242 INFO:teuthology.orchestra.run.smithi110.stdout: osd shutdown pgref assert = True 2024-04-19T01:02:35.242 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-19T01:02:35.242 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr] 2024-04-19T01:02:35.242 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2024-04-19T01:02:35.242 INFO:teuthology.orchestra.run.smithi110.stdout: debug mgr = 20 2024-04-19T01:02:35.242 INFO:teuthology.orchestra.run.smithi110.stdout: debug mon = 20 2024-04-19T01:02:35.242 INFO:teuthology.orchestra.run.smithi110.stdout: debug auth = 20 2024-04-19T01:02:35.242 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min pgs per osd = 4 2024-04-19T01:02:35.242 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min bytes per osd = 10 2024-04-19T01:02:35.242 INFO:teuthology.orchestra.run.smithi110.stdout: mgr/telemetry/nag = false 2024-04-19T01:02:35.242 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-19T01:02:35.243 INFO:teuthology.orchestra.run.smithi110.stdout:[mon] 2024-04-19T01:02:35.243 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2024-04-19T01:02:35.243 INFO:teuthology.orchestra.run.smithi110.stdout: debug mon = 20 2024-04-19T01:02:35.243 INFO:teuthology.orchestra.run.smithi110.stdout: debug paxos = 20 2024-04-19T01:02:35.243 INFO:teuthology.orchestra.run.smithi110.stdout: debug auth = 20 2024-04-19T01:02:35.243 INFO:teuthology.orchestra.run.smithi110.stdout: mon data avail warn = 5 2024-04-19T01:02:35.243 INFO:teuthology.orchestra.run.smithi110.stdout: mon mgr mkfs grace = 240 2024-04-19T01:02:35.243 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min pgs per osd = 4 2024-04-19T01:02:35.243 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd reporter subtree level = osd 2024-04-19T01:02:35.243 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd prime pg temp = true 2024-04-19T01:02:35.243 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min bytes per osd = 10 2024-04-19T01:02:35.243 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-19T01:02:35.243 INFO:teuthology.orchestra.run.smithi110.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-19T01:02:35.243 INFO:teuthology.orchestra.run.smithi110.stdout: auth mon ticket ttl = 660 # 11m 2024-04-19T01:02:35.243 INFO:teuthology.orchestra.run.smithi110.stdout: auth service ticket ttl = 240 # 4m 2024-04-19T01:02:35.244 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-19T01:02:35.244 INFO:teuthology.orchestra.run.smithi110.stdout: # don't complain about insecure global_id in the test suite 2024-04-19T01:02:35.244 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-19T01:02:35.244 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-19T01:02:35.244 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-19T01:02:35.244 INFO:teuthology.orchestra.run.smithi110.stdout: # 1m isn't quite enough 2024-04-19T01:02:35.244 INFO:teuthology.orchestra.run.smithi110.stdout: mon_down_mkfs_grace = 2m 2024-04-19T01:02:35.244 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-19T01:02:35.244 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_filestore_osds = false 2024-04-19T01:02:35.244 INFO:teuthology.orchestra.run.smithi110.stdout: mon scrub interval = 300 2024-04-19T01:02:35.244 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-19T01:02:35.244 INFO:teuthology.orchestra.run.smithi110.stdout:[client] 2024-04-19T01:02:35.244 INFO:teuthology.orchestra.run.smithi110.stdout: rgw cache enabled = true 2024-04-19T01:02:35.244 INFO:teuthology.orchestra.run.smithi110.stdout: rgw enable ops log = true 2024-04-19T01:02:35.244 INFO:teuthology.orchestra.run.smithi110.stdout: rgw enable usage log = true 2024-04-19T01:02:35.244 INFO:teuthology.orchestra.run.smithi110.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-19T01:02:35.245 INFO:teuthology.orchestra.run.smithi110.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-19T01:02:35.245 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.a] 2024-04-19T01:02:35.245 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.b] 2024-04-19T01:02:35.245 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.c] 2024-04-19T01:02:35.245 INFO:teuthology.orchestra.run.smithi118.stdout:[global] 2024-04-19T01:02:35.245 INFO:teuthology.orchestra.run.smithi118.stdout: chdir = "" 2024-04-19T01:02:35.245 INFO:teuthology.orchestra.run.smithi118.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-19T01:02:35.245 INFO:teuthology.orchestra.run.smithi118.stdout: auth supported = cephx 2024-04-19T01:02:35.245 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-19T01:02:35.245 INFO:teuthology.orchestra.run.smithi118.stdout: filestore xattr use omap = true 2024-04-19T01:02:35.245 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-19T01:02:35.245 INFO:teuthology.orchestra.run.smithi118.stdout: mon clock drift allowed = 1.000 2024-04-19T01:02:35.246 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-19T01:02:35.246 INFO:teuthology.orchestra.run.smithi118.stdout: osd crush chooseleaf type = 0 2024-04-19T01:02:35.246 INFO:teuthology.orchestra.run.smithi118.stdout: auth debug = true 2024-04-19T01:02:35.246 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-19T01:02:35.246 INFO:teuthology.orchestra.run.smithi118.stdout: ms die on old message = true 2024-04-19T01:02:35.246 INFO:teuthology.orchestra.run.smithi118.stdout: ms die on bug = true 2024-04-19T01:02:35.246 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-19T01:02:35.246 INFO:teuthology.orchestra.run.smithi118.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-19T01:02:35.246 INFO:teuthology.orchestra.run.smithi118.stdout: mon pg warn max object skew = 0 2024-04-19T01:02:35.246 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-19T01:02:35.246 INFO:teuthology.orchestra.run.smithi118.stdout: # disable pg_autoscaler by default for new pools 2024-04-19T01:02:35.246 INFO:teuthology.orchestra.run.smithi118.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-19T01:02:35.246 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-19T01:02:35.246 INFO:teuthology.orchestra.run.smithi118.stdout: osd pool default size = 2 2024-04-19T01:02:35.246 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-19T01:02:35.247 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd allow primary affinity = true 2024-04-19T01:02:35.247 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd allow pg remap = true 2024-04-19T01:02:35.247 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on legacy crush tunables = false 2024-04-19T01:02:35.247 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on crush straw calc version zero = false 2024-04-19T01:02:35.247 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on no sortbitwise = false 2024-04-19T01:02:35.247 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on osd down out interval zero = false 2024-04-19T01:02:35.247 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on too few osds = false 2024-04-19T01:02:35.247 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-19T01:02:35.247 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-19T01:02:35.247 INFO:teuthology.orchestra.run.smithi118.stdout: mon_allow_pool_size_one = true 2024-04-19T01:02:35.247 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-19T01:02:35.247 INFO:teuthology.orchestra.run.smithi118.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-19T01:02:35.247 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-19T01:02:35.247 INFO:teuthology.orchestra.run.smithi118.stdout: osd default data pool replay window = 5 2024-04-19T01:02:35.247 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-19T01:02:35.248 INFO:teuthology.orchestra.run.smithi118.stdout: mon allow pool delete = true 2024-04-19T01:02:35.248 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-19T01:02:35.248 INFO:teuthology.orchestra.run.smithi118.stdout: mon cluster log file level = debug 2024-04-19T01:02:35.248 INFO:teuthology.orchestra.run.smithi118.stdout: debug asserts on shutdown = true 2024-04-19T01:02:35.248 INFO:teuthology.orchestra.run.smithi118.stdout: mon health detail to clog = false 2024-04-19T01:02:35.248 INFO:teuthology.orchestra.run.smithi118.stdout: mon host = "172.21.15.33,172.21.15.110,172.21.15.118" 2024-04-19T01:02:35.248 INFO:teuthology.orchestra.run.smithi118.stdout: mon client directed command retry = 5 2024-04-19T01:02:35.248 INFO:teuthology.orchestra.run.smithi118.stdout: mon election default strategy = 1 2024-04-19T01:02:35.248 INFO:teuthology.orchestra.run.smithi118.stdout: ms inject socket failures = 2500 2024-04-19T01:02:35.248 INFO:teuthology.orchestra.run.smithi118.stdout: ms tcp read timeout = 5 2024-04-19T01:02:35.248 INFO:teuthology.orchestra.run.smithi118.stdout: osd_async_recovery_min_cost = 1 2024-04-19T01:02:35.248 INFO:teuthology.orchestra.run.smithi118.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-04-19T01:02:35.248 INFO:teuthology.orchestra.run.smithi118.stdout: fsid = f11bd2d5-4456-4ffa-83c0-722dddd22ad3 2024-04-19T01:02:35.248 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-19T01:02:35.248 INFO:teuthology.orchestra.run.smithi118.stdout:[osd] 2024-04-19T01:02:35.249 INFO:teuthology.orchestra.run.smithi118.stdout: osd journal size = 100 2024-04-19T01:02:35.249 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-19T01:02:35.249 INFO:teuthology.orchestra.run.smithi118.stdout: osd scrub load threshold = 5.0 2024-04-19T01:02:35.249 INFO:teuthology.orchestra.run.smithi118.stdout: osd scrub max interval = 120 2024-04-19T01:02:35.249 INFO:teuthology.orchestra.run.smithi118.stdout: osd mclock profile = high_recovery_ops 2024-04-19T01:02:35.249 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-19T01:02:35.249 INFO:teuthology.orchestra.run.smithi118.stdout: osd recover clone overlap = true 2024-04-19T01:02:35.249 INFO:teuthology.orchestra.run.smithi118.stdout: osd recovery max chunk = 1048576 2024-04-19T01:02:35.249 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-19T01:02:35.249 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug shutdown = true 2024-04-19T01:02:35.249 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug op order = true 2024-04-19T01:02:35.249 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug verify stray on activate = true 2024-04-19T01:02:35.249 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug trim objects = true 2024-04-19T01:02:35.249 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-19T01:02:35.250 INFO:teuthology.orchestra.run.smithi118.stdout: osd open classes on start = true 2024-04-19T01:02:35.250 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug pg log writeout = true 2024-04-19T01:02:35.250 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-19T01:02:35.250 INFO:teuthology.orchestra.run.smithi118.stdout: osd deep scrub update digest min age = 30 2024-04-19T01:02:35.250 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-19T01:02:35.250 INFO:teuthology.orchestra.run.smithi118.stdout: osd map max advance = 10 2024-04-19T01:02:35.250 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-19T01:02:35.250 INFO:teuthology.orchestra.run.smithi118.stdout: journal zero on create = true 2024-04-19T01:02:35.250 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-19T01:02:35.250 INFO:teuthology.orchestra.run.smithi118.stdout: filestore ondisk finisher threads = 3 2024-04-19T01:02:35.250 INFO:teuthology.orchestra.run.smithi118.stdout: filestore apply finisher threads = 3 2024-04-19T01:02:35.250 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-19T01:02:35.250 INFO:teuthology.orchestra.run.smithi118.stdout: bdev debug aio = true 2024-04-19T01:02:35.250 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug misdirected ops = true 2024-04-19T01:02:35.250 INFO:teuthology.orchestra.run.smithi118.stdout: debug monc = 20 2024-04-19T01:02:35.251 INFO:teuthology.orchestra.run.smithi118.stdout: bluestore block size = 96636764160 2024-04-19T01:02:35.251 INFO:teuthology.orchestra.run.smithi118.stdout: bluestore compression algorithm = zlib 2024-04-19T01:02:35.251 INFO:teuthology.orchestra.run.smithi118.stdout: bluestore compression mode = aggressive 2024-04-19T01:02:35.251 INFO:teuthology.orchestra.run.smithi118.stdout: bluestore fsck on mount = True 2024-04-19T01:02:35.251 INFO:teuthology.orchestra.run.smithi118.stdout: bluestore zero block detection = True 2024-04-19T01:02:35.251 INFO:teuthology.orchestra.run.smithi118.stdout: debug bluefs = 20 2024-04-19T01:02:35.251 INFO:teuthology.orchestra.run.smithi118.stdout: debug bluestore = 20 2024-04-19T01:02:35.251 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2024-04-19T01:02:35.251 INFO:teuthology.orchestra.run.smithi118.stdout: debug osd = 20 2024-04-19T01:02:35.251 INFO:teuthology.orchestra.run.smithi118.stdout: debug rocksdb = 10 2024-04-19T01:02:35.251 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd backfillfull_ratio = 0.85 2024-04-19T01:02:35.251 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd full ratio = 0.9 2024-04-19T01:02:35.251 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd nearfull ratio = 0.8 2024-04-19T01:02:35.251 INFO:teuthology.orchestra.run.smithi118.stdout: osd blocked scrub grace period = 3600 2024-04-19T01:02:35.251 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug reject backfill probability = 0.3 2024-04-19T01:02:35.252 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug verify cached snaps = True 2024-04-19T01:02:35.252 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug verify missing on start = True 2024-04-19T01:02:35.252 INFO:teuthology.orchestra.run.smithi118.stdout: osd failsafe full ratio = 0.95 2024-04-19T01:02:35.252 INFO:teuthology.orchestra.run.smithi118.stdout: osd max backfills = 3 2024-04-19T01:02:35.252 INFO:teuthology.orchestra.run.smithi118.stdout: osd max markdown count = 1000 2024-04-19T01:02:35.252 INFO:teuthology.orchestra.run.smithi118.stdout: osd mclock override recovery settings = True 2024-04-19T01:02:35.252 INFO:teuthology.orchestra.run.smithi118.stdout: osd objectstore = bluestore 2024-04-19T01:02:35.252 INFO:teuthology.orchestra.run.smithi118.stdout: osd op queue = debug_random 2024-04-19T01:02:35.252 INFO:teuthology.orchestra.run.smithi118.stdout: osd op queue cut off = debug_random 2024-04-19T01:02:35.252 INFO:teuthology.orchestra.run.smithi118.stdout: osd scrub min interval = 60 2024-04-19T01:02:35.252 INFO:teuthology.orchestra.run.smithi118.stdout: osd shutdown pgref assert = True 2024-04-19T01:02:35.252 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-19T01:02:35.252 INFO:teuthology.orchestra.run.smithi118.stdout:[mgr] 2024-04-19T01:02:35.252 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2024-04-19T01:02:35.252 INFO:teuthology.orchestra.run.smithi118.stdout: debug mgr = 20 2024-04-19T01:02:35.253 INFO:teuthology.orchestra.run.smithi118.stdout: debug mon = 20 2024-04-19T01:02:35.253 INFO:teuthology.orchestra.run.smithi118.stdout: debug auth = 20 2024-04-19T01:02:35.253 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min pgs per osd = 4 2024-04-19T01:02:35.253 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min bytes per osd = 10 2024-04-19T01:02:35.253 INFO:teuthology.orchestra.run.smithi118.stdout: mgr/telemetry/nag = false 2024-04-19T01:02:35.253 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-19T01:02:35.253 INFO:teuthology.orchestra.run.smithi118.stdout:[mon] 2024-04-19T01:02:35.253 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2024-04-19T01:02:35.253 INFO:teuthology.orchestra.run.smithi118.stdout: debug mon = 20 2024-04-19T01:02:35.253 INFO:teuthology.orchestra.run.smithi118.stdout: debug paxos = 20 2024-04-19T01:02:35.253 INFO:teuthology.orchestra.run.smithi118.stdout: debug auth = 20 2024-04-19T01:02:35.253 INFO:teuthology.orchestra.run.smithi118.stdout: mon data avail warn = 5 2024-04-19T01:02:35.253 INFO:teuthology.orchestra.run.smithi118.stdout: mon mgr mkfs grace = 240 2024-04-19T01:02:35.253 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min pgs per osd = 4 2024-04-19T01:02:35.253 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd reporter subtree level = osd 2024-04-19T01:02:35.254 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd prime pg temp = true 2024-04-19T01:02:35.254 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min bytes per osd = 10 2024-04-19T01:02:35.254 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-19T01:02:35.254 INFO:teuthology.orchestra.run.smithi118.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-19T01:02:35.254 INFO:teuthology.orchestra.run.smithi118.stdout: auth mon ticket ttl = 660 # 11m 2024-04-19T01:02:35.254 INFO:teuthology.orchestra.run.smithi118.stdout: auth service ticket ttl = 240 # 4m 2024-04-19T01:02:35.254 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-19T01:02:35.254 INFO:teuthology.orchestra.run.smithi118.stdout: # don't complain about insecure global_id in the test suite 2024-04-19T01:02:35.254 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-19T01:02:35.254 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-19T01:02:35.254 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-19T01:02:35.254 INFO:teuthology.orchestra.run.smithi118.stdout: # 1m isn't quite enough 2024-04-19T01:02:35.254 INFO:teuthology.orchestra.run.smithi118.stdout: mon_down_mkfs_grace = 2m 2024-04-19T01:02:35.254 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-19T01:02:35.255 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_filestore_osds = false 2024-04-19T01:02:35.255 INFO:teuthology.orchestra.run.smithi118.stdout: mon scrub interval = 300 2024-04-19T01:02:35.255 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-19T01:02:35.255 INFO:teuthology.orchestra.run.smithi118.stdout:[client] 2024-04-19T01:02:35.255 INFO:teuthology.orchestra.run.smithi118.stdout: rgw cache enabled = true 2024-04-19T01:02:35.255 INFO:teuthology.orchestra.run.smithi118.stdout: rgw enable ops log = true 2024-04-19T01:02:35.255 INFO:teuthology.orchestra.run.smithi118.stdout: rgw enable usage log = true 2024-04-19T01:02:35.255 INFO:teuthology.orchestra.run.smithi118.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-19T01:02:35.255 INFO:teuthology.orchestra.run.smithi118.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-19T01:02:35.255 INFO:teuthology.orchestra.run.smithi118.stdout:[mon.a] 2024-04-19T01:02:35.255 INFO:teuthology.orchestra.run.smithi118.stdout:[mon.b] 2024-04-19T01:02:35.255 INFO:teuthology.orchestra.run.smithi118.stdout:[mon.c] 2024-04-19T01:02:35.256 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-19T01:02:35.256 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-19T01:02:35.363 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-19T01:02:35.363 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-19T01:02:35.363 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-19T01:02:35.369 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-19T01:02:35.369 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-19T01:02:35.418 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi033.front.sepia.ceph.com 2024-04-19T01:02:35.418 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-19T01:02:35.418 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-19T01:02:35.418 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-19T01:02:35.486 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-19T01:02:35.486 DEBUG:teuthology.orchestra.run.smithi033:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-19T01:02:35.532 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi110.front.sepia.ceph.com 2024-04-19T01:02:35.532 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-19T01:02:35.532 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-19T01:02:35.532 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-19T01:02:35.556 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-19T01:02:35.556 DEBUG:teuthology.orchestra.run.smithi110:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-19T01:02:35.602 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi118.front.sepia.ceph.com 2024-04-19T01:02:35.603 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-04-19T01:02:35.603 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-19T01:02:35.603 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-19T01:02:35.627 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-04-19T01:02:35.627 DEBUG:teuthology.orchestra.run.smithi118:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-19T01:02:35.675 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi155.front.sepia.ceph.com 2024-04-19T01:02:35.676 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-19T01:02:35.676 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-19T01:02:35.676 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-19T01:02:35.699 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-19T01:02:35.699 DEBUG:teuthology.orchestra.run.smithi155:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-19T01:02:35.750 INFO:tasks.ceph:Setting up mon nodes... 2024-04-19T01:02:35.750 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-19T01:02:35.751 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-04-19T01:02:35.797 INFO:teuthology.orchestra.run.smithi033.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-04-19T01:02:35.800 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-19T01:02:35.843 INFO:teuthology.orchestra.run.smithi155.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-19T01:02:35.856 INFO:tasks.ceph:Setting up mds nodes... 2024-04-19T01:02:35.856 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-19T01:02:35.857 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-19T01:02:35.933 INFO:teuthology.orchestra.run.smithi155.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-19T01:02:35.943 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-19T01:02:35.943 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi033.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi110.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi118.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi155.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2024-04-19T01:02:35.944 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-19T01:02:35.958 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-04-19T01:02:35.958 INFO:tasks.ceph:role: osd.0 2024-04-19T01:02:35.958 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi033.front.sepia.ceph.com 2024-04-19T01:02:35.958 DEBUG:teuthology.orchestra.run.smithi033:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-19T01:02:36.019 INFO:teuthology.orchestra.run.smithi033.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T01:02:36.019 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T01:02:36.019 INFO:teuthology.orchestra.run.smithi033.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T01:02:36.020 INFO:teuthology.orchestra.run.smithi033.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T01:02:36.020 INFO:teuthology.orchestra.run.smithi033.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T01:02:36.020 INFO:teuthology.orchestra.run.smithi033.stdout: = sunit=0 swidth=0 blks 2024-04-19T01:02:36.020 INFO:teuthology.orchestra.run.smithi033.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T01:02:36.020 INFO:teuthology.orchestra.run.smithi033.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T01:02:36.020 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T01:02:36.021 INFO:teuthology.orchestra.run.smithi033.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T01:02:36.486 INFO:teuthology.orchestra.run.smithi033.stdout:Discarding blocks...Done. 2024-04-19T01:02:36.488 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi033.front.sepia.ceph.com -o noatime 2024-04-19T01:02:36.488 DEBUG:teuthology.orchestra.run.smithi033:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-19T01:02:36.573 DEBUG:teuthology.orchestra.run.smithi033:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-19T01:02:36.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T01:02:36.629 INFO:teuthology.orchestra.run.smithi033.stderr:sudo: /sbin/restorecon: command not found 2024-04-19T01:02:36.629 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-19T01:02:36.684 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-04-19T01:02:36.684 INFO:tasks.ceph:role: osd.4 2024-04-19T01:02:36.684 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi033.front.sepia.ceph.com 2024-04-19T01:02:36.684 DEBUG:teuthology.orchestra.run.smithi033:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-19T01:02:36.748 INFO:teuthology.orchestra.run.smithi033.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T01:02:36.749 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T01:02:36.749 INFO:teuthology.orchestra.run.smithi033.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T01:02:36.749 INFO:teuthology.orchestra.run.smithi033.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T01:02:36.749 INFO:teuthology.orchestra.run.smithi033.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T01:02:36.749 INFO:teuthology.orchestra.run.smithi033.stdout: = sunit=0 swidth=0 blks 2024-04-19T01:02:36.749 INFO:teuthology.orchestra.run.smithi033.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T01:02:36.749 INFO:teuthology.orchestra.run.smithi033.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T01:02:36.749 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T01:02:36.749 INFO:teuthology.orchestra.run.smithi033.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T01:02:37.226 INFO:teuthology.orchestra.run.smithi033.stdout:Discarding blocks...Done. 2024-04-19T01:02:37.228 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi033.front.sepia.ceph.com -o noatime 2024-04-19T01:02:37.228 DEBUG:teuthology.orchestra.run.smithi033:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-04-19T01:02:37.376 DEBUG:teuthology.orchestra.run.smithi033:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-19T01:02:37.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T01:02:37.388 INFO:teuthology.orchestra.run.smithi033.stderr:sudo: /sbin/restorecon: command not found 2024-04-19T01:02:37.389 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/osd/ceph-8 2024-04-19T01:02:37.444 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-04-19T01:02:37.444 INFO:tasks.ceph:role: osd.8 2024-04-19T01:02:37.444 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi033.front.sepia.ceph.com 2024-04-19T01:02:37.445 DEBUG:teuthology.orchestra.run.smithi033:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-19T01:02:37.510 INFO:teuthology.orchestra.run.smithi033.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T01:02:37.510 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T01:02:37.510 INFO:teuthology.orchestra.run.smithi033.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T01:02:37.510 INFO:teuthology.orchestra.run.smithi033.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T01:02:37.510 INFO:teuthology.orchestra.run.smithi033.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T01:02:37.510 INFO:teuthology.orchestra.run.smithi033.stdout: = sunit=0 swidth=0 blks 2024-04-19T01:02:37.510 INFO:teuthology.orchestra.run.smithi033.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T01:02:37.511 INFO:teuthology.orchestra.run.smithi033.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T01:02:37.511 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T01:02:37.511 INFO:teuthology.orchestra.run.smithi033.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T01:02:37.991 INFO:teuthology.orchestra.run.smithi033.stdout:Discarding blocks...Done. 2024-04-19T01:02:37.993 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi033.front.sepia.ceph.com -o noatime 2024-04-19T01:02:37.993 DEBUG:teuthology.orchestra.run.smithi033:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-8 2024-04-19T01:02:38.135 DEBUG:teuthology.orchestra.run.smithi033:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-8 2024-04-19T01:02:38.147 INFO:teuthology.orchestra.run.smithi033.stderr:sudo: /sbin/restorecon: command not found 2024-04-19T01:02:38.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T01:02:38.148 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/osd/ceph-12 2024-04-19T01:02:38.207 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-04-19T01:02:38.207 INFO:tasks.ceph:role: osd.12 2024-04-19T01:02:38.207 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi033.front.sepia.ceph.com 2024-04-19T01:02:38.207 DEBUG:teuthology.orchestra.run.smithi033:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-19T01:02:38.270 INFO:teuthology.orchestra.run.smithi033.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T01:02:38.271 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T01:02:38.271 INFO:teuthology.orchestra.run.smithi033.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T01:02:38.271 INFO:teuthology.orchestra.run.smithi033.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T01:02:38.271 INFO:teuthology.orchestra.run.smithi033.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T01:02:38.271 INFO:teuthology.orchestra.run.smithi033.stdout: = sunit=0 swidth=0 blks 2024-04-19T01:02:38.271 INFO:teuthology.orchestra.run.smithi033.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T01:02:38.271 INFO:teuthology.orchestra.run.smithi033.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T01:02:38.271 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T01:02:38.271 INFO:teuthology.orchestra.run.smithi033.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T01:02:38.718 INFO:teuthology.orchestra.run.smithi033.stdout:Discarding blocks...Done. 2024-04-19T01:02:38.720 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi033.front.sepia.ceph.com -o noatime 2024-04-19T01:02:38.721 DEBUG:teuthology.orchestra.run.smithi033:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-12 2024-04-19T01:02:38.808 DEBUG:teuthology.orchestra.run.smithi033:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-12 2024-04-19T01:02:38.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T01:02:38.818 INFO:teuthology.orchestra.run.smithi033.stderr:sudo: /sbin/restorecon: command not found 2024-04-19T01:02:38.818 DEBUG:teuthology.orchestra.run.smithi033:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-19T01:02:38.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:02:38.931+0000 7f3526ef77c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-19T01:02:38.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:02:38.931+0000 7f3526ef77c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-19T01:02:38.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:02:38.931+0000 7f3526ef77c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-19T01:02:38.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:02:38.931+0000 7f3526ef77c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-19T01:02:38.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:02:38.931+0000 7f3526ef77c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-19T01:02:38.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:02:38.935+0000 7f3526ef77c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-19T01:02:41.908 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-19T01:02:41.922 DEBUG:teuthology.orchestra.run.smithi033:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-19T01:02:42.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:02:42.010+0000 7fd7505417c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-19T01:02:42.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:02:42.010+0000 7fd7505417c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-19T01:02:42.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:02:42.010+0000 7fd7505417c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-19T01:02:42.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:02:42.010+0000 7fd7505417c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-19T01:02:42.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:02:42.010+0000 7fd7505417c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-19T01:02:42.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:02:42.010+0000 7fd7505417c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-19T01:02:44.975 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-19T01:02:45.033 DEBUG:teuthology.orchestra.run.smithi033:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 8 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-19T01:02:45.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:02:45.118+0000 7ff36473a7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-8/keyring: can't open /var/lib/ceph/osd/ceph-8/keyring: (2) No such file or directory 2024-04-19T01:02:45.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:02:45.118+0000 7ff36473a7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-8/keyring 2024-04-19T01:02:45.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:02:45.118+0000 7ff36473a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2024-04-19T01:02:45.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:02:45.118+0000 7ff36473a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2024-04-19T01:02:45.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:02:45.118+0000 7ff36473a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2024-04-19T01:02:45.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:02:45.118+0000 7ff36473a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-8) _read_fsid unparsable uuid 2024-04-19T01:02:48.112 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2024-04-19T01:02:48.168 DEBUG:teuthology.orchestra.run.smithi033:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 12 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-19T01:02:48.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:02:48.254+0000 7f62b71237c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-12/keyring: can't open /var/lib/ceph/osd/ceph-12/keyring: (2) No such file or directory 2024-04-19T01:02:48.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:02:48.254+0000 7f62b71237c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-12/keyring 2024-04-19T01:02:48.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:02:48.254+0000 7f62b71237c0 -1 bluestore(/var/lib/ceph/osd/ceph-12/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-12/block: (2) No such file or directory 2024-04-19T01:02:48.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:02:48.254+0000 7f62b71237c0 -1 bluestore(/var/lib/ceph/osd/ceph-12/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-12/block: (2) No such file or directory 2024-04-19T01:02:48.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:02:48.254+0000 7f62b71237c0 -1 bluestore(/var/lib/ceph/osd/ceph-12/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-12/block: (2) No such file or directory 2024-04-19T01:02:48.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:02:48.254+0000 7f62b71237c0 -1 bluestore(/var/lib/ceph/osd/ceph-12) _read_fsid unparsable uuid 2024-04-19T01:02:51.244 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-12 2024-04-19T01:02:51.301 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-19T01:02:51.315 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-04-19T01:02:51.315 INFO:tasks.ceph:role: osd.1 2024-04-19T01:02:51.315 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi110.front.sepia.ceph.com 2024-04-19T01:02:51.315 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-19T01:02:51.371 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T01:02:51.371 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T01:02:51.371 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T01:02:51.371 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T01:02:51.371 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T01:02:51.371 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2024-04-19T01:02:51.371 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T01:02:51.371 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T01:02:51.371 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T01:02:51.371 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T01:02:51.395 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2024-04-19T01:02:51.396 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2024-04-19T01:02:51.396 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-1 2024-04-19T01:02:51.457 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-19T01:02:51.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T01:02:51.511 INFO:teuthology.orchestra.run.smithi110.stderr:sudo: /sbin/restorecon: command not found 2024-04-19T01:02:51.511 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-19T01:02:51.564 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-04-19T01:02:51.564 INFO:tasks.ceph:role: osd.5 2024-04-19T01:02:51.564 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi110.front.sepia.ceph.com 2024-04-19T01:02:51.564 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-19T01:02:51.619 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T01:02:51.620 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T01:02:51.620 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T01:02:51.620 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T01:02:51.620 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T01:02:51.620 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2024-04-19T01:02:51.620 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T01:02:51.620 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T01:02:51.620 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T01:02:51.620 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T01:02:51.644 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2024-04-19T01:02:51.645 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2024-04-19T01:02:51.646 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-04-19T01:02:51.706 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-19T01:02:51.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T01:02:51.758 INFO:teuthology.orchestra.run.smithi110.stderr:sudo: /sbin/restorecon: command not found 2024-04-19T01:02:51.758 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-9 2024-04-19T01:02:51.814 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-04-19T01:02:51.814 INFO:tasks.ceph:role: osd.9 2024-04-19T01:02:51.814 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi110.front.sepia.ceph.com 2024-04-19T01:02:51.814 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-19T01:02:51.870 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T01:02:51.870 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T01:02:51.870 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T01:02:51.870 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T01:02:51.870 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T01:02:51.870 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2024-04-19T01:02:51.870 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T01:02:51.870 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T01:02:51.870 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T01:02:51.870 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T01:02:51.893 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2024-04-19T01:02:51.895 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2024-04-19T01:02:51.895 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-9 2024-04-19T01:02:51.959 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-9 2024-04-19T01:02:52.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T01:02:52.010 INFO:teuthology.orchestra.run.smithi110.stderr:sudo: /sbin/restorecon: command not found 2024-04-19T01:02:52.010 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-13 2024-04-19T01:02:52.066 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-04-19T01:02:52.066 INFO:tasks.ceph:role: osd.13 2024-04-19T01:02:52.066 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi110.front.sepia.ceph.com 2024-04-19T01:02:52.066 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-19T01:02:52.121 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T01:02:52.121 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T01:02:52.121 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T01:02:52.121 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T01:02:52.121 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T01:02:52.121 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2024-04-19T01:02:52.121 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T01:02:52.121 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T01:02:52.121 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T01:02:52.122 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T01:02:52.144 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2024-04-19T01:02:52.146 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2024-04-19T01:02:52.146 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-13 2024-04-19T01:02:52.208 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-13 2024-04-19T01:02:52.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T01:02:52.258 INFO:teuthology.orchestra.run.smithi110.stderr:sudo: /sbin/restorecon: command not found 2024-04-19T01:02:52.258 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-19T01:02:52.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-19T01:02:52.393+0000 7f2c76fd07c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-19T01:02:52.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-19T01:02:52.393+0000 7f2c76fd07c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-19T01:02:52.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-19T01:02:52.393+0000 7f2c76fd07c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-19T01:02:52.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-19T01:02:52.393+0000 7f2c76fd07c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-19T01:02:52.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-19T01:02:52.393+0000 7f2c76fd07c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-19T01:02:52.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-19T01:02:52.393+0000 7f2c76fd07c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-19T01:02:55.384 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-19T01:02:55.398 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-19T01:02:55.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-19T01:02:55.477+0000 7f964093b7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-19T01:02:55.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-19T01:02:55.477+0000 7f964093b7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-19T01:02:55.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-19T01:02:55.477+0000 7f964093b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-19T01:02:55.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-19T01:02:55.477+0000 7f964093b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-19T01:02:55.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-19T01:02:55.477+0000 7f964093b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-19T01:02:55.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-19T01:02:55.477+0000 7f964093b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-19T01:02:58.450 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-19T01:02:58.505 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 9 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-19T01:02:58.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-19T01:02:58.585+0000 7f9b58adc7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-9/keyring: can't open /var/lib/ceph/osd/ceph-9/keyring: (2) No such file or directory 2024-04-19T01:02:58.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-19T01:02:58.589+0000 7f9b58adc7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-9/keyring 2024-04-19T01:02:58.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-19T01:02:58.589+0000 7f9b58adc7c0 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2024-04-19T01:02:58.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-19T01:02:58.589+0000 7f9b58adc7c0 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2024-04-19T01:02:58.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-19T01:02:58.589+0000 7f9b58adc7c0 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2024-04-19T01:02:58.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-19T01:02:58.589+0000 7f9b58adc7c0 -1 bluestore(/var/lib/ceph/osd/ceph-9) _read_fsid unparsable uuid 2024-04-19T01:03:01.560 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-9 2024-04-19T01:03:01.614 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 13 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-19T01:03:01.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-19T01:03:01.693+0000 7fb2308d47c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-13/keyring: can't open /var/lib/ceph/osd/ceph-13/keyring: (2) No such file or directory 2024-04-19T01:03:01.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-19T01:03:01.693+0000 7fb2308d47c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-13/keyring 2024-04-19T01:03:01.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-19T01:03:01.693+0000 7fb2308d47c0 -1 bluestore(/var/lib/ceph/osd/ceph-13/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-13/block: (2) No such file or directory 2024-04-19T01:03:01.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-19T01:03:01.693+0000 7fb2308d47c0 -1 bluestore(/var/lib/ceph/osd/ceph-13/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-13/block: (2) No such file or directory 2024-04-19T01:03:01.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-19T01:03:01.693+0000 7fb2308d47c0 -1 bluestore(/var/lib/ceph/osd/ceph-13/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-13/block: (2) No such file or directory 2024-04-19T01:03:01.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-19T01:03:01.693+0000 7fb2308d47c0 -1 bluestore(/var/lib/ceph/osd/ceph-13) _read_fsid unparsable uuid 2024-04-19T01:03:04.689 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-13 2024-04-19T01:03:04.746 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-19T01:03:04.760 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-04-19T01:03:04.760 INFO:tasks.ceph:role: osd.2 2024-04-19T01:03:04.760 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi118.front.sepia.ceph.com 2024-04-19T01:03:04.760 DEBUG:teuthology.orchestra.run.smithi118:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-19T01:03:04.815 INFO:teuthology.orchestra.run.smithi118.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T01:03:04.815 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T01:03:04.815 INFO:teuthology.orchestra.run.smithi118.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T01:03:04.815 INFO:teuthology.orchestra.run.smithi118.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T01:03:04.815 INFO:teuthology.orchestra.run.smithi118.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T01:03:04.815 INFO:teuthology.orchestra.run.smithi118.stdout: = sunit=0 swidth=0 blks 2024-04-19T01:03:04.815 INFO:teuthology.orchestra.run.smithi118.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T01:03:04.816 INFO:teuthology.orchestra.run.smithi118.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T01:03:04.816 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T01:03:04.816 INFO:teuthology.orchestra.run.smithi118.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T01:03:04.839 INFO:teuthology.orchestra.run.smithi118.stdout:Discarding blocks...Done. 2024-04-19T01:03:04.842 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi118.front.sepia.ceph.com -o noatime 2024-04-19T01:03:04.842 DEBUG:teuthology.orchestra.run.smithi118:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2024-04-19T01:03:04.901 DEBUG:teuthology.orchestra.run.smithi118:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-19T01:03:04.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T01:03:04.956 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: /sbin/restorecon: command not found 2024-04-19T01:03:04.956 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-19T01:03:05.009 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-04-19T01:03:05.009 INFO:tasks.ceph:role: osd.6 2024-04-19T01:03:05.010 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi118.front.sepia.ceph.com 2024-04-19T01:03:05.010 DEBUG:teuthology.orchestra.run.smithi118:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-19T01:03:05.067 INFO:teuthology.orchestra.run.smithi118.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T01:03:05.067 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T01:03:05.067 INFO:teuthology.orchestra.run.smithi118.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T01:03:05.067 INFO:teuthology.orchestra.run.smithi118.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T01:03:05.067 INFO:teuthology.orchestra.run.smithi118.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T01:03:05.067 INFO:teuthology.orchestra.run.smithi118.stdout: = sunit=0 swidth=0 blks 2024-04-19T01:03:05.067 INFO:teuthology.orchestra.run.smithi118.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T01:03:05.067 INFO:teuthology.orchestra.run.smithi118.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T01:03:05.067 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T01:03:05.067 INFO:teuthology.orchestra.run.smithi118.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T01:03:05.091 INFO:teuthology.orchestra.run.smithi118.stdout:Discarding blocks...Done. 2024-04-19T01:03:05.092 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi118.front.sepia.ceph.com -o noatime 2024-04-19T01:03:05.093 DEBUG:teuthology.orchestra.run.smithi118:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-6 2024-04-19T01:03:05.150 DEBUG:teuthology.orchestra.run.smithi118:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-04-19T01:03:05.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T01:03:05.204 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: /sbin/restorecon: command not found 2024-04-19T01:03:05.204 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/osd/ceph-10 2024-04-19T01:03:05.257 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-04-19T01:03:05.257 INFO:tasks.ceph:role: osd.10 2024-04-19T01:03:05.257 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi118.front.sepia.ceph.com 2024-04-19T01:03:05.257 DEBUG:teuthology.orchestra.run.smithi118:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-19T01:03:05.311 INFO:teuthology.orchestra.run.smithi118.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T01:03:05.311 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T01:03:05.311 INFO:teuthology.orchestra.run.smithi118.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T01:03:05.311 INFO:teuthology.orchestra.run.smithi118.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T01:03:05.311 INFO:teuthology.orchestra.run.smithi118.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T01:03:05.311 INFO:teuthology.orchestra.run.smithi118.stdout: = sunit=0 swidth=0 blks 2024-04-19T01:03:05.311 INFO:teuthology.orchestra.run.smithi118.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T01:03:05.311 INFO:teuthology.orchestra.run.smithi118.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T01:03:05.312 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T01:03:05.312 INFO:teuthology.orchestra.run.smithi118.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T01:03:05.335 INFO:teuthology.orchestra.run.smithi118.stdout:Discarding blocks...Done. 2024-04-19T01:03:05.336 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi118.front.sepia.ceph.com -o noatime 2024-04-19T01:03:05.337 DEBUG:teuthology.orchestra.run.smithi118:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-10 2024-04-19T01:03:05.399 DEBUG:teuthology.orchestra.run.smithi118:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-10 2024-04-19T01:03:05.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T01:03:05.451 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: /sbin/restorecon: command not found 2024-04-19T01:03:05.452 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/osd/ceph-14 2024-04-19T01:03:05.506 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-04-19T01:03:05.506 INFO:tasks.ceph:role: osd.14 2024-04-19T01:03:05.506 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi118.front.sepia.ceph.com 2024-04-19T01:03:05.506 DEBUG:teuthology.orchestra.run.smithi118:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-19T01:03:05.563 INFO:teuthology.orchestra.run.smithi118.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T01:03:05.563 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T01:03:05.563 INFO:teuthology.orchestra.run.smithi118.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T01:03:05.563 INFO:teuthology.orchestra.run.smithi118.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T01:03:05.563 INFO:teuthology.orchestra.run.smithi118.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T01:03:05.563 INFO:teuthology.orchestra.run.smithi118.stdout: = sunit=0 swidth=0 blks 2024-04-19T01:03:05.563 INFO:teuthology.orchestra.run.smithi118.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T01:03:05.563 INFO:teuthology.orchestra.run.smithi118.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T01:03:05.563 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T01:03:05.563 INFO:teuthology.orchestra.run.smithi118.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T01:03:05.587 INFO:teuthology.orchestra.run.smithi118.stdout:Discarding blocks...Done. 2024-04-19T01:03:05.588 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi118.front.sepia.ceph.com -o noatime 2024-04-19T01:03:05.588 DEBUG:teuthology.orchestra.run.smithi118:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-14 2024-04-19T01:03:05.646 DEBUG:teuthology.orchestra.run.smithi118:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-14 2024-04-19T01:03:05.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T01:03:05.700 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: /sbin/restorecon: command not found 2024-04-19T01:03:05.700 DEBUG:teuthology.orchestra.run.smithi118:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-19T01:03:05.803 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-19T01:03:05.802+0000 7f22d18747c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-19T01:03:05.804 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-19T01:03:05.802+0000 7f22d18747c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-19T01:03:05.804 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-19T01:03:05.802+0000 7f22d18747c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-19T01:03:05.804 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-19T01:03:05.802+0000 7f22d18747c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-19T01:03:05.804 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-19T01:03:05.802+0000 7f22d18747c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-19T01:03:05.804 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-19T01:03:05.802+0000 7f22d18747c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-19T01:03:08.792 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-19T01:03:08.806 DEBUG:teuthology.orchestra.run.smithi118:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-19T01:03:08.889 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-19T01:03:08.886+0000 7f67c7f477c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-19T01:03:08.889 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-19T01:03:08.886+0000 7f67c7f477c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-19T01:03:08.889 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-19T01:03:08.886+0000 7f67c7f477c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-19T01:03:08.889 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-19T01:03:08.886+0000 7f67c7f477c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-19T01:03:08.889 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-19T01:03:08.886+0000 7f67c7f477c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-19T01:03:08.889 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-19T01:03:08.886+0000 7f67c7f477c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-19T01:03:11.853 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-19T01:03:11.907 DEBUG:teuthology.orchestra.run.smithi118:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 10 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-19T01:03:11.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-19T01:03:11.986+0000 7f0230ce47c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-10/keyring: can't open /var/lib/ceph/osd/ceph-10/keyring: (2) No such file or directory 2024-04-19T01:03:11.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-19T01:03:11.986+0000 7f0230ce47c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-10/keyring 2024-04-19T01:03:11.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-19T01:03:11.986+0000 7f0230ce47c0 -1 bluestore(/var/lib/ceph/osd/ceph-10/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-10/block: (2) No such file or directory 2024-04-19T01:03:11.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-19T01:03:11.986+0000 7f0230ce47c0 -1 bluestore(/var/lib/ceph/osd/ceph-10/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-10/block: (2) No such file or directory 2024-04-19T01:03:11.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-19T01:03:11.986+0000 7f0230ce47c0 -1 bluestore(/var/lib/ceph/osd/ceph-10/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-10/block: (2) No such file or directory 2024-04-19T01:03:11.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-19T01:03:11.986+0000 7f0230ce47c0 -1 bluestore(/var/lib/ceph/osd/ceph-10) _read_fsid unparsable uuid 2024-04-19T01:03:14.951 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-10 2024-04-19T01:03:15.007 DEBUG:teuthology.orchestra.run.smithi118:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 14 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-19T01:03:15.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-19T01:03:15.086+0000 7fbfee21e7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-14/keyring: can't open /var/lib/ceph/osd/ceph-14/keyring: (2) No such file or directory 2024-04-19T01:03:15.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-19T01:03:15.086+0000 7fbfee21e7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-14/keyring 2024-04-19T01:03:15.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-19T01:03:15.086+0000 7fbfee21e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-14/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-14/block: (2) No such file or directory 2024-04-19T01:03:15.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-19T01:03:15.086+0000 7fbfee21e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-14/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-14/block: (2) No such file or directory 2024-04-19T01:03:15.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-19T01:03:15.086+0000 7fbfee21e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-14/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-14/block: (2) No such file or directory 2024-04-19T01:03:15.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-19T01:03:15.086+0000 7fbfee21e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-14) _read_fsid unparsable uuid 2024-04-19T01:03:18.055 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-14 2024-04-19T01:03:18.114 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-19T01:03:18.128 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-04-19T01:03:18.129 INFO:tasks.ceph:role: osd.3 2024-04-19T01:03:18.129 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi155.front.sepia.ceph.com 2024-04-19T01:03:18.129 DEBUG:teuthology.orchestra.run.smithi155:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-19T01:03:18.192 INFO:teuthology.orchestra.run.smithi155.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T01:03:18.192 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T01:03:18.192 INFO:teuthology.orchestra.run.smithi155.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T01:03:18.192 INFO:teuthology.orchestra.run.smithi155.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T01:03:18.192 INFO:teuthology.orchestra.run.smithi155.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T01:03:18.192 INFO:teuthology.orchestra.run.smithi155.stdout: = sunit=0 swidth=0 blks 2024-04-19T01:03:18.192 INFO:teuthology.orchestra.run.smithi155.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T01:03:18.192 INFO:teuthology.orchestra.run.smithi155.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T01:03:18.192 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T01:03:18.192 INFO:teuthology.orchestra.run.smithi155.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T01:03:18.642 INFO:teuthology.orchestra.run.smithi155.stdout:Discarding blocks...Done. 2024-04-19T01:03:18.644 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi155.front.sepia.ceph.com -o noatime 2024-04-19T01:03:18.644 DEBUG:teuthology.orchestra.run.smithi155:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-04-19T01:03:18.748 DEBUG:teuthology.orchestra.run.smithi155:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-19T01:03:18.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T01:03:18.759 INFO:teuthology.orchestra.run.smithi155.stderr:sudo: /sbin/restorecon: command not found 2024-04-19T01:03:18.759 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-19T01:03:18.814 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-04-19T01:03:18.814 INFO:tasks.ceph:role: osd.7 2024-04-19T01:03:18.814 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi155.front.sepia.ceph.com 2024-04-19T01:03:18.814 DEBUG:teuthology.orchestra.run.smithi155:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-19T01:03:18.878 INFO:teuthology.orchestra.run.smithi155.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T01:03:18.879 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T01:03:18.879 INFO:teuthology.orchestra.run.smithi155.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T01:03:18.879 INFO:teuthology.orchestra.run.smithi155.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T01:03:18.879 INFO:teuthology.orchestra.run.smithi155.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T01:03:18.879 INFO:teuthology.orchestra.run.smithi155.stdout: = sunit=0 swidth=0 blks 2024-04-19T01:03:18.879 INFO:teuthology.orchestra.run.smithi155.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T01:03:18.879 INFO:teuthology.orchestra.run.smithi155.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T01:03:18.879 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T01:03:18.879 INFO:teuthology.orchestra.run.smithi155.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T01:03:19.346 INFO:teuthology.orchestra.run.smithi155.stdout:Discarding blocks...Done. 2024-04-19T01:03:19.348 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi155.front.sepia.ceph.com -o noatime 2024-04-19T01:03:19.349 DEBUG:teuthology.orchestra.run.smithi155:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-7 2024-04-19T01:03:19.503 DEBUG:teuthology.orchestra.run.smithi155:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-04-19T01:03:19.515 INFO:teuthology.orchestra.run.smithi155.stderr:sudo: /sbin/restorecon: command not found 2024-04-19T01:03:19.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T01:03:19.516 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/osd/ceph-11 2024-04-19T01:03:19.571 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-04-19T01:03:19.571 INFO:tasks.ceph:role: osd.11 2024-04-19T01:03:19.571 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi155.front.sepia.ceph.com 2024-04-19T01:03:19.571 DEBUG:teuthology.orchestra.run.smithi155:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-19T01:03:19.635 INFO:teuthology.orchestra.run.smithi155.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T01:03:19.635 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T01:03:19.636 INFO:teuthology.orchestra.run.smithi155.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T01:03:19.636 INFO:teuthology.orchestra.run.smithi155.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T01:03:19.636 INFO:teuthology.orchestra.run.smithi155.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T01:03:19.636 INFO:teuthology.orchestra.run.smithi155.stdout: = sunit=0 swidth=0 blks 2024-04-19T01:03:19.636 INFO:teuthology.orchestra.run.smithi155.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T01:03:19.636 INFO:teuthology.orchestra.run.smithi155.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T01:03:19.636 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T01:03:19.636 INFO:teuthology.orchestra.run.smithi155.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T01:03:20.107 INFO:teuthology.orchestra.run.smithi155.stdout:Discarding blocks...Done. 2024-04-19T01:03:20.109 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi155.front.sepia.ceph.com -o noatime 2024-04-19T01:03:20.110 DEBUG:teuthology.orchestra.run.smithi155:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-11 2024-04-19T01:03:20.270 DEBUG:teuthology.orchestra.run.smithi155:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-11 2024-04-19T01:03:20.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T01:03:20.282 INFO:teuthology.orchestra.run.smithi155.stderr:sudo: /sbin/restorecon: command not found 2024-04-19T01:03:20.282 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/osd/ceph-15 2024-04-19T01:03:20.337 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-04-19T01:03:20.337 INFO:tasks.ceph:role: osd.15 2024-04-19T01:03:20.337 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi155.front.sepia.ceph.com 2024-04-19T01:03:20.337 DEBUG:teuthology.orchestra.run.smithi155:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-19T01:03:20.398 INFO:teuthology.orchestra.run.smithi155.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T01:03:20.399 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T01:03:20.399 INFO:teuthology.orchestra.run.smithi155.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T01:03:20.399 INFO:teuthology.orchestra.run.smithi155.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T01:03:20.399 INFO:teuthology.orchestra.run.smithi155.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T01:03:20.399 INFO:teuthology.orchestra.run.smithi155.stdout: = sunit=0 swidth=0 blks 2024-04-19T01:03:20.399 INFO:teuthology.orchestra.run.smithi155.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T01:03:20.399 INFO:teuthology.orchestra.run.smithi155.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T01:03:20.399 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T01:03:20.399 INFO:teuthology.orchestra.run.smithi155.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T01:03:20.819 INFO:teuthology.orchestra.run.smithi155.stdout:Discarding blocks...Done. 2024-04-19T01:03:20.821 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi155.front.sepia.ceph.com -o noatime 2024-04-19T01:03:20.821 DEBUG:teuthology.orchestra.run.smithi155:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-15 2024-04-19T01:03:20.908 DEBUG:teuthology.orchestra.run.smithi155:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-15 2024-04-19T01:03:20.918 INFO:teuthology.orchestra.run.smithi155.stderr:sudo: /sbin/restorecon: command not found 2024-04-19T01:03:20.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T01:03:20.920 DEBUG:teuthology.orchestra.run.smithi155:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-19T01:03:21.035 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-19T01:03:21.034+0000 7fb18b39f7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-19T01:03:21.038 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-19T01:03:21.034+0000 7fb18b39f7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-19T01:03:21.039 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-19T01:03:21.034+0000 7fb18b39f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-19T01:03:21.039 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-19T01:03:21.034+0000 7fb18b39f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-19T01:03:21.039 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-19T01:03:21.034+0000 7fb18b39f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-19T01:03:21.042 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-19T01:03:21.038+0000 7fb18b39f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-19T01:03:24.022 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-19T01:03:24.036 DEBUG:teuthology.orchestra.run.smithi155:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-19T01:03:24.117 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-19T01:03:24.114+0000 7f5886a327c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-19T01:03:24.118 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-19T01:03:24.114+0000 7f5886a327c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-19T01:03:24.118 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-19T01:03:24.114+0000 7f5886a327c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-19T01:03:24.118 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-19T01:03:24.114+0000 7f5886a327c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-19T01:03:24.118 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-19T01:03:24.114+0000 7f5886a327c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-19T01:03:24.118 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-19T01:03:24.114+0000 7f5886a327c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-19T01:03:27.099 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-19T01:03:27.160 DEBUG:teuthology.orchestra.run.smithi155:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 11 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-19T01:03:27.244 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-19T01:03:27.238+0000 7fca0a5e57c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-11/keyring: can't open /var/lib/ceph/osd/ceph-11/keyring: (2) No such file or directory 2024-04-19T01:03:27.244 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-19T01:03:27.242+0000 7fca0a5e57c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-11/keyring 2024-04-19T01:03:27.244 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-19T01:03:27.242+0000 7fca0a5e57c0 -1 bluestore(/var/lib/ceph/osd/ceph-11/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-11/block: (2) No such file or directory 2024-04-19T01:03:27.244 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-19T01:03:27.242+0000 7fca0a5e57c0 -1 bluestore(/var/lib/ceph/osd/ceph-11/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-11/block: (2) No such file or directory 2024-04-19T01:03:27.244 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-19T01:03:27.242+0000 7fca0a5e57c0 -1 bluestore(/var/lib/ceph/osd/ceph-11/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-11/block: (2) No such file or directory 2024-04-19T01:03:27.244 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-19T01:03:27.242+0000 7fca0a5e57c0 -1 bluestore(/var/lib/ceph/osd/ceph-11) _read_fsid unparsable uuid 2024-04-19T01:03:30.231 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-11 2024-04-19T01:03:30.292 DEBUG:teuthology.orchestra.run.smithi155:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 15 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-19T01:03:30.374 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-19T01:03:30.370+0000 7f41ab5c77c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-15/keyring: can't open /var/lib/ceph/osd/ceph-15/keyring: (2) No such file or directory 2024-04-19T01:03:30.375 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-19T01:03:30.370+0000 7f41ab5c77c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-15/keyring 2024-04-19T01:03:30.375 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-19T01:03:30.370+0000 7f41ab5c77c0 -1 bluestore(/var/lib/ceph/osd/ceph-15/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-15/block: (2) No such file or directory 2024-04-19T01:03:30.375 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-19T01:03:30.370+0000 7f41ab5c77c0 -1 bluestore(/var/lib/ceph/osd/ceph-15/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-15/block: (2) No such file or directory 2024-04-19T01:03:30.375 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-19T01:03:30.370+0000 7f41ab5c77c0 -1 bluestore(/var/lib/ceph/osd/ceph-15/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-15/block: (2) No such file or directory 2024-04-19T01:03:30.375 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-19T01:03:30.374+0000 7f41ab5c77c0 -1 bluestore(/var/lib/ceph/osd/ceph-15) _read_fsid unparsable uuid 2024-04-19T01:03:33.339 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-15 2024-04-19T01:03:33.397 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-19T01:03:33.397 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-19T01:03:33.397 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-04-19T01:03:33.412 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-19T01:03:33.412 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-19T01:03:33.467 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-19T01:03:33.468 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-19T01:03:33.527 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-19T01:03:33.527 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-8/keyring of=/dev/stdout 2024-04-19T01:03:33.583 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-19T01:03:33.583 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-12/keyring of=/dev/stdout 2024-04-19T01:03:33.639 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-19T01:03:33.639 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-19T01:03:33.654 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-19T01:03:33.654 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-19T01:03:33.707 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-19T01:03:33.708 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-9/keyring of=/dev/stdout 2024-04-19T01:03:33.759 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-19T01:03:33.759 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-13/keyring of=/dev/stdout 2024-04-19T01:03:33.811 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-04-19T01:03:33.811 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-19T01:03:33.826 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-04-19T01:03:33.826 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-19T01:03:33.883 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-04-19T01:03:33.883 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-10/keyring of=/dev/stdout 2024-04-19T01:03:33.938 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-04-19T01:03:33.938 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-14/keyring of=/dev/stdout 2024-04-19T01:03:33.998 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-19T01:03:33.998 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-19T01:03:34.015 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-19T01:03:34.015 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-19T01:03:34.071 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-19T01:03:34.071 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-19T01:03:34.127 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-19T01:03:34.127 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-11/keyring of=/dev/stdout 2024-04-19T01:03:34.182 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-19T01:03:34.182 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-15/keyring of=/dev/stdout 2024-04-19T01:03:34.235 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-19T01:03:34.236 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-19T01:03:34.283 INFO:tasks.ceph:Adding keys to all mons... 2024-04-19T01:03:34.284 DEBUG:teuthology.orchestra.run.smithi033:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-19T01:03:34.286 DEBUG:teuthology.orchestra.run.smithi110:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-19T01:03:34.288 DEBUG:teuthology.orchestra.run.smithi118:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-19T01:03:34.296 INFO:teuthology.orchestra.run.smithi033.stdout:[mgr.y] 2024-04-19T01:03:34.296 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQCrwiFmpvXILxAAGeFawyr5fMeNBoYHSMFYpQ== 2024-04-19T01:03:34.296 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.0] 2024-04-19T01:03:34.296 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQCuwiFmevuTNxAAAc7tRIU0nLP/sdvcJvgLQQ== 2024-04-19T01:03:34.296 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.4] 2024-04-19T01:03:34.296 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQCywiFmXo6yABAAtSH1hB/XEOu9x1C4vj3how== 2024-04-19T01:03:34.296 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.8] 2024-04-19T01:03:34.296 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQC1wiFm398gBxAAvG7XQNTfYV/5cJFV8FSE1A== 2024-04-19T01:03:34.296 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.12] 2024-04-19T01:03:34.297 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQC4wiFmpTg9DxAAfUZRdhbo5UBLa2ZWfye2Qw== 2024-04-19T01:03:34.297 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.1] 2024-04-19T01:03:34.297 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQC8wiFmFhK4FxAAFfT0bZlo79nY99PPTz5/gw== 2024-04-19T01:03:34.297 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.5] 2024-04-19T01:03:34.297 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQC/wiFmWDPBHBAAIhHaoDKmizrJSLue9adzgQ== 2024-04-19T01:03:34.297 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.9] 2024-04-19T01:03:34.297 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQDCwiFmiq5XIxAA9OfcY78r1kA0QiEsXWvtSg== 2024-04-19T01:03:34.297 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.13] 2024-04-19T01:03:34.297 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQDFwiFmPHy2KRAA/t+VPgFQ0X2bPBFTXjhw4g== 2024-04-19T01:03:34.297 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.2] 2024-04-19T01:03:34.297 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQDJwiFmk2UbMBAAt8FZR0RV9aXP4Gkp+sY+wA== 2024-04-19T01:03:34.297 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.6] 2024-04-19T01:03:34.297 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQDMwiFm5mMyNRAA60llWfe+shWEJbp1DC1UEA== 2024-04-19T01:03:34.298 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.10] 2024-04-19T01:03:34.298 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQDPwiFmu54uOxAATlJ7qXFn2shVm1fedi0E4g== 2024-04-19T01:03:34.298 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.14] 2024-04-19T01:03:34.298 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQDTwiFm1BmIBRAAZR0Kl6entjPn1lGZ5R8QbA== 2024-04-19T01:03:34.298 INFO:teuthology.orchestra.run.smithi033.stdout:[mgr.x] 2024-04-19T01:03:34.298 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQCrwiFmmh9jMhAAwAxdytlF+0YtOWFPXPxOmQ== 2024-04-19T01:03:34.298 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.3] 2024-04-19T01:03:34.298 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQDZwiFmBhNJAhAAwhTuGEPm4lpum9z9XpXnIA== 2024-04-19T01:03:34.298 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.7] 2024-04-19T01:03:34.298 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQDcwiFmvy0nBxAAPZpcF85cSmRw7/MxFf88Xw== 2024-04-19T01:03:34.298 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.11] 2024-04-19T01:03:34.298 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQDfwiFmUFWmDhAAUPu8nxNigGepuqT/d7F00g== 2024-04-19T01:03:34.298 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.15] 2024-04-19T01:03:34.298 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQDiwiFmV8d3FhAANTWoATlxtyZj+FBzdYzObw== 2024-04-19T01:03:34.298 INFO:teuthology.orchestra.run.smithi033.stdout:[client.0] 2024-04-19T01:03:34.299 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQCrwiFmQKO9NxAAI8SOyJGrfdONv0Kv2hh44Q== 2024-04-19T01:03:34.300 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr.y] 2024-04-19T01:03:34.300 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCrwiFmpvXILxAAGeFawyr5fMeNBoYHSMFYpQ== 2024-04-19T01:03:34.300 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.0] 2024-04-19T01:03:34.300 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCuwiFmevuTNxAAAc7tRIU0nLP/sdvcJvgLQQ== 2024-04-19T01:03:34.300 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.4] 2024-04-19T01:03:34.300 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCywiFmXo6yABAAtSH1hB/XEOu9x1C4vj3how== 2024-04-19T01:03:34.300 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.8] 2024-04-19T01:03:34.300 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQC1wiFm398gBxAAvG7XQNTfYV/5cJFV8FSE1A== 2024-04-19T01:03:34.301 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.12] 2024-04-19T01:03:34.301 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQC4wiFmpTg9DxAAfUZRdhbo5UBLa2ZWfye2Qw== 2024-04-19T01:03:34.301 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.1] 2024-04-19T01:03:34.301 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQC8wiFmFhK4FxAAFfT0bZlo79nY99PPTz5/gw== 2024-04-19T01:03:34.301 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.5] 2024-04-19T01:03:34.301 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQC/wiFmWDPBHBAAIhHaoDKmizrJSLue9adzgQ== 2024-04-19T01:03:34.301 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.9] 2024-04-19T01:03:34.301 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDCwiFmiq5XIxAA9OfcY78r1kA0QiEsXWvtSg== 2024-04-19T01:03:34.301 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.13] 2024-04-19T01:03:34.301 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDFwiFmPHy2KRAA/t+VPgFQ0X2bPBFTXjhw4g== 2024-04-19T01:03:34.301 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.2] 2024-04-19T01:03:34.301 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDJwiFmk2UbMBAAt8FZR0RV9aXP4Gkp+sY+wA== 2024-04-19T01:03:34.301 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.6] 2024-04-19T01:03:34.301 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDMwiFm5mMyNRAA60llWfe+shWEJbp1DC1UEA== 2024-04-19T01:03:34.302 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.10] 2024-04-19T01:03:34.302 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDPwiFmu54uOxAATlJ7qXFn2shVm1fedi0E4g== 2024-04-19T01:03:34.302 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.14] 2024-04-19T01:03:34.302 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDTwiFm1BmIBRAAZR0Kl6entjPn1lGZ5R8QbA== 2024-04-19T01:03:34.302 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr.x] 2024-04-19T01:03:34.302 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCrwiFmmh9jMhAAwAxdytlF+0YtOWFPXPxOmQ== 2024-04-19T01:03:34.302 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.3] 2024-04-19T01:03:34.302 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDZwiFmBhNJAhAAwhTuGEPm4lpum9z9XpXnIA== 2024-04-19T01:03:34.302 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.7] 2024-04-19T01:03:34.302 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDcwiFmvy0nBxAAPZpcF85cSmRw7/MxFf88Xw== 2024-04-19T01:03:34.302 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.11] 2024-04-19T01:03:34.302 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDfwiFmUFWmDhAAUPu8nxNigGepuqT/d7F00g== 2024-04-19T01:03:34.302 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.15] 2024-04-19T01:03:34.302 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDiwiFmV8d3FhAANTWoATlxtyZj+FBzdYzObw== 2024-04-19T01:03:34.302 INFO:teuthology.orchestra.run.smithi110.stdout:[client.0] 2024-04-19T01:03:34.303 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCrwiFmQKO9NxAAI8SOyJGrfdONv0Kv2hh44Q== 2024-04-19T01:03:34.303 INFO:teuthology.orchestra.run.smithi118.stdout:[mgr.y] 2024-04-19T01:03:34.304 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQCrwiFmpvXILxAAGeFawyr5fMeNBoYHSMFYpQ== 2024-04-19T01:03:34.304 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.0] 2024-04-19T01:03:34.304 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQCuwiFmevuTNxAAAc7tRIU0nLP/sdvcJvgLQQ== 2024-04-19T01:03:34.304 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.4] 2024-04-19T01:03:34.304 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQCywiFmXo6yABAAtSH1hB/XEOu9x1C4vj3how== 2024-04-19T01:03:34.304 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.8] 2024-04-19T01:03:34.304 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQC1wiFm398gBxAAvG7XQNTfYV/5cJFV8FSE1A== 2024-04-19T01:03:34.304 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.12] 2024-04-19T01:03:34.304 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQC4wiFmpTg9DxAAfUZRdhbo5UBLa2ZWfye2Qw== 2024-04-19T01:03:34.305 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.1] 2024-04-19T01:03:34.305 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQC8wiFmFhK4FxAAFfT0bZlo79nY99PPTz5/gw== 2024-04-19T01:03:34.305 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.5] 2024-04-19T01:03:34.305 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQC/wiFmWDPBHBAAIhHaoDKmizrJSLue9adzgQ== 2024-04-19T01:03:34.305 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.9] 2024-04-19T01:03:34.305 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQDCwiFmiq5XIxAA9OfcY78r1kA0QiEsXWvtSg== 2024-04-19T01:03:34.305 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.13] 2024-04-19T01:03:34.305 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQDFwiFmPHy2KRAA/t+VPgFQ0X2bPBFTXjhw4g== 2024-04-19T01:03:34.305 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.2] 2024-04-19T01:03:34.305 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQDJwiFmk2UbMBAAt8FZR0RV9aXP4Gkp+sY+wA== 2024-04-19T01:03:34.305 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.6] 2024-04-19T01:03:34.305 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQDMwiFm5mMyNRAA60llWfe+shWEJbp1DC1UEA== 2024-04-19T01:03:34.305 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.10] 2024-04-19T01:03:34.305 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQDPwiFmu54uOxAATlJ7qXFn2shVm1fedi0E4g== 2024-04-19T01:03:34.305 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.14] 2024-04-19T01:03:34.306 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQDTwiFm1BmIBRAAZR0Kl6entjPn1lGZ5R8QbA== 2024-04-19T01:03:34.306 INFO:teuthology.orchestra.run.smithi118.stdout:[mgr.x] 2024-04-19T01:03:34.306 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQCrwiFmmh9jMhAAwAxdytlF+0YtOWFPXPxOmQ== 2024-04-19T01:03:34.306 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.3] 2024-04-19T01:03:34.306 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQDZwiFmBhNJAhAAwhTuGEPm4lpum9z9XpXnIA== 2024-04-19T01:03:34.306 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.7] 2024-04-19T01:03:34.306 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQDcwiFmvy0nBxAAPZpcF85cSmRw7/MxFf88Xw== 2024-04-19T01:03:34.306 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.11] 2024-04-19T01:03:34.306 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQDfwiFmUFWmDhAAUPu8nxNigGepuqT/d7F00g== 2024-04-19T01:03:34.306 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.15] 2024-04-19T01:03:34.306 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQDiwiFmV8d3FhAANTWoATlxtyZj+FBzdYzObw== 2024-04-19T01:03:34.306 INFO:teuthology.orchestra.run.smithi118.stdout:[client.0] 2024-04-19T01:03:34.306 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQCrwiFmQKO9NxAAI8SOyJGrfdONv0Kv2hh44Q== 2024-04-19T01:03:34.307 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-19T01:03:34.345 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-19T01:03:34.347 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-19T01:03:34.396 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:34.432 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:34.434 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:34.477 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:34.520 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:34.523 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:34.566 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:34.604 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:34.610 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:34.654 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:34.688 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:34.698 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:34.743 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:34.772 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:34.782 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:34.826 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:34.856 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:34.866 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:34.909 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:34.940 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:34.946 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:34.996 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:35.024 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:35.026 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:35.082 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:35.112 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:35.114 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:35.165 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:35.196 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:35.198 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:35.246 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:35.280 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:35.282 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:35.327 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:35.364 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:35.366 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:35.408 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-19T01:03:35.448 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-19T01:03:35.451 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-19T01:03:35.496 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:35.532 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:35.534 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:35.578 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:35.616 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:35.618 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:35.667 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:35.704 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:35.708 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:35.753 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:35.792 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:35.794 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:03:35.841 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-19T01:03:35.880 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-19T01:03:35.882 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-19T01:03:35.928 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-19T01:03:35.928 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-19T01:03:35.945 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-19T01:03:36.038 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-19T01:03:36.095 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-19T01:03:36.109 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-19T01:03:36.209 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-19T01:03:36.220 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-19T01:03:36.235 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-19T01:03:36.331 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-19T01:03:36.341 DEBUG:teuthology.orchestra.run.smithi033:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-19T01:03:36.344 DEBUG:teuthology.orchestra.run.smithi110:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-19T01:03:36.346 DEBUG:teuthology.orchestra.run.smithi118:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-19T01:03:36.389 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-19T01:03:36.390 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-19T01:03:36.390 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-19T01:03:36.392 INFO:tasks.ceph.mon.a:Started 2024-04-19T01:03:36.393 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-19T01:03:36.393 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-19T01:03:36.395 INFO:tasks.ceph.mon.b:Started 2024-04-19T01:03:36.395 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-19T01:03:36.396 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-19T01:03:36.435 INFO:tasks.ceph.mon.c:Started 2024-04-19T01:03:36.435 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-19T01:03:36.435 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-19T01:03:36.436 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-04-19T01:03:36.438 INFO:tasks.ceph.mgr.y:Started 2024-04-19T01:03:36.438 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-19T01:03:36.439 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-19T01:03:36.441 INFO:tasks.ceph.mgr.x:Started 2024-04-19T01:03:36.441 INFO:tasks.ceph:Setting crush tunables to default 2024-04-19T01:03:36.441 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd crush tunables default 2024-04-19T01:03:36.905 INFO:teuthology.orchestra.run.smithi033.stderr:adjusted tunables profile to default 2024-04-19T01:03:36.925 INFO:tasks.ceph:check_enable_crimson: False 2024-04-19T01:03:36.925 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-19T01:03:36.926 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-19T01:03:36.926 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-19T01:03:36.939 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-19T01:03:36.939 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-19T01:03:36.995 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-19T01:03:36.995 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-8/fsid of=/dev/stdout 2024-04-19T01:03:37.043 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-04-19T01:03:37.046+0000 7f04fafb2180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-19T01:03:37.049 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-04-19T01:03:37.046+0000 7f6250660180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-19T01:03:37.051 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-19T01:03:37.051 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-12/fsid of=/dev/stdout 2024-04-19T01:03:37.106 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-19T01:03:37.106 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-19T01:03:37.121 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-19T01:03:37.121 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-19T01:03:37.176 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-19T01:03:37.176 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-9/fsid of=/dev/stdout 2024-04-19T01:03:37.236 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-19T01:03:37.236 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-13/fsid of=/dev/stdout 2024-04-19T01:03:37.297 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-04-19T01:03:37.297 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-19T01:03:37.313 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-04-19T01:03:37.313 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-19T01:03:37.370 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-04-19T01:03:37.371 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-10/fsid of=/dev/stdout 2024-04-19T01:03:37.425 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-04-19T01:03:37.425 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-14/fsid of=/dev/stdout 2024-04-19T01:03:37.480 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-19T01:03:37.481 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-19T01:03:37.494 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-19T01:03:37.494 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-19T01:03:37.546 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-19T01:03:37.546 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-11/fsid of=/dev/stdout 2024-04-19T01:03:37.569 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-04-19T01:03:37.566+0000 7f6250660180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-19T01:03:37.580 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-04-19T01:03:37.582+0000 7f04fafb2180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-19T01:03:37.581 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-19T01:03:37.581 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-15/fsid of=/dev/stdout 2024-04-19T01:03:37.635 DEBUG:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd new 296899fd-2b2b-4071-8b1f-0f92c3d0bbc0 0 2024-04-19T01:03:37.778 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-04-19T01:03:37.774+0000 7f6250660180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-19T01:03:37.787 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-04-19T01:03:37.790+0000 7f04fafb2180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-19T01:03:37.894 INFO:teuthology.orchestra.run.smithi155.stdout:0 2024-04-19T01:03:37.912 DEBUG:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd new 021f7190-0ec3-4553-a44f-15095765cccb 1 2024-04-19T01:03:38.116 INFO:teuthology.orchestra.run.smithi155.stdout:1 2024-04-19T01:03:38.133 DEBUG:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd new c283d0bc-5e34-4763-9019-328ee9b6c07a 2 2024-04-19T01:03:38.328 INFO:teuthology.orchestra.run.smithi155.stdout:2 2024-04-19T01:03:38.345 DEBUG:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd new d45888f2-eecd-480b-80f5-c0731a06ff3d 3 2024-04-19T01:03:38.355 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-04-19T01:03:38.354+0000 7f6250660180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-19T01:03:38.382 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-04-19T01:03:38.382+0000 7f04fafb2180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-19T01:03:38.411 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-04-19T01:03:38.410+0000 7f6250660180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-19T01:03:38.465 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-04-19T01:03:38.462+0000 7f6250660180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-19T01:03:38.544 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-04-19T01:03:38.542+0000 7f6250660180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-19T01:03:38.547 INFO:teuthology.orchestra.run.smithi155.stdout:3 2024-04-19T01:03:38.558 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-04-19T01:03:38.558+0000 7f04fafb2180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-19T01:03:38.564 DEBUG:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd new d060d939-8c94-4a07-9fd5-42af0044a16f 4 2024-04-19T01:03:38.597 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-04-19T01:03:38.594+0000 7f6250660180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-19T01:03:38.614 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-04-19T01:03:38.614+0000 7f04fafb2180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-19T01:03:38.676 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-04-19T01:03:38.674+0000 7f6250660180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-19T01:03:38.694 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-04-19T01:03:38.694+0000 7f04fafb2180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-19T01:03:38.734 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-04-19T01:03:38.730+0000 7f6250660180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-19T01:03:38.748 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-04-19T01:03:38.750+0000 7f04fafb2180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-19T01:03:38.803 INFO:teuthology.orchestra.run.smithi155.stdout:4 2024-04-19T01:03:38.817 DEBUG:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd new 9700d2f7-ed56-4143-ba9d-6cebe524e877 5 2024-04-19T01:03:38.829 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-04-19T01:03:38.830+0000 7f04fafb2180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-19T01:03:38.884 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-04-19T01:03:38.886+0000 7f04fafb2180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-19T01:03:38.886 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-04-19T01:03:38.882+0000 7f6250660180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-19T01:03:38.950 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-04-19T01:03:38.946+0000 7f6250660180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-19T01:03:39.013 INFO:teuthology.orchestra.run.smithi155.stdout:5 2024-04-19T01:03:39.029 DEBUG:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd new 49324695-d83b-41ad-92e1-5348e8e9149f 6 2024-04-19T01:03:39.041 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-04-19T01:03:39.042+0000 7f04fafb2180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-19T01:03:39.105 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-04-19T01:03:39.106+0000 7f04fafb2180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-19T01:03:39.221 INFO:teuthology.orchestra.run.smithi155.stdout:6 2024-04-19T01:03:39.238 DEBUG:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd new afef160d-0911-4406-ad49-cd6d61ee8a2f 7 2024-04-19T01:03:39.468 INFO:teuthology.orchestra.run.smithi155.stdout:7 2024-04-19T01:03:39.486 DEBUG:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd new 298396bf-31a2-4832-91a4-55fcb050130f 8 2024-04-19T01:03:39.538 INFO:tasks.ceph.mgr.x.smithi155.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-19T01:03:39.539 INFO:tasks.ceph.mgr.x.smithi155.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-19T01:03:39.539 INFO:tasks.ceph.mgr.x.smithi155.stderr: from numpy import show_config as show_numpy_config 2024-04-19T01:03:39.609 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-04-19T01:03:39.606+0000 7f6250660180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-19T01:03:39.686 INFO:teuthology.orchestra.run.smithi155.stdout:8 2024-04-19T01:03:39.702 DEBUG:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd new c151ae6b-279d-4174-aceb-4671fbdac3b3 9 2024-04-19T01:03:39.739 INFO:tasks.ceph.mgr.y.smithi033.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-19T01:03:39.739 INFO:tasks.ceph.mgr.y.smithi033.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-19T01:03:39.739 INFO:tasks.ceph.mgr.y.smithi033.stderr: from numpy import show_config as show_numpy_config 2024-04-19T01:03:39.775 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-04-19T01:03:39.774+0000 7f6250660180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-19T01:03:39.811 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-04-19T01:03:39.814+0000 7f04fafb2180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-19T01:03:39.904 INFO:teuthology.orchestra.run.smithi155.stdout:9 2024-04-19T01:03:39.922 DEBUG:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd new 0b292dfb-2029-4e1e-b206-ea7e0f6969ab 10 2024-04-19T01:03:39.926 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-04-19T01:03:39.922+0000 7f6250660180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-19T01:03:39.991 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-04-19T01:03:39.990+0000 7f6250660180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-19T01:03:40.002 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-04-19T01:03:40.002+0000 7f04fafb2180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-19T01:03:40.115 INFO:teuthology.orchestra.run.smithi155.stdout:10 2024-04-19T01:03:40.129 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-04-19T01:03:40.126+0000 7f6250660180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-19T01:03:40.130 DEBUG:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd new 10c61619-ad11-40f0-b013-9ef52bcabf59 11 2024-04-19T01:03:40.180 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-04-19T01:03:40.182+0000 7f04fafb2180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-19T01:03:40.251 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-04-19T01:03:40.254+0000 7f04fafb2180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-19T01:03:40.377 INFO:teuthology.orchestra.run.smithi155.stdout:11 2024-04-19T01:03:40.394 DEBUG:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd new 3f0d26e5-f3e2-47b6-91a8-0daa30d0544f 12 2024-04-19T01:03:40.398 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-04-19T01:03:40.398+0000 7f04fafb2180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-19T01:03:40.589 INFO:teuthology.orchestra.run.smithi155.stdout:12 2024-04-19T01:03:40.605 DEBUG:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd new c8d6213f-af5e-4646-9241-6f91589b0e1b 13 2024-04-19T01:03:40.771 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-04-19T01:03:40.770+0000 7f6250660180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-19T01:03:40.801 INFO:teuthology.orchestra.run.smithi155.stdout:13 2024-04-19T01:03:40.818 DEBUG:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd new 7b3d56df-62bf-4877-b984-6f17fe1fab33 14 2024-04-19T01:03:40.849 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-04-19T01:03:40.846+0000 7f6250660180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-19T01:03:41.017 INFO:teuthology.orchestra.run.smithi155.stdout:14 2024-04-19T01:03:41.024 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-04-19T01:03:41.026+0000 7f04fafb2180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-19T01:03:41.034 DEBUG:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd new 6d526edb-f1d8-415a-ad55-81e810ff4944 15 2024-04-19T01:03:41.036 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-04-19T01:03:41.034+0000 7f6250660180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-19T01:03:41.103 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-04-19T01:03:41.106+0000 7f04fafb2180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-19T01:03:41.112 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-04-19T01:03:41.110+0000 7f6250660180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-19T01:03:41.226 INFO:teuthology.orchestra.run.smithi155.stdout:15 2024-04-19T01:03:41.241 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-19T01:03:41.242 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-19T01:03:41.245 INFO:tasks.ceph.osd.0:Started 2024-04-19T01:03:41.245 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-19T01:03:41.246 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-19T01:03:41.248 INFO:tasks.ceph.osd.4:Started 2024-04-19T01:03:41.249 INFO:tasks.ceph.osd.8:Restarting daemon 2024-04-19T01:03:41.249 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 8 2024-04-19T01:03:41.251 INFO:tasks.ceph.osd.8:Started 2024-04-19T01:03:41.251 INFO:tasks.ceph.osd.12:Restarting daemon 2024-04-19T01:03:41.251 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 12 2024-04-19T01:03:41.255 INFO:tasks.ceph.osd.12:Started 2024-04-19T01:03:41.255 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-19T01:03:41.255 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-19T01:03:41.258 INFO:tasks.ceph.osd.1:Started 2024-04-19T01:03:41.258 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-19T01:03:41.258 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-19T01:03:41.261 INFO:tasks.ceph.osd.5:Started 2024-04-19T01:03:41.261 INFO:tasks.ceph.osd.9:Restarting daemon 2024-04-19T01:03:41.261 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 9 2024-04-19T01:03:41.263 INFO:tasks.ceph.osd.9:Started 2024-04-19T01:03:41.263 INFO:tasks.ceph.osd.13:Restarting daemon 2024-04-19T01:03:41.263 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 13 2024-04-19T01:03:41.266 INFO:tasks.ceph.osd.13:Started 2024-04-19T01:03:41.266 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-19T01:03:41.266 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-19T01:03:41.270 INFO:tasks.ceph.osd.2:Started 2024-04-19T01:03:41.271 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-19T01:03:41.271 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-04-19T01:03:41.273 INFO:tasks.ceph.osd.6:Started 2024-04-19T01:03:41.274 INFO:tasks.ceph.osd.10:Restarting daemon 2024-04-19T01:03:41.274 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 10 2024-04-19T01:03:41.276 INFO:tasks.ceph.osd.10:Started 2024-04-19T01:03:41.276 INFO:tasks.ceph.osd.14:Restarting daemon 2024-04-19T01:03:41.277 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 14 2024-04-19T01:03:41.279 INFO:tasks.ceph.osd.14:Started 2024-04-19T01:03:41.279 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-19T01:03:41.279 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-19T01:03:41.281 INFO:tasks.ceph.osd.3:Started 2024-04-19T01:03:41.281 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-19T01:03:41.281 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-04-19T01:03:41.283 INFO:tasks.ceph.osd.7:Started 2024-04-19T01:03:41.283 INFO:tasks.ceph.osd.11:Restarting daemon 2024-04-19T01:03:41.283 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 11 2024-04-19T01:03:41.285 INFO:tasks.ceph.osd.11:Started 2024-04-19T01:03:41.286 INFO:tasks.ceph.osd.15:Restarting daemon 2024-04-19T01:03:41.286 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 15 2024-04-19T01:03:41.289 INFO:tasks.ceph.osd.15:Started 2024-04-19T01:03:41.290 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-19T01:03:41.300 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-04-19T01:03:41.302+0000 7f04fafb2180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-19T01:03:41.384 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-04-19T01:03:41.386+0000 7f04fafb2180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-19T01:03:41.423 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-04-19T01:03:41.422+0000 7f6250660180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-19T01:03:41.479 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:03:41.479 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":18,"fsid":"f11bd2d5-4456-4ffa-83c0-722dddd22ad3","created":"2024-04-19T01:03:36.766089+0000","modified":"2024-04-19T01:03:41.227644+0000","last_up_change":"0.000000","last_in_change":"2024-04-19T01:03:41.227644+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"296899fd-2b2b-4071-8b1f-0f92c3d0bbc0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"021f7190-0ec3-4553-a44f-15095765cccb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c283d0bc-5e34-4763-9019-328ee9b6c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d45888f2-eecd-480b-80f5-c0731a06ff3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d060d939-8c94-4a07-9fd5-42af0044a16f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9700d2f7-ed56-4143-ba9d-6cebe524e877","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"49324695-d83b-41ad-92e1-5348e8e9149f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"afef160d-0911-4406-ad49-cd6d61ee8a2f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"298396bf-31a2-4832-91a4-55fcb050130f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"c151ae6b-279d-4174-aceb-4671fbdac3b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"0b292dfb-2029-4e1e-b206-ea7e0f6969ab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"10c61619-ad11-40f0-b013-9ef52bcabf59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"3f0d26e5-f3e2-47b6-91a8-0daa30d0544f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"c8d6213f-af5e-4646-9241-6f91589b0e1b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"7b3d56df-62bf-4877-b984-6f17fe1fab33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"6d526edb-f1d8-415a-ad55-81e810ff4944","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T01:03:41.484 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-04-19T01:03:41.482+0000 7f6250660180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-19T01:03:41.495 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-19T01:03:41.495 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-19T01:03:41.691 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-04-19T01:03:41.694+0000 7f04fafb2180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-19T01:03:41.756 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-04-19T01:03:41.758+0000 7f04fafb2180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-19T01:03:41.868 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:03:41.870+0000 7f2d7377c7c0 -1 Falling back to public interface 2024-04-19T01:03:41.876 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:03:41.878+0000 7fbc9be6f7c0 -1 Falling back to public interface 2024-04-19T01:03:41.883 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:03:41.881+0000 7f23fd5047c0 -1 Falling back to public interface 2024-04-19T01:03:41.883 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:03:41.881+0000 7f4acf1d17c0 -1 Falling back to public interface 2024-04-19T01:03:41.884 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:03:41.881+0000 7fe03715e7c0 -1 Falling back to public interface 2024-04-19T01:03:41.884 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:03:41.886+0000 7f6ac05c37c0 -1 Falling back to public interface 2024-04-19T01:03:41.884 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:03:41.882+0000 7f283f4c07c0 -1 Falling back to public interface 2024-04-19T01:03:41.897 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:03:41.897+0000 7f2729ec07c0 -1 Falling back to public interface 2024-04-19T01:03:41.899 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:03:41.898+0000 7f1280b027c0 -1 Falling back to public interface 2024-04-19T01:03:41.899 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:03:41.902+0000 7f66a9bfc7c0 -1 Falling back to public interface 2024-04-19T01:03:41.904 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:03:41.902+0000 7fc1591497c0 -1 Falling back to public interface 2024-04-19T01:03:41.906 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:03:41.902+0000 7f4f7f6cd7c0 -1 Falling back to public interface 2024-04-19T01:03:41.906 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:03:41.902+0000 7f0563b187c0 -1 Falling back to public interface 2024-04-19T01:03:41.918 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:03:41.918+0000 7fe4c097d7c0 -1 Falling back to public interface 2024-04-19T01:03:41.918 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:03:41.918+0000 7f744d75c7c0 -1 Falling back to public interface 2024-04-19T01:03:42.097 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:03:42.094+0000 7f43c61627c0 -1 Falling back to public interface 2024-04-19T01:03:42.526 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-04-19T01:03:42.522+0000 7f61f7aa9640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-19T01:03:42.527 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-04-19T01:03:42.526+0000 7f61f7aa9640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-19T01:03:43.108 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-19T01:03:43.412 INFO:teuthology.misc.health.smithi033.stdout: 2024-04-19T01:03:43.412 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":18,"fsid":"f11bd2d5-4456-4ffa-83c0-722dddd22ad3","created":"2024-04-19T01:03:36.766089+0000","modified":"2024-04-19T01:03:41.227644+0000","last_up_change":"0.000000","last_in_change":"2024-04-19T01:03:41.227644+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"296899fd-2b2b-4071-8b1f-0f92c3d0bbc0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"021f7190-0ec3-4553-a44f-15095765cccb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c283d0bc-5e34-4763-9019-328ee9b6c07a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d45888f2-eecd-480b-80f5-c0731a06ff3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d060d939-8c94-4a07-9fd5-42af0044a16f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9700d2f7-ed56-4143-ba9d-6cebe524e877","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"49324695-d83b-41ad-92e1-5348e8e9149f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"afef160d-0911-4406-ad49-cd6d61ee8a2f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"298396bf-31a2-4832-91a4-55fcb050130f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"c151ae6b-279d-4174-aceb-4671fbdac3b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"0b292dfb-2029-4e1e-b206-ea7e0f6969ab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"10c61619-ad11-40f0-b013-9ef52bcabf59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"3f0d26e5-f3e2-47b6-91a8-0daa30d0544f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"c8d6213f-af5e-4646-9241-6f91589b0e1b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"7b3d56df-62bf-4877-b984-6f17fe1fab33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"6d526edb-f1d8-415a-ad55-81e810ff4944","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T01:03:43.429 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-04-19T01:03:45.497 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:03:45.494+0000 7f744d75c7c0 -1 osd.10 0 log_to_monitors true 2024-04-19T01:03:45.596 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:03:45.594+0000 7f283f4c07c0 -1 osd.14 0 log_to_monitors true 2024-04-19T01:03:45.606 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:03:45.606+0000 7f2d7377c7c0 -1 osd.4 0 log_to_monitors true 2024-04-19T01:03:45.638 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:03:45.634+0000 7fc1591497c0 -1 osd.3 0 log_to_monitors true 2024-04-19T01:03:45.664 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:03:45.661+0000 7fe03715e7c0 -1 osd.1 0 log_to_monitors true 2024-04-19T01:03:45.700 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:03:45.698+0000 7f1280b027c0 -1 osd.2 0 log_to_monitors true 2024-04-19T01:03:45.704 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:03:45.706+0000 7fbc9be6f7c0 -1 osd.12 0 log_to_monitors true 2024-04-19T01:03:45.730 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:03:45.726+0000 7f0563b187c0 -1 osd.15 0 log_to_monitors true 2024-04-19T01:03:45.736 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:03:45.734+0000 7fe4c097d7c0 -1 osd.6 0 log_to_monitors true 2024-04-19T01:03:45.743 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:03:45.746+0000 7f6ac05c37c0 -1 osd.8 0 log_to_monitors true 2024-04-19T01:03:45.811 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:03:45.809+0000 7f4acf1d17c0 -1 osd.13 0 log_to_monitors true 2024-04-19T01:03:45.815 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:03:45.813+0000 7f2729ec07c0 -1 osd.5 0 log_to_monitors true 2024-04-19T01:03:45.816 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:03:45.818+0000 7f66a9bfc7c0 -1 osd.0 0 log_to_monitors true 2024-04-19T01:03:45.823 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:03:45.821+0000 7f23fd5047c0 -1 osd.9 0 log_to_monitors true 2024-04-19T01:03:45.855 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:03:45.850+0000 7f4f7f6cd7c0 -1 osd.11 0 log_to_monitors true 2024-04-19T01:03:45.869 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:03:45.866+0000 7f43c61627c0 -1 osd.7 0 log_to_monitors true 2024-04-19T01:03:46.545 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:03:46.542+0000 7f7449705640 -1 osd.10 0 waiting for initial osdmap 2024-04-19T01:03:46.567 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:03:46.566+0000 7f7445541640 -1 osd.10 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T01:03:47.551 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:03:47.549+0000 7f2725e69640 -1 osd.5 0 waiting for initial osdmap 2024-04-19T01:03:47.552 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:03:47.553+0000 7fe033107640 -1 osd.1 0 waiting for initial osdmap 2024-04-19T01:03:47.553 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:03:47.550+0000 7f4f7b676640 -1 osd.11 0 waiting for initial osdmap 2024-04-19T01:03:47.553 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:03:47.549+0000 7fe4bc926640 -1 osd.6 0 waiting for initial osdmap 2024-04-19T01:03:47.554 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:03:47.550+0000 7f055fac1640 -1 osd.15 0 waiting for initial osdmap 2024-04-19T01:03:47.554 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:03:47.550+0000 7fc1550f2640 -1 osd.3 0 waiting for initial osdmap 2024-04-19T01:03:47.566 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:03:47.566+0000 7fbc97e18640 -1 osd.12 0 waiting for initial osdmap 2024-04-19T01:03:47.567 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:03:47.566+0000 7f2d6f725640 -1 osd.4 0 waiting for initial osdmap 2024-04-19T01:03:47.567 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:03:47.566+0000 7f6abc56c640 -1 osd.8 0 waiting for initial osdmap 2024-04-19T01:03:47.574 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:03:47.573+0000 7fe02ef43640 -1 osd.1 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T01:03:47.576 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:03:47.578+0000 7f6ab83a8640 -1 osd.8 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T01:03:47.577 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:03:47.578+0000 7f2d6b561640 -1 osd.4 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T01:03:47.579 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:03:47.577+0000 7fe4b8762640 -1 osd.6 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T01:03:47.580 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:03:47.582+0000 7fbc93c54640 -1 osd.12 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T01:03:47.581 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:03:47.578+0000 7f4f774b2640 -1 osd.11 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T01:03:47.581 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:03:47.578+0000 7f055b8fd640 -1 osd.15 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T01:03:47.581 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:03:47.581+0000 7f2721ca5640 -1 osd.5 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T01:03:47.585 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:03:47.582+0000 7fc150f2e640 -1 osd.3 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T01:03:50.184 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:03:50.182+0000 7f43c210b640 -1 osd.7 0 waiting for initial osdmap 2024-04-19T01:03:50.209 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:03:50.206+0000 7f43b9f3f640 -1 osd.7 23 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T01:03:50.303 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:03:50.305+0000 7f66a63b7640 -1 osd.0 0 waiting for initial osdmap 2024-04-19T01:03:50.324 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:03:50.325+0000 7f669d9d9640 -1 osd.0 23 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T01:03:50.341 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:03:50.340+0000 7f4acb98c640 -1 osd.13 0 waiting for initial osdmap 2024-04-19T01:03:50.344 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:03:50.341+0000 7f127d2bd640 -1 osd.2 0 waiting for initial osdmap 2024-04-19T01:03:50.361 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:03:50.364+0000 7f4ac2fae640 -1 osd.13 23 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T01:03:50.364 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:03:50.361+0000 7f283bc7b640 -1 osd.14 0 waiting for initial osdmap 2024-04-19T01:03:50.374 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:03:50.373+0000 7f12748df640 -1 osd.2 23 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T01:03:50.381 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:03:50.384+0000 7f23f9cbf640 -1 osd.9 0 waiting for initial osdmap 2024-04-19T01:03:50.382 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:03:50.381+0000 7f283329d640 -1 osd.14 23 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T01:03:50.409 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:03:50.412+0000 7f23f12e1640 -1 osd.9 23 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T01:03:51.041 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-19T01:03:51.372 INFO:teuthology.misc.health.smithi033.stderr:2024-04-19T01:03:51.373+0000 7feca6ffd640 0 --2- 172.21.15.33:0/1469890206 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec90003c90 0x7fec90004090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-04-19T01:03:51.582 INFO:teuthology.misc.health.smithi033.stdout: 2024-04-19T01:03:51.582 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":24,"fsid":"f11bd2d5-4456-4ffa-83c0-722dddd22ad3","created":"2024-04-19T01:03:36.766089+0000","modified":"2024-04-19T01:03:50.591452+0000","last_up_change":"2024-04-19T01:03:50.591452+0000","last_in_change":"2024-04-19T01:03:41.227644+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-19T01:03:49.571609+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"296899fd-2b2b-4071-8b1f-0f92c3d0bbc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":1633616806},{"type":"v1","addr":"172.21.15.33:6825","nonce":1633616806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":1633616806},{"type":"v1","addr":"172.21.15.33:6827","nonce":1633616806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":1633616806},{"type":"v1","addr":"172.21.15.33:6831","nonce":1633616806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":1633616806},{"type":"v1","addr":"172.21.15.33:6829","nonce":1633616806}]},"public_addr":"172.21.15.33:6825/1633616806","cluster_addr":"172.21.15.33:6827/1633616806","heartbeat_back_addr":"172.21.15.33:6831/1633616806","heartbeat_front_addr":"172.21.15.33:6829/1633616806","state":["exists","up"]},{"osd":1,"uuid":"021f7190-0ec3-4553-a44f-15095765cccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":4270148213},{"type":"v1","addr":"172.21.15.110:6808","nonce":4270148213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6811","nonce":4270148213},{"type":"v1","addr":"172.21.15.110:6814","nonce":4270148213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":4270148213},{"type":"v1","addr":"172.21.15.110:6823","nonce":4270148213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6817","nonce":4270148213},{"type":"v1","addr":"172.21.15.110:6820","nonce":4270148213}]},"public_addr":"172.21.15.110:6808/4270148213","cluster_addr":"172.21.15.110:6814/4270148213","heartbeat_back_addr":"172.21.15.110:6823/4270148213","heartbeat_front_addr":"172.21.15.110:6820/4270148213","state":["exists","up"]},{"osd":2,"uuid":"c283d0bc-5e34-4763-9019-328ee9b6c07a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":13248188},{"type":"v1","addr":"172.21.15.118:6809","nonce":13248188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":13248188},{"type":"v1","addr":"172.21.15.118:6811","nonce":13248188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":13248188},{"type":"v1","addr":"172.21.15.118:6815","nonce":13248188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":13248188},{"type":"v1","addr":"172.21.15.118:6813","nonce":13248188}]},"public_addr":"172.21.15.118:6809/13248188","cluster_addr":"172.21.15.118:6811/13248188","heartbeat_back_addr":"172.21.15.118:6815/13248188","heartbeat_front_addr":"172.21.15.118:6813/13248188","state":["exists","up"]},{"osd":3,"uuid":"d45888f2-eecd-480b-80f5-c0731a06ff3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":2658385904},{"type":"v1","addr":"172.21.15.155:6803","nonce":2658385904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":2658385904},{"type":"v1","addr":"172.21.15.155:6805","nonce":2658385904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6813","nonce":2658385904},{"type":"v1","addr":"172.21.15.155:6816","nonce":2658385904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":2658385904},{"type":"v1","addr":"172.21.15.155:6809","nonce":2658385904}]},"public_addr":"172.21.15.155:6803/2658385904","cluster_addr":"172.21.15.155:6805/2658385904","heartbeat_back_addr":"172.21.15.155:6816/2658385904","heartbeat_front_addr":"172.21.15.155:6809/2658385904","state":["exists","up"]},{"osd":4,"uuid":"d060d939-8c94-4a07-9fd5-42af0044a16f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":2389566155},{"type":"v1","addr":"172.21.15.33:6801","nonce":2389566155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":2389566155},{"type":"v1","addr":"172.21.15.33:6803","nonce":2389566155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":2389566155},{"type":"v1","addr":"172.21.15.33:6807","nonce":2389566155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":2389566155},{"type":"v1","addr":"172.21.15.33:6805","nonce":2389566155}]},"public_addr":"172.21.15.33:6801/2389566155","cluster_addr":"172.21.15.33:6803/2389566155","heartbeat_back_addr":"172.21.15.33:6807/2389566155","heartbeat_front_addr":"172.21.15.33:6805/2389566155","state":["exists","up"]},{"osd":5,"uuid":"9700d2f7-ed56-4143-ba9d-6cebe524e877","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":2821892844},{"type":"v1","addr":"172.21.15.110:6825","nonce":2821892844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":2821892844},{"type":"v1","addr":"172.21.15.110:6827","nonce":2821892844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":2821892844},{"type":"v1","addr":"172.21.15.110:6831","nonce":2821892844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":2821892844},{"type":"v1","addr":"172.21.15.110:6829","nonce":2821892844}]},"public_addr":"172.21.15.110:6825/2821892844","cluster_addr":"172.21.15.110:6827/2821892844","heartbeat_back_addr":"172.21.15.110:6831/2821892844","heartbeat_front_addr":"172.21.15.110:6829/2821892844","state":["exists","up"]},{"osd":6,"uuid":"49324695-d83b-41ad-92e1-5348e8e9149f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":273776428},{"type":"v1","addr":"172.21.15.118:6818","nonce":273776428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":273776428},{"type":"v1","addr":"172.21.15.118:6822","nonce":273776428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":273776428},{"type":"v1","addr":"172.21.15.118:6830","nonce":273776428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":273776428},{"type":"v1","addr":"172.21.15.118:6826","nonce":273776428}]},"public_addr":"172.21.15.118:6818/273776428","cluster_addr":"172.21.15.118:6822/273776428","heartbeat_back_addr":"172.21.15.118:6830/273776428","heartbeat_front_addr":"172.21.15.118:6826/273776428","state":["exists","up"]},{"osd":7,"uuid":"afef160d-0911-4406-ad49-cd6d61ee8a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1006872173},{"type":"v1","addr":"172.21.15.155:6827","nonce":1006872173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1006872173},{"type":"v1","addr":"172.21.15.155:6829","nonce":1006872173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1006872173},{"type":"v1","addr":"172.21.15.155:6833","nonce":1006872173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1006872173},{"type":"v1","addr":"172.21.15.155:6831","nonce":1006872173}]},"public_addr":"172.21.15.155:6827/1006872173","cluster_addr":"172.21.15.155:6829/1006872173","heartbeat_back_addr":"172.21.15.155:6833/1006872173","heartbeat_front_addr":"172.21.15.155:6831/1006872173","state":["exists","up"]},{"osd":8,"uuid":"298396bf-31a2-4832-91a4-55fcb050130f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":744651323},{"type":"v1","addr":"172.21.15.33:6817","nonce":744651323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":744651323},{"type":"v1","addr":"172.21.15.33:6819","nonce":744651323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":744651323},{"type":"v1","addr":"172.21.15.33:6823","nonce":744651323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":744651323},{"type":"v1","addr":"172.21.15.33:6821","nonce":744651323}]},"public_addr":"172.21.15.33:6817/744651323","cluster_addr":"172.21.15.33:6819/744651323","heartbeat_back_addr":"172.21.15.33:6823/744651323","heartbeat_front_addr":"172.21.15.33:6821/744651323","state":["exists","up"]},{"osd":9,"uuid":"c151ae6b-279d-4174-aceb-4671fbdac3b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":872519654},{"type":"v1","addr":"172.21.15.110:6802","nonce":872519654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":872519654},{"type":"v1","addr":"172.21.15.110:6807","nonce":872519654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":872519654},{"type":"v1","addr":"172.21.15.110:6819","nonce":872519654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":872519654},{"type":"v1","addr":"172.21.15.110:6813","nonce":872519654}]},"public_addr":"172.21.15.110:6802/872519654","cluster_addr":"172.21.15.110:6807/872519654","heartbeat_back_addr":"172.21.15.110:6819/872519654","heartbeat_front_addr":"172.21.15.110:6813/872519654","state":["exists","up"]},{"osd":10,"uuid":"0b292dfb-2029-4e1e-b206-ea7e0f6969ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6817","nonce":4099141917},{"type":"v1","addr":"172.21.15.118:6819","nonce":4099141917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6821","nonce":4099141917},{"type":"v1","addr":"172.21.15.118:6823","nonce":4099141917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6829","nonce":4099141917},{"type":"v1","addr":"172.21.15.118:6831","nonce":4099141917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":4099141917},{"type":"v1","addr":"172.21.15.118:6827","nonce":4099141917}]},"public_addr":"172.21.15.118:6819/4099141917","cluster_addr":"172.21.15.118:6823/4099141917","heartbeat_back_addr":"172.21.15.118:6831/4099141917","heartbeat_front_addr":"172.21.15.118:6827/4099141917","state":["exists","up"]},{"osd":11,"uuid":"10c61619-ad11-40f0-b013-9ef52bcabf59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":3981951559},{"type":"v1","addr":"172.21.15.155:6811","nonce":3981951559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":3981951559},{"type":"v1","addr":"172.21.15.155:6817","nonce":3981951559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":3981951559},{"type":"v1","addr":"172.21.15.155:6825","nonce":3981951559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3981951559},{"type":"v1","addr":"172.21.15.155:6823","nonce":3981951559}]},"public_addr":"172.21.15.155:6811/3981951559","cluster_addr":"172.21.15.155:6817/3981951559","heartbeat_back_addr":"172.21.15.155:6825/3981951559","heartbeat_front_addr":"172.21.15.155:6823/3981951559","state":["exists","up"]},{"osd":12,"uuid":"3f0d26e5-f3e2-47b6-91a8-0daa30d0544f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":4133738119},{"type":"v1","addr":"172.21.15.33:6809","nonce":4133738119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":4133738119},{"type":"v1","addr":"172.21.15.33:6811","nonce":4133738119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":4133738119},{"type":"v1","addr":"172.21.15.33:6815","nonce":4133738119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":4133738119},{"type":"v1","addr":"172.21.15.33:6813","nonce":4133738119}]},"public_addr":"172.21.15.33:6809/4133738119","cluster_addr":"172.21.15.33:6811/4133738119","heartbeat_back_addr":"172.21.15.33:6815/4133738119","heartbeat_front_addr":"172.21.15.33:6813/4133738119","state":["exists","up"]},{"osd":13,"uuid":"c8d6213f-af5e-4646-9241-6f91589b0e1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":1583232449},{"type":"v1","addr":"172.21.15.110:6803","nonce":1583232449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":1583232449},{"type":"v1","addr":"172.21.15.110:6809","nonce":1583232449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":1583232449},{"type":"v1","addr":"172.21.15.110:6821","nonce":1583232449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":1583232449},{"type":"v1","addr":"172.21.15.110:6815","nonce":1583232449}]},"public_addr":"172.21.15.110:6803/1583232449","cluster_addr":"172.21.15.110:6809/1583232449","heartbeat_back_addr":"172.21.15.110:6821/1583232449","heartbeat_front_addr":"172.21.15.110:6815/1583232449","state":["exists","up"]},{"osd":14,"uuid":"7b3d56df-62bf-4877-b984-6f17fe1fab33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":506416463},{"type":"v1","addr":"172.21.15.118:6801","nonce":506416463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":506416463},{"type":"v1","addr":"172.21.15.118:6803","nonce":506416463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":506416463},{"type":"v1","addr":"172.21.15.118:6807","nonce":506416463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":506416463},{"type":"v1","addr":"172.21.15.118:6805","nonce":506416463}]},"public_addr":"172.21.15.118:6801/506416463","cluster_addr":"172.21.15.118:6803/506416463","heartbeat_back_addr":"172.21.15.118:6807/506416463","heartbeat_front_addr":"172.21.15.118:6805/506416463","state":["exists","up"]},{"osd":15,"uuid":"6d526edb-f1d8-415a-ad55-81e810ff4944","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6807","nonce":741132602},{"type":"v1","addr":"172.21.15.155:6810","nonce":741132602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":741132602},{"type":"v1","addr":"172.21.15.155:6815","nonce":741132602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6821","nonce":741132602},{"type":"v1","addr":"172.21.15.155:6822","nonce":741132602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":741132602},{"type":"v1","addr":"172.21.15.155:6819","nonce":741132602}]},"public_addr":"172.21.15.155:6810/741132602","cluster_addr":"172.21.15.155:6815/741132602","heartbeat_back_addr":"172.21.15.155:6822/741132602","heartbeat_front_addr":"172.21.15.155:6819/741132602","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.677618+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.659122+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.648122+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.859759+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.759643+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.779763+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.479489+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.853984+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.736100+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.712166+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T01:03:51.600 DEBUG:teuthology.misc:16 of 16 OSDs are up 2024-04-19T01:03:51.601 INFO:tasks.ceph:Creating RBD pool 2024-04-19T01:03:51.601 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-19T01:03:52.624 INFO:teuthology.orchestra.run.smithi033.stderr:pool 'rbd' created 2024-04-19T01:03:52.651 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-19T01:03:53.634 INFO:teuthology.orchestra.run.smithi033.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-19T01:03:53.660 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-19T01:03:53.660 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-19T01:03:53.660 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-19T01:03:53.661 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-19T01:03:53.661 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-19T01:03:54.013 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:03:54.032 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":5,"flags":0,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":1678763788},{"type":"v1","addr":"172.21.15.155:6801","nonce":1678763788}]},"active_addr":"172.21.15.155:6801/1678763788","active_change":"2024-04-19T01:03:41.509114+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4104,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.155:0","nonce":2585906020}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.155:0","nonce":1127060541}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.155:0","nonce":1473393247}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.155:0","nonce":622623710}]}]} 2024-04-19T01:03:54.036 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-19T01:03:54.036 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-19T01:03:54.036 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-19T01:03:54.341 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:03:54.341 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":27,"fsid":"f11bd2d5-4456-4ffa-83c0-722dddd22ad3","created":"2024-04-19T01:03:36.766089+0000","modified":"2024-04-19T01:03:53.634481+0000","last_up_change":"2024-04-19T01:03:50.591452+0000","last_in_change":"2024-04-19T01:03:41.227644+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-19T01:03:49.571609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-19T01:03:51.913782+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.0199999809265137,"score_stable":6.0199999809265137,"optimal_score":0.81000000238418579,"raw_score_acting":4.880000114440918,"raw_score_stable":4.880000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"296899fd-2b2b-4071-8b1f-0f92c3d0bbc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":1633616806},{"type":"v1","addr":"172.21.15.33:6825","nonce":1633616806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":1633616806},{"type":"v1","addr":"172.21.15.33:6827","nonce":1633616806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":1633616806},{"type":"v1","addr":"172.21.15.33:6831","nonce":1633616806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":1633616806},{"type":"v1","addr":"172.21.15.33:6829","nonce":1633616806}]},"public_addr":"172.21.15.33:6825/1633616806","cluster_addr":"172.21.15.33:6827/1633616806","heartbeat_back_addr":"172.21.15.33:6831/1633616806","heartbeat_front_addr":"172.21.15.33:6829/1633616806","state":["exists","up"]},{"osd":1,"uuid":"021f7190-0ec3-4553-a44f-15095765cccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":4270148213},{"type":"v1","addr":"172.21.15.110:6808","nonce":4270148213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6811","nonce":4270148213},{"type":"v1","addr":"172.21.15.110:6814","nonce":4270148213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":4270148213},{"type":"v1","addr":"172.21.15.110:6823","nonce":4270148213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6817","nonce":4270148213},{"type":"v1","addr":"172.21.15.110:6820","nonce":4270148213}]},"public_addr":"172.21.15.110:6808/4270148213","cluster_addr":"172.21.15.110:6814/4270148213","heartbeat_back_addr":"172.21.15.110:6823/4270148213","heartbeat_front_addr":"172.21.15.110:6820/4270148213","state":["exists","up"]},{"osd":2,"uuid":"c283d0bc-5e34-4763-9019-328ee9b6c07a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":13248188},{"type":"v1","addr":"172.21.15.118:6809","nonce":13248188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":13248188},{"type":"v1","addr":"172.21.15.118:6811","nonce":13248188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":13248188},{"type":"v1","addr":"172.21.15.118:6815","nonce":13248188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":13248188},{"type":"v1","addr":"172.21.15.118:6813","nonce":13248188}]},"public_addr":"172.21.15.118:6809/13248188","cluster_addr":"172.21.15.118:6811/13248188","heartbeat_back_addr":"172.21.15.118:6815/13248188","heartbeat_front_addr":"172.21.15.118:6813/13248188","state":["exists","up"]},{"osd":3,"uuid":"d45888f2-eecd-480b-80f5-c0731a06ff3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":2658385904},{"type":"v1","addr":"172.21.15.155:6803","nonce":2658385904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":2658385904},{"type":"v1","addr":"172.21.15.155:6805","nonce":2658385904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6813","nonce":2658385904},{"type":"v1","addr":"172.21.15.155:6816","nonce":2658385904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":2658385904},{"type":"v1","addr":"172.21.15.155:6809","nonce":2658385904}]},"public_addr":"172.21.15.155:6803/2658385904","cluster_addr":"172.21.15.155:6805/2658385904","heartbeat_back_addr":"172.21.15.155:6816/2658385904","heartbeat_front_addr":"172.21.15.155:6809/2658385904","state":["exists","up"]},{"osd":4,"uuid":"d060d939-8c94-4a07-9fd5-42af0044a16f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":2389566155},{"type":"v1","addr":"172.21.15.33:6801","nonce":2389566155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":2389566155},{"type":"v1","addr":"172.21.15.33:6803","nonce":2389566155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":2389566155},{"type":"v1","addr":"172.21.15.33:6807","nonce":2389566155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":2389566155},{"type":"v1","addr":"172.21.15.33:6805","nonce":2389566155}]},"public_addr":"172.21.15.33:6801/2389566155","cluster_addr":"172.21.15.33:6803/2389566155","heartbeat_back_addr":"172.21.15.33:6807/2389566155","heartbeat_front_addr":"172.21.15.33:6805/2389566155","state":["exists","up"]},{"osd":5,"uuid":"9700d2f7-ed56-4143-ba9d-6cebe524e877","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":2821892844},{"type":"v1","addr":"172.21.15.110:6825","nonce":2821892844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":2821892844},{"type":"v1","addr":"172.21.15.110:6827","nonce":2821892844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":2821892844},{"type":"v1","addr":"172.21.15.110:6831","nonce":2821892844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":2821892844},{"type":"v1","addr":"172.21.15.110:6829","nonce":2821892844}]},"public_addr":"172.21.15.110:6825/2821892844","cluster_addr":"172.21.15.110:6827/2821892844","heartbeat_back_addr":"172.21.15.110:6831/2821892844","heartbeat_front_addr":"172.21.15.110:6829/2821892844","state":["exists","up"]},{"osd":6,"uuid":"49324695-d83b-41ad-92e1-5348e8e9149f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":273776428},{"type":"v1","addr":"172.21.15.118:6818","nonce":273776428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":273776428},{"type":"v1","addr":"172.21.15.118:6822","nonce":273776428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":273776428},{"type":"v1","addr":"172.21.15.118:6830","nonce":273776428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":273776428},{"type":"v1","addr":"172.21.15.118:6826","nonce":273776428}]},"public_addr":"172.21.15.118:6818/273776428","cluster_addr":"172.21.15.118:6822/273776428","heartbeat_back_addr":"172.21.15.118:6830/273776428","heartbeat_front_addr":"172.21.15.118:6826/273776428","state":["exists","up"]},{"osd":7,"uuid":"afef160d-0911-4406-ad49-cd6d61ee8a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1006872173},{"type":"v1","addr":"172.21.15.155:6827","nonce":1006872173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1006872173},{"type":"v1","addr":"172.21.15.155:6829","nonce":1006872173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1006872173},{"type":"v1","addr":"172.21.15.155:6833","nonce":1006872173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1006872173},{"type":"v1","addr":"172.21.15.155:6831","nonce":1006872173}]},"public_addr":"172.21.15.155:6827/1006872173","cluster_addr":"172.21.15.155:6829/1006872173","heartbeat_back_addr":"172.21.15.155:6833/1006872173","heartbeat_front_addr":"172.21.15.155:6831/1006872173","state":["exists","up"]},{"osd":8,"uuid":"298396bf-31a2-4832-91a4-55fcb050130f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":744651323},{"type":"v1","addr":"172.21.15.33:6817","nonce":744651323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":744651323},{"type":"v1","addr":"172.21.15.33:6819","nonce":744651323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":744651323},{"type":"v1","addr":"172.21.15.33:6823","nonce":744651323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":744651323},{"type":"v1","addr":"172.21.15.33:6821","nonce":744651323}]},"public_addr":"172.21.15.33:6817/744651323","cluster_addr":"172.21.15.33:6819/744651323","heartbeat_back_addr":"172.21.15.33:6823/744651323","heartbeat_front_addr":"172.21.15.33:6821/744651323","state":["exists","up"]},{"osd":9,"uuid":"c151ae6b-279d-4174-aceb-4671fbdac3b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":872519654},{"type":"v1","addr":"172.21.15.110:6802","nonce":872519654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":872519654},{"type":"v1","addr":"172.21.15.110:6807","nonce":872519654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":872519654},{"type":"v1","addr":"172.21.15.110:6819","nonce":872519654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":872519654},{"type":"v1","addr":"172.21.15.110:6813","nonce":872519654}]},"public_addr":"172.21.15.110:6802/872519654","cluster_addr":"172.21.15.110:6807/872519654","heartbeat_back_addr":"172.21.15.110:6819/872519654","heartbeat_front_addr":"172.21.15.110:6813/872519654","state":["exists","up"]},{"osd":10,"uuid":"0b292dfb-2029-4e1e-b206-ea7e0f6969ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6817","nonce":4099141917},{"type":"v1","addr":"172.21.15.118:6819","nonce":4099141917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6821","nonce":4099141917},{"type":"v1","addr":"172.21.15.118:6823","nonce":4099141917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6829","nonce":4099141917},{"type":"v1","addr":"172.21.15.118:6831","nonce":4099141917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":4099141917},{"type":"v1","addr":"172.21.15.118:6827","nonce":4099141917}]},"public_addr":"172.21.15.118:6819/4099141917","cluster_addr":"172.21.15.118:6823/4099141917","heartbeat_back_addr":"172.21.15.118:6831/4099141917","heartbeat_front_addr":"172.21.15.118:6827/4099141917","state":["exists","up"]},{"osd":11,"uuid":"10c61619-ad11-40f0-b013-9ef52bcabf59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":3981951559},{"type":"v1","addr":"172.21.15.155:6811","nonce":3981951559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":3981951559},{"type":"v1","addr":"172.21.15.155:6817","nonce":3981951559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":3981951559},{"type":"v1","addr":"172.21.15.155:6825","nonce":3981951559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3981951559},{"type":"v1","addr":"172.21.15.155:6823","nonce":3981951559}]},"public_addr":"172.21.15.155:6811/3981951559","cluster_addr":"172.21.15.155:6817/3981951559","heartbeat_back_addr":"172.21.15.155:6825/3981951559","heartbeat_front_addr":"172.21.15.155:6823/3981951559","state":["exists","up"]},{"osd":12,"uuid":"3f0d26e5-f3e2-47b6-91a8-0daa30d0544f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":4133738119},{"type":"v1","addr":"172.21.15.33:6809","nonce":4133738119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":4133738119},{"type":"v1","addr":"172.21.15.33:6811","nonce":4133738119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":4133738119},{"type":"v1","addr":"172.21.15.33:6815","nonce":4133738119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":4133738119},{"type":"v1","addr":"172.21.15.33:6813","nonce":4133738119}]},"public_addr":"172.21.15.33:6809/4133738119","cluster_addr":"172.21.15.33:6811/4133738119","heartbeat_back_addr":"172.21.15.33:6815/4133738119","heartbeat_front_addr":"172.21.15.33:6813/4133738119","state":["exists","up"]},{"osd":13,"uuid":"c8d6213f-af5e-4646-9241-6f91589b0e1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":1583232449},{"type":"v1","addr":"172.21.15.110:6803","nonce":1583232449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":1583232449},{"type":"v1","addr":"172.21.15.110:6809","nonce":1583232449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":1583232449},{"type":"v1","addr":"172.21.15.110:6821","nonce":1583232449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":1583232449},{"type":"v1","addr":"172.21.15.110:6815","nonce":1583232449}]},"public_addr":"172.21.15.110:6803/1583232449","cluster_addr":"172.21.15.110:6809/1583232449","heartbeat_back_addr":"172.21.15.110:6821/1583232449","heartbeat_front_addr":"172.21.15.110:6815/1583232449","state":["exists","up"]},{"osd":14,"uuid":"7b3d56df-62bf-4877-b984-6f17fe1fab33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":506416463},{"type":"v1","addr":"172.21.15.118:6801","nonce":506416463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":506416463},{"type":"v1","addr":"172.21.15.118:6803","nonce":506416463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":506416463},{"type":"v1","addr":"172.21.15.118:6807","nonce":506416463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":506416463},{"type":"v1","addr":"172.21.15.118:6805","nonce":506416463}]},"public_addr":"172.21.15.118:6801/506416463","cluster_addr":"172.21.15.118:6803/506416463","heartbeat_back_addr":"172.21.15.118:6807/506416463","heartbeat_front_addr":"172.21.15.118:6805/506416463","state":["exists","up"]},{"osd":15,"uuid":"6d526edb-f1d8-415a-ad55-81e810ff4944","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6807","nonce":741132602},{"type":"v1","addr":"172.21.15.155:6810","nonce":741132602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":741132602},{"type":"v1","addr":"172.21.15.155:6815","nonce":741132602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6821","nonce":741132602},{"type":"v1","addr":"172.21.15.155:6822","nonce":741132602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":741132602},{"type":"v1","addr":"172.21.15.155:6819","nonce":741132602}]},"public_addr":"172.21.15.155:6810/741132602","cluster_addr":"172.21.15.155:6815/741132602","heartbeat_back_addr":"172.21.15.155:6822/741132602","heartbeat_front_addr":"172.21.15.155:6819/741132602","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.813806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.677618+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.676628+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.659122+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.648122+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.859759+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.759643+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.897847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.779763+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.813460+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.479489+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.853984+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.736100+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.838084+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.629904+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.712166+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T01:03:54.359 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-19T01:03:54.359 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-19T01:03:54.668 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:03:54.668 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":28,"fsid":"f11bd2d5-4456-4ffa-83c0-722dddd22ad3","created":"2024-04-19T01:03:36.766089+0000","modified":"2024-04-19T01:03:54.645990+0000","last_up_change":"2024-04-19T01:03:50.591452+0000","last_in_change":"2024-04-19T01:03:41.227644+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-19T01:03:49.571609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-19T01:03:51.913782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.0199999809265137,"score_stable":6.0199999809265137,"optimal_score":0.81000000238418579,"raw_score_acting":4.880000114440918,"raw_score_stable":4.880000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"296899fd-2b2b-4071-8b1f-0f92c3d0bbc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":1633616806},{"type":"v1","addr":"172.21.15.33:6825","nonce":1633616806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":1633616806},{"type":"v1","addr":"172.21.15.33:6827","nonce":1633616806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":1633616806},{"type":"v1","addr":"172.21.15.33:6831","nonce":1633616806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":1633616806},{"type":"v1","addr":"172.21.15.33:6829","nonce":1633616806}]},"public_addr":"172.21.15.33:6825/1633616806","cluster_addr":"172.21.15.33:6827/1633616806","heartbeat_back_addr":"172.21.15.33:6831/1633616806","heartbeat_front_addr":"172.21.15.33:6829/1633616806","state":["exists","up"]},{"osd":1,"uuid":"021f7190-0ec3-4553-a44f-15095765cccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":4270148213},{"type":"v1","addr":"172.21.15.110:6808","nonce":4270148213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6811","nonce":4270148213},{"type":"v1","addr":"172.21.15.110:6814","nonce":4270148213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":4270148213},{"type":"v1","addr":"172.21.15.110:6823","nonce":4270148213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6817","nonce":4270148213},{"type":"v1","addr":"172.21.15.110:6820","nonce":4270148213}]},"public_addr":"172.21.15.110:6808/4270148213","cluster_addr":"172.21.15.110:6814/4270148213","heartbeat_back_addr":"172.21.15.110:6823/4270148213","heartbeat_front_addr":"172.21.15.110:6820/4270148213","state":["exists","up"]},{"osd":2,"uuid":"c283d0bc-5e34-4763-9019-328ee9b6c07a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":13248188},{"type":"v1","addr":"172.21.15.118:6809","nonce":13248188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":13248188},{"type":"v1","addr":"172.21.15.118:6811","nonce":13248188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":13248188},{"type":"v1","addr":"172.21.15.118:6815","nonce":13248188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":13248188},{"type":"v1","addr":"172.21.15.118:6813","nonce":13248188}]},"public_addr":"172.21.15.118:6809/13248188","cluster_addr":"172.21.15.118:6811/13248188","heartbeat_back_addr":"172.21.15.118:6815/13248188","heartbeat_front_addr":"172.21.15.118:6813/13248188","state":["exists","up"]},{"osd":3,"uuid":"d45888f2-eecd-480b-80f5-c0731a06ff3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":2658385904},{"type":"v1","addr":"172.21.15.155:6803","nonce":2658385904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":2658385904},{"type":"v1","addr":"172.21.15.155:6805","nonce":2658385904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6813","nonce":2658385904},{"type":"v1","addr":"172.21.15.155:6816","nonce":2658385904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":2658385904},{"type":"v1","addr":"172.21.15.155:6809","nonce":2658385904}]},"public_addr":"172.21.15.155:6803/2658385904","cluster_addr":"172.21.15.155:6805/2658385904","heartbeat_back_addr":"172.21.15.155:6816/2658385904","heartbeat_front_addr":"172.21.15.155:6809/2658385904","state":["exists","up"]},{"osd":4,"uuid":"d060d939-8c94-4a07-9fd5-42af0044a16f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":2389566155},{"type":"v1","addr":"172.21.15.33:6801","nonce":2389566155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":2389566155},{"type":"v1","addr":"172.21.15.33:6803","nonce":2389566155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":2389566155},{"type":"v1","addr":"172.21.15.33:6807","nonce":2389566155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":2389566155},{"type":"v1","addr":"172.21.15.33:6805","nonce":2389566155}]},"public_addr":"172.21.15.33:6801/2389566155","cluster_addr":"172.21.15.33:6803/2389566155","heartbeat_back_addr":"172.21.15.33:6807/2389566155","heartbeat_front_addr":"172.21.15.33:6805/2389566155","state":["exists","up"]},{"osd":5,"uuid":"9700d2f7-ed56-4143-ba9d-6cebe524e877","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":2821892844},{"type":"v1","addr":"172.21.15.110:6825","nonce":2821892844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":2821892844},{"type":"v1","addr":"172.21.15.110:6827","nonce":2821892844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":2821892844},{"type":"v1","addr":"172.21.15.110:6831","nonce":2821892844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":2821892844},{"type":"v1","addr":"172.21.15.110:6829","nonce":2821892844}]},"public_addr":"172.21.15.110:6825/2821892844","cluster_addr":"172.21.15.110:6827/2821892844","heartbeat_back_addr":"172.21.15.110:6831/2821892844","heartbeat_front_addr":"172.21.15.110:6829/2821892844","state":["exists","up"]},{"osd":6,"uuid":"49324695-d83b-41ad-92e1-5348e8e9149f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":273776428},{"type":"v1","addr":"172.21.15.118:6818","nonce":273776428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":273776428},{"type":"v1","addr":"172.21.15.118:6822","nonce":273776428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":273776428},{"type":"v1","addr":"172.21.15.118:6830","nonce":273776428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":273776428},{"type":"v1","addr":"172.21.15.118:6826","nonce":273776428}]},"public_addr":"172.21.15.118:6818/273776428","cluster_addr":"172.21.15.118:6822/273776428","heartbeat_back_addr":"172.21.15.118:6830/273776428","heartbeat_front_addr":"172.21.15.118:6826/273776428","state":["exists","up"]},{"osd":7,"uuid":"afef160d-0911-4406-ad49-cd6d61ee8a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1006872173},{"type":"v1","addr":"172.21.15.155:6827","nonce":1006872173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1006872173},{"type":"v1","addr":"172.21.15.155:6829","nonce":1006872173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1006872173},{"type":"v1","addr":"172.21.15.155:6833","nonce":1006872173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1006872173},{"type":"v1","addr":"172.21.15.155:6831","nonce":1006872173}]},"public_addr":"172.21.15.155:6827/1006872173","cluster_addr":"172.21.15.155:6829/1006872173","heartbeat_back_addr":"172.21.15.155:6833/1006872173","heartbeat_front_addr":"172.21.15.155:6831/1006872173","state":["exists","up"]},{"osd":8,"uuid":"298396bf-31a2-4832-91a4-55fcb050130f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":744651323},{"type":"v1","addr":"172.21.15.33:6817","nonce":744651323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":744651323},{"type":"v1","addr":"172.21.15.33:6819","nonce":744651323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":744651323},{"type":"v1","addr":"172.21.15.33:6823","nonce":744651323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":744651323},{"type":"v1","addr":"172.21.15.33:6821","nonce":744651323}]},"public_addr":"172.21.15.33:6817/744651323","cluster_addr":"172.21.15.33:6819/744651323","heartbeat_back_addr":"172.21.15.33:6823/744651323","heartbeat_front_addr":"172.21.15.33:6821/744651323","state":["exists","up"]},{"osd":9,"uuid":"c151ae6b-279d-4174-aceb-4671fbdac3b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":872519654},{"type":"v1","addr":"172.21.15.110:6802","nonce":872519654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":872519654},{"type":"v1","addr":"172.21.15.110:6807","nonce":872519654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":872519654},{"type":"v1","addr":"172.21.15.110:6819","nonce":872519654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":872519654},{"type":"v1","addr":"172.21.15.110:6813","nonce":872519654}]},"public_addr":"172.21.15.110:6802/872519654","cluster_addr":"172.21.15.110:6807/872519654","heartbeat_back_addr":"172.21.15.110:6819/872519654","heartbeat_front_addr":"172.21.15.110:6813/872519654","state":["exists","up"]},{"osd":10,"uuid":"0b292dfb-2029-4e1e-b206-ea7e0f6969ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6817","nonce":4099141917},{"type":"v1","addr":"172.21.15.118:6819","nonce":4099141917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6821","nonce":4099141917},{"type":"v1","addr":"172.21.15.118:6823","nonce":4099141917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6829","nonce":4099141917},{"type":"v1","addr":"172.21.15.118:6831","nonce":4099141917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":4099141917},{"type":"v1","addr":"172.21.15.118:6827","nonce":4099141917}]},"public_addr":"172.21.15.118:6819/4099141917","cluster_addr":"172.21.15.118:6823/4099141917","heartbeat_back_addr":"172.21.15.118:6831/4099141917","heartbeat_front_addr":"172.21.15.118:6827/4099141917","state":["exists","up"]},{"osd":11,"uuid":"10c61619-ad11-40f0-b013-9ef52bcabf59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":3981951559},{"type":"v1","addr":"172.21.15.155:6811","nonce":3981951559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":3981951559},{"type":"v1","addr":"172.21.15.155:6817","nonce":3981951559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":3981951559},{"type":"v1","addr":"172.21.15.155:6825","nonce":3981951559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3981951559},{"type":"v1","addr":"172.21.15.155:6823","nonce":3981951559}]},"public_addr":"172.21.15.155:6811/3981951559","cluster_addr":"172.21.15.155:6817/3981951559","heartbeat_back_addr":"172.21.15.155:6825/3981951559","heartbeat_front_addr":"172.21.15.155:6823/3981951559","state":["exists","up"]},{"osd":12,"uuid":"3f0d26e5-f3e2-47b6-91a8-0daa30d0544f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":4133738119},{"type":"v1","addr":"172.21.15.33:6809","nonce":4133738119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":4133738119},{"type":"v1","addr":"172.21.15.33:6811","nonce":4133738119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":4133738119},{"type":"v1","addr":"172.21.15.33:6815","nonce":4133738119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":4133738119},{"type":"v1","addr":"172.21.15.33:6813","nonce":4133738119}]},"public_addr":"172.21.15.33:6809/4133738119","cluster_addr":"172.21.15.33:6811/4133738119","heartbeat_back_addr":"172.21.15.33:6815/4133738119","heartbeat_front_addr":"172.21.15.33:6813/4133738119","state":["exists","up"]},{"osd":13,"uuid":"c8d6213f-af5e-4646-9241-6f91589b0e1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":1583232449},{"type":"v1","addr":"172.21.15.110:6803","nonce":1583232449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":1583232449},{"type":"v1","addr":"172.21.15.110:6809","nonce":1583232449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":1583232449},{"type":"v1","addr":"172.21.15.110:6821","nonce":1583232449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":1583232449},{"type":"v1","addr":"172.21.15.110:6815","nonce":1583232449}]},"public_addr":"172.21.15.110:6803/1583232449","cluster_addr":"172.21.15.110:6809/1583232449","heartbeat_back_addr":"172.21.15.110:6821/1583232449","heartbeat_front_addr":"172.21.15.110:6815/1583232449","state":["exists","up"]},{"osd":14,"uuid":"7b3d56df-62bf-4877-b984-6f17fe1fab33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":506416463},{"type":"v1","addr":"172.21.15.118:6801","nonce":506416463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":506416463},{"type":"v1","addr":"172.21.15.118:6803","nonce":506416463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":506416463},{"type":"v1","addr":"172.21.15.118:6807","nonce":506416463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":506416463},{"type":"v1","addr":"172.21.15.118:6805","nonce":506416463}]},"public_addr":"172.21.15.118:6801/506416463","cluster_addr":"172.21.15.118:6803/506416463","heartbeat_back_addr":"172.21.15.118:6807/506416463","heartbeat_front_addr":"172.21.15.118:6805/506416463","state":["exists","up"]},{"osd":15,"uuid":"6d526edb-f1d8-415a-ad55-81e810ff4944","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6807","nonce":741132602},{"type":"v1","addr":"172.21.15.155:6810","nonce":741132602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":741132602},{"type":"v1","addr":"172.21.15.155:6815","nonce":741132602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6821","nonce":741132602},{"type":"v1","addr":"172.21.15.155:6822","nonce":741132602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":741132602},{"type":"v1","addr":"172.21.15.155:6819","nonce":741132602}]},"public_addr":"172.21.15.155:6810/741132602","cluster_addr":"172.21.15.155:6815/741132602","heartbeat_back_addr":"172.21.15.155:6822/741132602","heartbeat_front_addr":"172.21.15.155:6819/741132602","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.813806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.677618+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.676628+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.659122+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.648122+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.859759+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.759643+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.897847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.779763+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.813460+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.479489+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.853984+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.736100+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.838084+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.629904+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:03:46.712166+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T01:03:54.687 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-19T01:03:54.687 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-19T01:03:54.688 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-19T01:03:54.688 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-19T01:03:54.688 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-19T01:03:54.689 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-19T01:03:54.689 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-19T01:03:54.690 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-19T01:03:54.690 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-04-19T01:03:54.690 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-04-19T01:03:54.691 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2024-04-19T01:03:54.691 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2024-04-19T01:03:54.692 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 flush_pg_stats 2024-04-19T01:03:54.692 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 flush_pg_stats 2024-04-19T01:03:54.693 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 flush_pg_stats 2024-04-19T01:03:54.694 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 flush_pg_stats 2024-04-19T01:03:55.115 INFO:teuthology.orchestra.run.smithi033.stdout:94489280515 2024-04-19T01:03:55.116 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-04-19T01:03:55.128 INFO:teuthology.orchestra.run.smithi033.stdout:94489280515 2024-04-19T01:03:55.129 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-19T01:03:55.150 INFO:teuthology.orchestra.run.smithi033.stdout:103079215108 2024-04-19T01:03:55.150 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-04-19T01:03:55.153 INFO:teuthology.orchestra.run.smithi033.stdout:94489280516 2024-04-19T01:03:55.154 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-19T01:03:55.159 INFO:teuthology.orchestra.run.smithi033.stdout:94489280516 2024-04-19T01:03:55.159 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-19T01:03:55.169 INFO:teuthology.orchestra.run.smithi033.stdout:94489280517 2024-04-19T01:03:55.169 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-19T01:03:55.179 INFO:teuthology.orchestra.run.smithi033.stdout:103079215108 2024-04-19T01:03:55.180 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-19T01:03:55.196 INFO:teuthology.orchestra.run.smithi033.stdout:94489280515 2024-04-19T01:03:55.196 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-04-19T01:03:55.210 INFO:teuthology.orchestra.run.smithi033.stdout:94489280516 2024-04-19T01:03:55.211 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-19T01:03:55.249 INFO:teuthology.orchestra.run.smithi033.stdout:103079215109 2024-04-19T01:03:55.249 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-04-19T01:03:55.253 INFO:teuthology.orchestra.run.smithi033.stdout:90194313221 2024-04-19T01:03:55.253 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-04-19T01:03:55.255 INFO:teuthology.orchestra.run.smithi033.stdout:103079215109 2024-04-19T01:03:55.255 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-19T01:03:55.288 INFO:teuthology.orchestra.run.smithi033.stdout:94489280517 2024-04-19T01:03:55.289 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-04-19T01:03:55.289 INFO:teuthology.orchestra.run.smithi033.stdout:103079215110 2024-04-19T01:03:55.290 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-04-19T01:03:55.291 INFO:teuthology.orchestra.run.smithi033.stdout:94489280516 2024-04-19T01:03:55.291 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-04-19T01:03:55.345 INFO:teuthology.orchestra.run.smithi033.stdout:103079215108 2024-04-19T01:03:55.346 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-19T01:03:56.064 INFO:teuthology.orchestra.run.smithi033.stdout:94489280516 2024-04-19T01:03:56.106 INFO:teuthology.orchestra.run.smithi033.stdout:94489280515 2024-04-19T01:03:56.112 INFO:teuthology.orchestra.run.smithi033.stdout:103079215109 2024-04-19T01:03:56.123 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.3 2024-04-19T01:03:56.123 INFO:teuthology.orchestra.run.smithi033.stdout:94489280516 2024-04-19T01:03:56.124 INFO:teuthology.orchestra.run.smithi033.stdout:94489280515 2024-04-19T01:03:56.124 DEBUG:teuthology.parallel:result is None 2024-04-19T01:03:56.175 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.6 2024-04-19T01:03:56.175 DEBUG:teuthology.parallel:result is None 2024-04-19T01:03:56.181 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215109 got 103079215109 for osd.13 2024-04-19T01:03:56.181 DEBUG:teuthology.parallel:result is None 2024-04-19T01:03:56.185 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.5 2024-04-19T01:03:56.186 DEBUG:teuthology.parallel:result is None 2024-04-19T01:03:56.193 INFO:teuthology.orchestra.run.smithi033.stdout:103079215108 2024-04-19T01:03:56.194 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.11 2024-04-19T01:03:56.194 DEBUG:teuthology.parallel:result is None 2024-04-19T01:03:56.200 INFO:teuthology.orchestra.run.smithi033.stdout:90194313221 2024-04-19T01:03:56.205 INFO:teuthology.orchestra.run.smithi033.stdout:103079215108 2024-04-19T01:03:56.210 INFO:teuthology.orchestra.run.smithi033.stdout:94489280516 2024-04-19T01:03:56.239 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215108 got 103079215108 for osd.14 2024-04-19T01:03:56.239 DEBUG:teuthology.parallel:result is None 2024-04-19T01:03:56.250 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313221 got 90194313221 for osd.10 2024-04-19T01:03:56.250 DEBUG:teuthology.parallel:result is None 2024-04-19T01:03:56.254 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215108 got 103079215108 for osd.2 2024-04-19T01:03:56.254 DEBUG:teuthology.parallel:result is None 2024-04-19T01:03:56.263 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.4 2024-04-19T01:03:56.263 DEBUG:teuthology.parallel:result is None 2024-04-19T01:03:56.318 INFO:teuthology.orchestra.run.smithi033.stdout:103079215110 2024-04-19T01:03:56.319 INFO:teuthology.orchestra.run.smithi033.stdout:94489280516 2024-04-19T01:03:56.319 INFO:teuthology.orchestra.run.smithi033.stdout:94489280515 2024-04-19T01:03:56.347 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.15 2024-04-19T01:03:56.347 DEBUG:teuthology.parallel:result is None 2024-04-19T01:03:56.348 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215110 got 103079215110 for osd.9 2024-04-19T01:03:56.348 INFO:teuthology.orchestra.run.smithi033.stdout:94489280517 2024-04-19T01:03:56.349 DEBUG:teuthology.parallel:result is None 2024-04-19T01:03:56.349 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.12 2024-04-19T01:03:56.350 DEBUG:teuthology.parallel:result is None 2024-04-19T01:03:56.352 INFO:teuthology.orchestra.run.smithi033.stdout:103079215109 2024-04-19T01:03:56.352 INFO:teuthology.orchestra.run.smithi033.stdout:94489280517 2024-04-19T01:03:56.373 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215109 got 103079215109 for osd.0 2024-04-19T01:03:56.374 INFO:teuthology.orchestra.run.smithi033.stdout:103079215108 2024-04-19T01:03:56.374 DEBUG:teuthology.parallel:result is None 2024-04-19T01:03:56.379 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280517 for osd.8 2024-04-19T01:03:56.379 DEBUG:teuthology.parallel:result is None 2024-04-19T01:03:56.381 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280517 for osd.1 2024-04-19T01:03:56.381 DEBUG:teuthology.parallel:result is None 2024-04-19T01:03:56.391 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215108 got 103079215108 for osd.7 2024-04-19T01:03:56.392 DEBUG:teuthology.parallel:result is None 2024-04-19T01:03:56.392 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-19T01:03:56.392 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T01:03:56.712 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:03:56.713 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-04-19T01:03:56.729 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-04-19T01:03:55.516724+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":13,"kb":1509949440,"kb_used":453232,"kb_used_data":4120,"kb_used_omap":24,"kb_used_meta":448743,"kb_avail":1509496208,"statfs":{"total":1546188226560,"available":1545724116992,"internally_reserved":0,"allocated":4218880,"data_stored":2791144,"data_compressed":141998,"data_compressed_allocated":344064,"data_compressed_original":1581056,"omap_allocated":25520,"internal_metadata":459512912},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.861441"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-19T01:03:54.658079+0000","last_change":"2024-04-19T01:03:53.939441+0000","last_active":"2024-04-19T01:03:54.658079+0000","last_peered":"2024-04-19T01:03:54.658079+0000","last_clean":"2024-04-19T01:03:54.658079+0000","last_became_active":"2024-04-19T01:03:53.938774+0000","last_became_peered":"2024-04-19T01:03:53.938774+0000","last_unstale":"2024-04-19T01:03:54.658079+0000","last_undegraded":"2024-04-19T01:03:54.658079+0000","last_fullsized":"2024-04-19T01:03:54.658079+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:14.081709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-19T01:03:53.737367+0000","last_change":"2024-04-19T01:03:53.736290+0000","last_active":"2024-04-19T01:03:53.737367+0000","last_peered":"2024-04-19T01:03:53.737367+0000","last_clean":"2024-04-19T01:03:53.737367+0000","last_became_active":"2024-04-19T01:03:53.735549+0000","last_became_peered":"2024-04-19T01:03:53.735549+0000","last_unstale":"2024-04-19T01:03:53.737367+0000","last_undegraded":"2024-04-19T01:03:53.737367+0000","last_fullsized":"2024-04-19T01:03:53.737367+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:18.742234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13],"acting":[14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-19T01:03:53.773801+0000","last_change":"2024-04-19T01:03:53.772709+0000","last_active":"2024-04-19T01:03:53.773801+0000","last_peered":"2024-04-19T01:03:53.773801+0000","last_clean":"2024-04-19T01:03:53.773801+0000","last_became_active":"2024-04-19T01:03:53.771887+0000","last_became_peered":"2024-04-19T01:03:53.771887+0000","last_unstale":"2024-04-19T01:03:53.773801+0000","last_undegraded":"2024-04-19T01:03:53.773801+0000","last_fullsized":"2024-04-19T01:03:53.773801+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:08.024630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-19T01:03:54.659881+0000","last_change":"2024-04-19T01:03:53.716606+0000","last_active":"2024-04-19T01:03:54.659881+0000","last_peered":"2024-04-19T01:03:54.659881+0000","last_clean":"2024-04-19T01:03:54.659881+0000","last_became_active":"2024-04-19T01:03:53.716040+0000","last_became_peered":"2024-04-19T01:03:53.716040+0000","last_unstale":"2024-04-19T01:03:54.659881+0000","last_undegraded":"2024-04-19T01:03:54.659881+0000","last_fullsized":"2024-04-19T01:03:54.659881+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:01.975515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-19T01:03:54.658077+0000","last_change":"2024-04-19T01:03:54.012138+0000","last_active":"2024-04-19T01:03:54.658077+0000","last_peered":"2024-04-19T01:03:54.658077+0000","last_clean":"2024-04-19T01:03:54.658077+0000","last_became_active":"2024-04-19T01:03:54.011460+0000","last_became_peered":"2024-04-19T01:03:54.011460+0000","last_unstale":"2024-04-19T01:03:54.658077+0000","last_undegraded":"2024-04-19T01:03:54.658077+0000","last_fullsized":"2024-04-19T01:03:54.658077+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:04:58.336612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-19T01:03:53.653989+0000","last_change":"2024-04-19T01:03:53.650434+0000","last_active":"2024-04-19T01:03:53.653989+0000","last_peered":"2024-04-19T01:03:53.653989+0000","last_clean":"2024-04-19T01:03:53.653989+0000","last_became_active":"2024-04-19T01:03:53.649606+0000","last_became_peered":"2024-04-19T01:03:53.649606+0000","last_unstale":"2024-04-19T01:03:53.653989+0000","last_undegraded":"2024-04-19T01:03:53.653989+0000","last_fullsized":"2024-04-19T01:03:53.653989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:07.117052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-19T01:03:54.657910+0000","last_change":"2024-04-19T01:03:53.760075+0000","last_active":"2024-04-19T01:03:54.657910+0000","last_peered":"2024-04-19T01:03:54.657910+0000","last_clean":"2024-04-19T01:03:54.657910+0000","last_became_active":"2024-04-19T01:03:53.759326+0000","last_became_peered":"2024-04-19T01:03:53.759326+0000","last_unstale":"2024-04-19T01:03:54.657910+0000","last_undegraded":"2024-04-19T01:03:54.657910+0000","last_fullsized":"2024-04-19T01:03:54.657910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:02.718012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-19T01:03:53.769740+0000","last_change":"2024-04-19T01:03:53.766587+0000","last_active":"2024-04-19T01:03:53.769740+0000","last_peered":"2024-04-19T01:03:53.769740+0000","last_clean":"2024-04-19T01:03:53.769740+0000","last_became_active":"2024-04-19T01:03:53.765967+0000","last_became_peered":"2024-04-19T01:03:53.765967+0000","last_unstale":"2024-04-19T01:03:53.769740+0000","last_undegraded":"2024-04-19T01:03:53.769740+0000","last_fullsized":"2024-04-19T01:03:53.769740+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:04:58.925212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"25'157","reported_seq":212,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-19T01:03:53.645658+0000","last_change":"2024-04-19T01:03:50.776344+0000","last_active":"2024-04-19T01:03:53.645658+0000","last_peered":"2024-04-19T01:03:53.645658+0000","last_clean":"2024-04-19T01:03:53.645658+0000","last_became_active":"2024-04-19T01:03:50.774078+0000","last_became_peered":"2024-04-19T01:03:50.774078+0000","last_unstale":"2024-04-19T01:03:53.645658+0000","last_undegraded":"2024-04-19T01:03:53.645658+0000","last_fullsized":"2024-04-19T01:03:53.645658+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:49.580851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:49.580851+0000","last_clean_scrub_stamp":"2024-04-19T01:03:49.580851+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:07.495976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":13},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":15546,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":14,"up_from":24,"seq":103079215108,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":253952,"data_stored":105965,"data_compressed":7435,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,7,8,9,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":24,"seq":103079215109,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31452,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340388,"statfs":{"total":96636764160,"available":96604557312,"internally_reserved":0,"allocated":266240,"data_stored":115427,"data_compressed":9308,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,8,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":24,"seq":103079215110,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31432,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340408,"statfs":{"total":96636764160,"available":96604577792,"internally_reserved":0,"allocated":253952,"data_stored":105965,"data_compressed":7435,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313221,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":105965,"data_compressed":7435,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[3,4,5,6,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":115427,"data_compressed":9308,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,7,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":105965,"data_compressed":7435,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,5,6,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":105965,"data_compressed":7435,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[2,4,5,6,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":105965,"data_compressed":7435,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":307200,"data_stored":634893,"data_compressed":15208,"data_compressed_allocated":49152,"data_compressed_original":544768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,6,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280517,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":115427,"data_compressed":9308,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[3,4,5,6,7,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":307200,"data_stored":634893,"data_compressed":15208,"data_compressed_allocated":49152,"data_compressed_original":544768,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,3,4,5,6,8,9,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":105965,"data_compressed":7435,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,8,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":105965,"data_compressed":7435,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,4,5,6,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":24,"seq":103079215108,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31428,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340412,"statfs":{"total":96636764160,"available":96604581888,"internally_reserved":0,"allocated":253952,"data_stored":105965,"data_compressed":7435,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215109,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":105965,"data_compressed":7435,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,10,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":24,"seq":103079215108,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":115427,"data_compressed":9308,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T01:03:56.730 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T01:03:57.019 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:03:57.020 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-04-19T01:03:57.036 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-04-19T01:03:55.516724+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":13,"kb":1509949440,"kb_used":453232,"kb_used_data":4120,"kb_used_omap":24,"kb_used_meta":448743,"kb_avail":1509496208,"statfs":{"total":1546188226560,"available":1545724116992,"internally_reserved":0,"allocated":4218880,"data_stored":2791144,"data_compressed":141998,"data_compressed_allocated":344064,"data_compressed_original":1581056,"omap_allocated":25520,"internal_metadata":459512912},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.861441"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-19T01:03:54.658079+0000","last_change":"2024-04-19T01:03:53.939441+0000","last_active":"2024-04-19T01:03:54.658079+0000","last_peered":"2024-04-19T01:03:54.658079+0000","last_clean":"2024-04-19T01:03:54.658079+0000","last_became_active":"2024-04-19T01:03:53.938774+0000","last_became_peered":"2024-04-19T01:03:53.938774+0000","last_unstale":"2024-04-19T01:03:54.658079+0000","last_undegraded":"2024-04-19T01:03:54.658079+0000","last_fullsized":"2024-04-19T01:03:54.658079+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:14.081709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-19T01:03:53.737367+0000","last_change":"2024-04-19T01:03:53.736290+0000","last_active":"2024-04-19T01:03:53.737367+0000","last_peered":"2024-04-19T01:03:53.737367+0000","last_clean":"2024-04-19T01:03:53.737367+0000","last_became_active":"2024-04-19T01:03:53.735549+0000","last_became_peered":"2024-04-19T01:03:53.735549+0000","last_unstale":"2024-04-19T01:03:53.737367+0000","last_undegraded":"2024-04-19T01:03:53.737367+0000","last_fullsized":"2024-04-19T01:03:53.737367+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:18.742234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13],"acting":[14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-19T01:03:53.773801+0000","last_change":"2024-04-19T01:03:53.772709+0000","last_active":"2024-04-19T01:03:53.773801+0000","last_peered":"2024-04-19T01:03:53.773801+0000","last_clean":"2024-04-19T01:03:53.773801+0000","last_became_active":"2024-04-19T01:03:53.771887+0000","last_became_peered":"2024-04-19T01:03:53.771887+0000","last_unstale":"2024-04-19T01:03:53.773801+0000","last_undegraded":"2024-04-19T01:03:53.773801+0000","last_fullsized":"2024-04-19T01:03:53.773801+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:08.024630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-19T01:03:54.659881+0000","last_change":"2024-04-19T01:03:53.716606+0000","last_active":"2024-04-19T01:03:54.659881+0000","last_peered":"2024-04-19T01:03:54.659881+0000","last_clean":"2024-04-19T01:03:54.659881+0000","last_became_active":"2024-04-19T01:03:53.716040+0000","last_became_peered":"2024-04-19T01:03:53.716040+0000","last_unstale":"2024-04-19T01:03:54.659881+0000","last_undegraded":"2024-04-19T01:03:54.659881+0000","last_fullsized":"2024-04-19T01:03:54.659881+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:01.975515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-19T01:03:54.658077+0000","last_change":"2024-04-19T01:03:54.012138+0000","last_active":"2024-04-19T01:03:54.658077+0000","last_peered":"2024-04-19T01:03:54.658077+0000","last_clean":"2024-04-19T01:03:54.658077+0000","last_became_active":"2024-04-19T01:03:54.011460+0000","last_became_peered":"2024-04-19T01:03:54.011460+0000","last_unstale":"2024-04-19T01:03:54.658077+0000","last_undegraded":"2024-04-19T01:03:54.658077+0000","last_fullsized":"2024-04-19T01:03:54.658077+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:04:58.336612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-19T01:03:53.653989+0000","last_change":"2024-04-19T01:03:53.650434+0000","last_active":"2024-04-19T01:03:53.653989+0000","last_peered":"2024-04-19T01:03:53.653989+0000","last_clean":"2024-04-19T01:03:53.653989+0000","last_became_active":"2024-04-19T01:03:53.649606+0000","last_became_peered":"2024-04-19T01:03:53.649606+0000","last_unstale":"2024-04-19T01:03:53.653989+0000","last_undegraded":"2024-04-19T01:03:53.653989+0000","last_fullsized":"2024-04-19T01:03:53.653989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:07.117052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-19T01:03:54.657910+0000","last_change":"2024-04-19T01:03:53.760075+0000","last_active":"2024-04-19T01:03:54.657910+0000","last_peered":"2024-04-19T01:03:54.657910+0000","last_clean":"2024-04-19T01:03:54.657910+0000","last_became_active":"2024-04-19T01:03:53.759326+0000","last_became_peered":"2024-04-19T01:03:53.759326+0000","last_unstale":"2024-04-19T01:03:54.657910+0000","last_undegraded":"2024-04-19T01:03:54.657910+0000","last_fullsized":"2024-04-19T01:03:54.657910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:02.718012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-19T01:03:53.769740+0000","last_change":"2024-04-19T01:03:53.766587+0000","last_active":"2024-04-19T01:03:53.769740+0000","last_peered":"2024-04-19T01:03:53.769740+0000","last_clean":"2024-04-19T01:03:53.769740+0000","last_became_active":"2024-04-19T01:03:53.765967+0000","last_became_peered":"2024-04-19T01:03:53.765967+0000","last_unstale":"2024-04-19T01:03:53.769740+0000","last_undegraded":"2024-04-19T01:03:53.769740+0000","last_fullsized":"2024-04-19T01:03:53.769740+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:04:58.925212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"25'157","reported_seq":212,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-19T01:03:53.645658+0000","last_change":"2024-04-19T01:03:50.776344+0000","last_active":"2024-04-19T01:03:53.645658+0000","last_peered":"2024-04-19T01:03:53.645658+0000","last_clean":"2024-04-19T01:03:53.645658+0000","last_became_active":"2024-04-19T01:03:50.774078+0000","last_became_peered":"2024-04-19T01:03:50.774078+0000","last_unstale":"2024-04-19T01:03:53.645658+0000","last_undegraded":"2024-04-19T01:03:53.645658+0000","last_fullsized":"2024-04-19T01:03:53.645658+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:49.580851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:49.580851+0000","last_clean_scrub_stamp":"2024-04-19T01:03:49.580851+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:07.495976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":13},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":15546,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":14,"up_from":24,"seq":103079215108,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":253952,"data_stored":105965,"data_compressed":7435,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,7,8,9,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":24,"seq":103079215109,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31452,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340388,"statfs":{"total":96636764160,"available":96604557312,"internally_reserved":0,"allocated":266240,"data_stored":115427,"data_compressed":9308,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,8,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":24,"seq":103079215110,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31432,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340408,"statfs":{"total":96636764160,"available":96604577792,"internally_reserved":0,"allocated":253952,"data_stored":105965,"data_compressed":7435,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313221,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":105965,"data_compressed":7435,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[3,4,5,6,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":115427,"data_compressed":9308,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,7,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":105965,"data_compressed":7435,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,5,6,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":105965,"data_compressed":7435,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[2,4,5,6,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":105965,"data_compressed":7435,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":307200,"data_stored":634893,"data_compressed":15208,"data_compressed_allocated":49152,"data_compressed_original":544768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,6,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280517,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":115427,"data_compressed":9308,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[3,4,5,6,7,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":307200,"data_stored":634893,"data_compressed":15208,"data_compressed_allocated":49152,"data_compressed_original":544768,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,3,4,5,6,8,9,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":105965,"data_compressed":7435,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,8,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":105965,"data_compressed":7435,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,4,5,6,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":24,"seq":103079215108,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31428,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340412,"statfs":{"total":96636764160,"available":96604581888,"internally_reserved":0,"allocated":253952,"data_stored":105965,"data_compressed":7435,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215109,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":105965,"data_compressed":7435,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,10,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":24,"seq":103079215108,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":115427,"data_compressed":9308,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T01:03:57.037 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-19T01:03:57.037 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-19T01:03:57.037 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-19T01:03:57.037 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-19T01:03:57.351 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:03:57.351 INFO:teuthology.orchestra.run.smithi033.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-19T01:03:57.369 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-19T01:03:57.369 INFO:teuthology.run_tasks:Running task thrashosds... 2024-04-19T01:03:57.383 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-19T01:03:57.384 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-yuri5-testing-2024-04-17-1400'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': '8b4ab94f9c30da63487fa2f241ce12f483378860'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8b4ab94f9c30da63487fa2f241ce12f483378860'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri5-testing-2024-04-17-1400', 'sha1': '8b4ab94f9c30da63487fa2f241ce12f483378860'}} 2024-04-19T01:03:57.384 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-04-19T01:03:57.384 INFO:tasks.thrashosds:Beginning thrashosds... 2024-04-19T01:03:57.384 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-19T01:03:57.384 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T01:03:57.719 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:03:57.720 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-04-19T01:03:57.736 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-04-19T01:03:57.517389+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":13,"kb":1509949440,"kb_used":453244,"kb_used_data":4144,"kb_used_omap":24,"kb_used_meta":448743,"kb_avail":1509496196,"statfs":{"total":1546188226560,"available":1545724104704,"internally_reserved":0,"allocated":4243456,"data_stored":2810068,"data_compressed":145744,"data_compressed_allocated":352256,"data_compressed_original":1597440,"omap_allocated":25520,"internal_metadata":459512912},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.862107"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-19T01:03:54.658079+0000","last_change":"2024-04-19T01:03:53.939441+0000","last_active":"2024-04-19T01:03:54.658079+0000","last_peered":"2024-04-19T01:03:54.658079+0000","last_clean":"2024-04-19T01:03:54.658079+0000","last_became_active":"2024-04-19T01:03:53.938774+0000","last_became_peered":"2024-04-19T01:03:53.938774+0000","last_unstale":"2024-04-19T01:03:54.658079+0000","last_undegraded":"2024-04-19T01:03:54.658079+0000","last_fullsized":"2024-04-19T01:03:54.658079+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:14.081709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-19T01:03:55.308912+0000","last_change":"2024-04-19T01:03:53.736290+0000","last_active":"2024-04-19T01:03:55.308912+0000","last_peered":"2024-04-19T01:03:55.308912+0000","last_clean":"2024-04-19T01:03:55.308912+0000","last_became_active":"2024-04-19T01:03:53.735549+0000","last_became_peered":"2024-04-19T01:03:53.735549+0000","last_unstale":"2024-04-19T01:03:55.308912+0000","last_undegraded":"2024-04-19T01:03:55.308912+0000","last_fullsized":"2024-04-19T01:03:55.308912+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:07.640918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13],"acting":[14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-19T01:03:55.734940+0000","last_change":"2024-04-19T01:03:53.772709+0000","last_active":"2024-04-19T01:03:55.734940+0000","last_peered":"2024-04-19T01:03:55.734940+0000","last_clean":"2024-04-19T01:03:55.734940+0000","last_became_active":"2024-04-19T01:03:53.771887+0000","last_became_peered":"2024-04-19T01:03:53.771887+0000","last_unstale":"2024-04-19T01:03:55.734940+0000","last_undegraded":"2024-04-19T01:03:55.734940+0000","last_fullsized":"2024-04-19T01:03:55.734940+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:17.479766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-19T01:03:54.659881+0000","last_change":"2024-04-19T01:03:53.716606+0000","last_active":"2024-04-19T01:03:54.659881+0000","last_peered":"2024-04-19T01:03:54.659881+0000","last_clean":"2024-04-19T01:03:54.659881+0000","last_became_active":"2024-04-19T01:03:53.716040+0000","last_became_peered":"2024-04-19T01:03:53.716040+0000","last_unstale":"2024-04-19T01:03:54.659881+0000","last_undegraded":"2024-04-19T01:03:54.659881+0000","last_fullsized":"2024-04-19T01:03:54.659881+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:01.975515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-19T01:03:54.658077+0000","last_change":"2024-04-19T01:03:54.012138+0000","last_active":"2024-04-19T01:03:54.658077+0000","last_peered":"2024-04-19T01:03:54.658077+0000","last_clean":"2024-04-19T01:03:54.658077+0000","last_became_active":"2024-04-19T01:03:54.011460+0000","last_became_peered":"2024-04-19T01:03:54.011460+0000","last_unstale":"2024-04-19T01:03:54.658077+0000","last_undegraded":"2024-04-19T01:03:54.658077+0000","last_fullsized":"2024-04-19T01:03:54.658077+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:04:58.336612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-19T01:03:53.653989+0000","last_change":"2024-04-19T01:03:53.650434+0000","last_active":"2024-04-19T01:03:53.653989+0000","last_peered":"2024-04-19T01:03:53.653989+0000","last_clean":"2024-04-19T01:03:53.653989+0000","last_became_active":"2024-04-19T01:03:53.649606+0000","last_became_peered":"2024-04-19T01:03:53.649606+0000","last_unstale":"2024-04-19T01:03:53.653989+0000","last_undegraded":"2024-04-19T01:03:53.653989+0000","last_fullsized":"2024-04-19T01:03:53.653989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:07.117052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-19T01:03:54.657910+0000","last_change":"2024-04-19T01:03:53.760075+0000","last_active":"2024-04-19T01:03:54.657910+0000","last_peered":"2024-04-19T01:03:54.657910+0000","last_clean":"2024-04-19T01:03:54.657910+0000","last_became_active":"2024-04-19T01:03:53.759326+0000","last_became_peered":"2024-04-19T01:03:53.759326+0000","last_unstale":"2024-04-19T01:03:54.657910+0000","last_undegraded":"2024-04-19T01:03:54.657910+0000","last_fullsized":"2024-04-19T01:03:54.657910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:02.718012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-19T01:03:53.769740+0000","last_change":"2024-04-19T01:03:53.766587+0000","last_active":"2024-04-19T01:03:53.769740+0000","last_peered":"2024-04-19T01:03:53.769740+0000","last_clean":"2024-04-19T01:03:53.769740+0000","last_became_active":"2024-04-19T01:03:53.765967+0000","last_became_peered":"2024-04-19T01:03:53.765967+0000","last_unstale":"2024-04-19T01:03:53.769740+0000","last_undegraded":"2024-04-19T01:03:53.769740+0000","last_fullsized":"2024-04-19T01:03:53.769740+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:04:58.925212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"25'157","reported_seq":212,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-19T01:03:53.645658+0000","last_change":"2024-04-19T01:03:50.776344+0000","last_active":"2024-04-19T01:03:53.645658+0000","last_peered":"2024-04-19T01:03:53.645658+0000","last_clean":"2024-04-19T01:03:53.645658+0000","last_became_active":"2024-04-19T01:03:50.774078+0000","last_became_peered":"2024-04-19T01:03:50.774078+0000","last_unstale":"2024-04-19T01:03:53.645658+0000","last_undegraded":"2024-04-19T01:03:53.645658+0000","last_fullsized":"2024-04-19T01:03:53.645658+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:49.580851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:49.580851+0000","last_clean_scrub_stamp":"2024-04-19T01:03:49.580851+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:07.495976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":13},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":15546,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":14,"up_from":24,"seq":103079215109,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31440,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340400,"statfs":{"total":96636764160,"available":96604569600,"internally_reserved":0,"allocated":266240,"data_stored":115427,"data_compressed":9308,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,7,8,9,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":13,"up_from":24,"seq":103079215110,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31452,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340388,"statfs":{"total":96636764160,"available":96604557312,"internally_reserved":0,"allocated":266240,"data_stored":115427,"data_compressed":9308,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,8,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":24,"seq":103079215111,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31440,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340400,"statfs":{"total":96636764160,"available":96604569600,"internally_reserved":0,"allocated":266240,"data_stored":115427,"data_compressed":9308,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313221,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":105965,"data_compressed":7435,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[3,4,5,6,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":115427,"data_compressed":9308,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,7,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":105965,"data_compressed":7435,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,5,6,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":105965,"data_compressed":7435,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[2,4,5,6,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":105965,"data_compressed":7435,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":307200,"data_stored":634893,"data_compressed":15208,"data_compressed_allocated":49152,"data_compressed_original":544768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,6,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280517,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":115427,"data_compressed":9308,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[3,4,5,6,7,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":307200,"data_stored":634893,"data_compressed":15208,"data_compressed_allocated":49152,"data_compressed_original":544768,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,3,4,5,6,8,9,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":105965,"data_compressed":7435,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,8,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":105965,"data_compressed":7435,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,4,5,6,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":24,"seq":103079215109,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31428,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340412,"statfs":{"total":96636764160,"available":96604581888,"internally_reserved":0,"allocated":253952,"data_stored":105965,"data_compressed":7435,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215110,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":105965,"data_compressed":7435,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,10,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":24,"seq":103079215109,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":115427,"data_compressed":9308,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T01:03:57.737 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T01:03:58.027 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:03:58.027 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-04-19T01:03:58.044 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-04-19T01:03:57.517389+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":13,"kb":1509949440,"kb_used":453244,"kb_used_data":4144,"kb_used_omap":24,"kb_used_meta":448743,"kb_avail":1509496196,"statfs":{"total":1546188226560,"available":1545724104704,"internally_reserved":0,"allocated":4243456,"data_stored":2810068,"data_compressed":145744,"data_compressed_allocated":352256,"data_compressed_original":1597440,"omap_allocated":25520,"internal_metadata":459512912},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.862107"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-19T01:03:54.658079+0000","last_change":"2024-04-19T01:03:53.939441+0000","last_active":"2024-04-19T01:03:54.658079+0000","last_peered":"2024-04-19T01:03:54.658079+0000","last_clean":"2024-04-19T01:03:54.658079+0000","last_became_active":"2024-04-19T01:03:53.938774+0000","last_became_peered":"2024-04-19T01:03:53.938774+0000","last_unstale":"2024-04-19T01:03:54.658079+0000","last_undegraded":"2024-04-19T01:03:54.658079+0000","last_fullsized":"2024-04-19T01:03:54.658079+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:14.081709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-19T01:03:55.308912+0000","last_change":"2024-04-19T01:03:53.736290+0000","last_active":"2024-04-19T01:03:55.308912+0000","last_peered":"2024-04-19T01:03:55.308912+0000","last_clean":"2024-04-19T01:03:55.308912+0000","last_became_active":"2024-04-19T01:03:53.735549+0000","last_became_peered":"2024-04-19T01:03:53.735549+0000","last_unstale":"2024-04-19T01:03:55.308912+0000","last_undegraded":"2024-04-19T01:03:55.308912+0000","last_fullsized":"2024-04-19T01:03:55.308912+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:07.640918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13],"acting":[14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-19T01:03:55.734940+0000","last_change":"2024-04-19T01:03:53.772709+0000","last_active":"2024-04-19T01:03:55.734940+0000","last_peered":"2024-04-19T01:03:55.734940+0000","last_clean":"2024-04-19T01:03:55.734940+0000","last_became_active":"2024-04-19T01:03:53.771887+0000","last_became_peered":"2024-04-19T01:03:53.771887+0000","last_unstale":"2024-04-19T01:03:55.734940+0000","last_undegraded":"2024-04-19T01:03:55.734940+0000","last_fullsized":"2024-04-19T01:03:55.734940+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:17.479766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-19T01:03:54.659881+0000","last_change":"2024-04-19T01:03:53.716606+0000","last_active":"2024-04-19T01:03:54.659881+0000","last_peered":"2024-04-19T01:03:54.659881+0000","last_clean":"2024-04-19T01:03:54.659881+0000","last_became_active":"2024-04-19T01:03:53.716040+0000","last_became_peered":"2024-04-19T01:03:53.716040+0000","last_unstale":"2024-04-19T01:03:54.659881+0000","last_undegraded":"2024-04-19T01:03:54.659881+0000","last_fullsized":"2024-04-19T01:03:54.659881+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:01.975515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-19T01:03:54.658077+0000","last_change":"2024-04-19T01:03:54.012138+0000","last_active":"2024-04-19T01:03:54.658077+0000","last_peered":"2024-04-19T01:03:54.658077+0000","last_clean":"2024-04-19T01:03:54.658077+0000","last_became_active":"2024-04-19T01:03:54.011460+0000","last_became_peered":"2024-04-19T01:03:54.011460+0000","last_unstale":"2024-04-19T01:03:54.658077+0000","last_undegraded":"2024-04-19T01:03:54.658077+0000","last_fullsized":"2024-04-19T01:03:54.658077+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:04:58.336612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-19T01:03:53.653989+0000","last_change":"2024-04-19T01:03:53.650434+0000","last_active":"2024-04-19T01:03:53.653989+0000","last_peered":"2024-04-19T01:03:53.653989+0000","last_clean":"2024-04-19T01:03:53.653989+0000","last_became_active":"2024-04-19T01:03:53.649606+0000","last_became_peered":"2024-04-19T01:03:53.649606+0000","last_unstale":"2024-04-19T01:03:53.653989+0000","last_undegraded":"2024-04-19T01:03:53.653989+0000","last_fullsized":"2024-04-19T01:03:53.653989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:07.117052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-19T01:03:54.657910+0000","last_change":"2024-04-19T01:03:53.760075+0000","last_active":"2024-04-19T01:03:54.657910+0000","last_peered":"2024-04-19T01:03:54.657910+0000","last_clean":"2024-04-19T01:03:54.657910+0000","last_became_active":"2024-04-19T01:03:53.759326+0000","last_became_peered":"2024-04-19T01:03:53.759326+0000","last_unstale":"2024-04-19T01:03:54.657910+0000","last_undegraded":"2024-04-19T01:03:54.657910+0000","last_fullsized":"2024-04-19T01:03:54.657910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:02.718012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-19T01:03:53.769740+0000","last_change":"2024-04-19T01:03:53.766587+0000","last_active":"2024-04-19T01:03:53.769740+0000","last_peered":"2024-04-19T01:03:53.769740+0000","last_clean":"2024-04-19T01:03:53.769740+0000","last_became_active":"2024-04-19T01:03:53.765967+0000","last_became_peered":"2024-04-19T01:03:53.765967+0000","last_unstale":"2024-04-19T01:03:53.769740+0000","last_undegraded":"2024-04-19T01:03:53.769740+0000","last_fullsized":"2024-04-19T01:03:53.769740+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:04:58.925212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"25'157","reported_seq":212,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-19T01:03:53.645658+0000","last_change":"2024-04-19T01:03:50.776344+0000","last_active":"2024-04-19T01:03:53.645658+0000","last_peered":"2024-04-19T01:03:53.645658+0000","last_clean":"2024-04-19T01:03:53.645658+0000","last_became_active":"2024-04-19T01:03:50.774078+0000","last_became_peered":"2024-04-19T01:03:50.774078+0000","last_unstale":"2024-04-19T01:03:53.645658+0000","last_undegraded":"2024-04-19T01:03:53.645658+0000","last_fullsized":"2024-04-19T01:03:53.645658+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:49.580851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:49.580851+0000","last_clean_scrub_stamp":"2024-04-19T01:03:49.580851+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:07.495976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":13},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":15546,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":14,"up_from":24,"seq":103079215109,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31440,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340400,"statfs":{"total":96636764160,"available":96604569600,"internally_reserved":0,"allocated":266240,"data_stored":115427,"data_compressed":9308,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,7,8,9,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":13,"up_from":24,"seq":103079215110,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31452,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340388,"statfs":{"total":96636764160,"available":96604557312,"internally_reserved":0,"allocated":266240,"data_stored":115427,"data_compressed":9308,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,8,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":24,"seq":103079215111,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31440,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340400,"statfs":{"total":96636764160,"available":96604569600,"internally_reserved":0,"allocated":266240,"data_stored":115427,"data_compressed":9308,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313221,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":105965,"data_compressed":7435,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[3,4,5,6,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":115427,"data_compressed":9308,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,7,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":105965,"data_compressed":7435,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,5,6,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":105965,"data_compressed":7435,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[2,4,5,6,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":105965,"data_compressed":7435,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":307200,"data_stored":634893,"data_compressed":15208,"data_compressed_allocated":49152,"data_compressed_original":544768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,6,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280517,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":115427,"data_compressed":9308,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[3,4,5,6,7,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":307200,"data_stored":634893,"data_compressed":15208,"data_compressed_allocated":49152,"data_compressed_original":544768,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,3,4,5,6,8,9,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":105965,"data_compressed":7435,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,8,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":105965,"data_compressed":7435,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,4,5,6,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":24,"seq":103079215109,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31428,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340412,"statfs":{"total":96636764160,"available":96604581888,"internally_reserved":0,"allocated":253952,"data_stored":105965,"data_compressed":7435,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215110,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":105965,"data_compressed":7435,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,10,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":24,"seq":103079215109,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":115427,"data_compressed":9308,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T01:03:58.046 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-19T01:03:58.046 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-04-19T01:03:58.328 INFO:teuthology.orchestra.run.smithi033.stdout:epoch 28 2024-04-19T01:03:58.328 INFO:teuthology.orchestra.run.smithi033.stdout:fsid f11bd2d5-4456-4ffa-83c0-722dddd22ad3 2024-04-19T01:03:58.328 INFO:teuthology.orchestra.run.smithi033.stdout:created 2024-04-19T01:03:36.766089+0000 2024-04-19T01:03:58.329 INFO:teuthology.orchestra.run.smithi033.stdout:modified 2024-04-19T01:03:54.645990+0000 2024-04-19T01:03:58.329 INFO:teuthology.orchestra.run.smithi033.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-19T01:03:58.329 INFO:teuthology.orchestra.run.smithi033.stdout:crush_version 5 2024-04-19T01:03:58.329 INFO:teuthology.orchestra.run.smithi033.stdout:full_ratio 0.95 2024-04-19T01:03:58.329 INFO:teuthology.orchestra.run.smithi033.stdout:backfillfull_ratio 0.9 2024-04-19T01:03:58.329 INFO:teuthology.orchestra.run.smithi033.stdout:nearfull_ratio 0.85 2024-04-19T01:03:58.329 INFO:teuthology.orchestra.run.smithi033.stdout:require_min_compat_client luminous 2024-04-19T01:03:58.329 INFO:teuthology.orchestra.run.smithi033.stdout:min_compat_client jewel 2024-04-19T01:03:58.329 INFO:teuthology.orchestra.run.smithi033.stdout:require_osd_release squid 2024-04-19T01:03:58.329 INFO:teuthology.orchestra.run.smithi033.stdout:stretch_mode_enabled false 2024-04-19T01:03:58.329 INFO:teuthology.orchestra.run.smithi033.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 25 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 15.38 2024-04-19T01:03:58.329 INFO:teuthology.orchestra.run.smithi033.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 28 flags hashpspool stripe_width 0 application rbd read_balance_score 6.02 2024-04-19T01:03:58.329 INFO:teuthology.orchestra.run.smithi033.stdout:max_osd 16 2024-04-19T01:03:58.329 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.33:6824/1633616806,v1:172.21.15.33:6825/1633616806] [v2:172.21.15.33:6826/1633616806,v1:172.21.15.33:6827/1633616806] exists,up 296899fd-2b2b-4071-8b1f-0f92c3d0bbc0 2024-04-19T01:03:58.330 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.110:6805/4270148213,v1:172.21.15.110:6808/4270148213] [v2:172.21.15.110:6811/4270148213,v1:172.21.15.110:6814/4270148213] exists,up 021f7190-0ec3-4553-a44f-15095765cccb 2024-04-19T01:03:58.330 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.118:6808/13248188,v1:172.21.15.118:6809/13248188] [v2:172.21.15.118:6810/13248188,v1:172.21.15.118:6811/13248188] exists,up c283d0bc-5e34-4763-9019-328ee9b6c07a 2024-04-19T01:03:58.330 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.155:6802/2658385904,v1:172.21.15.155:6803/2658385904] [v2:172.21.15.155:6804/2658385904,v1:172.21.15.155:6805/2658385904] exists,up d45888f2-eecd-480b-80f5-c0731a06ff3d 2024-04-19T01:03:58.330 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.33:6800/2389566155,v1:172.21.15.33:6801/2389566155] [v2:172.21.15.33:6802/2389566155,v1:172.21.15.33:6803/2389566155] exists,up d060d939-8c94-4a07-9fd5-42af0044a16f 2024-04-19T01:03:58.330 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.110:6824/2821892844,v1:172.21.15.110:6825/2821892844] [v2:172.21.15.110:6826/2821892844,v1:172.21.15.110:6827/2821892844] exists,up 9700d2f7-ed56-4143-ba9d-6cebe524e877 2024-04-19T01:03:58.330 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.118:6816/273776428,v1:172.21.15.118:6818/273776428] [v2:172.21.15.118:6820/273776428,v1:172.21.15.118:6822/273776428] exists,up 49324695-d83b-41ad-92e1-5348e8e9149f 2024-04-19T01:03:58.330 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.155:6826/1006872173,v1:172.21.15.155:6827/1006872173] [v2:172.21.15.155:6828/1006872173,v1:172.21.15.155:6829/1006872173] exists,up afef160d-0911-4406-ad49-cd6d61ee8a2f 2024-04-19T01:03:58.330 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.33:6816/744651323,v1:172.21.15.33:6817/744651323] [v2:172.21.15.33:6818/744651323,v1:172.21.15.33:6819/744651323] exists,up 298396bf-31a2-4832-91a4-55fcb050130f 2024-04-19T01:03:58.330 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9 up in weight 1 up_from 24 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.110:6800/872519654,v1:172.21.15.110:6802/872519654] [v2:172.21.15.110:6804/872519654,v1:172.21.15.110:6807/872519654] exists,up c151ae6b-279d-4174-aceb-4671fbdac3b3 2024-04-19T01:03:58.330 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.118:6817/4099141917,v1:172.21.15.118:6819/4099141917] [v2:172.21.15.118:6821/4099141917,v1:172.21.15.118:6823/4099141917] exists,up 0b292dfb-2029-4e1e-b206-ea7e0f6969ab 2024-04-19T01:03:58.330 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.155:6808/3981951559,v1:172.21.15.155:6811/3981951559] [v2:172.21.15.155:6814/3981951559,v1:172.21.15.155:6817/3981951559] exists,up 10c61619-ad11-40f0-b013-9ef52bcabf59 2024-04-19T01:03:58.330 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.33:6808/4133738119,v1:172.21.15.33:6809/4133738119] [v2:172.21.15.33:6810/4133738119,v1:172.21.15.33:6811/4133738119] exists,up 3f0d26e5-f3e2-47b6-91a8-0daa30d0544f 2024-04-19T01:03:58.330 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.110:6801/1583232449,v1:172.21.15.110:6803/1583232449] [v2:172.21.15.110:6806/1583232449,v1:172.21.15.110:6809/1583232449] exists,up c8d6213f-af5e-4646-9241-6f91589b0e1b 2024-04-19T01:03:58.330 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14 up in weight 1 up_from 24 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.118:6800/506416463,v1:172.21.15.118:6801/506416463] [v2:172.21.15.118:6802/506416463,v1:172.21.15.118:6803/506416463] exists,up 7b3d56df-62bf-4877-b984-6f17fe1fab33 2024-04-19T01:03:58.331 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.155:6807/741132602,v1:172.21.15.155:6810/741132602] [v2:172.21.15.155:6812/741132602,v1:172.21.15.155:6815/741132602] exists,up 6d526edb-f1d8-415a-ad55-81e810ff4944 2024-04-19T01:03:58.348 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.33:6824/1633616806,v1:172.21.15.33:6825/1633616806] [v2:172.21.15.33:6826/1633616806,v1:172.21.15.33:6827/1633616806] exists,up 296899fd-2b2b-4071-8b1f-0f92c3d0bbc0', 'osd.1 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.110:6805/4270148213,v1:172.21.15.110:6808/4270148213] [v2:172.21.15.110:6811/4270148213,v1:172.21.15.110:6814/4270148213] exists,up 021f7190-0ec3-4553-a44f-15095765cccb', 'osd.2 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.118:6808/13248188,v1:172.21.15.118:6809/13248188] [v2:172.21.15.118:6810/13248188,v1:172.21.15.118:6811/13248188] exists,up c283d0bc-5e34-4763-9019-328ee9b6c07a', 'osd.3 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.155:6802/2658385904,v1:172.21.15.155:6803/2658385904] [v2:172.21.15.155:6804/2658385904,v1:172.21.15.155:6805/2658385904] exists,up d45888f2-eecd-480b-80f5-c0731a06ff3d', 'osd.4 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.33:6800/2389566155,v1:172.21.15.33:6801/2389566155] [v2:172.21.15.33:6802/2389566155,v1:172.21.15.33:6803/2389566155] exists,up d060d939-8c94-4a07-9fd5-42af0044a16f', 'osd.5 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.110:6824/2821892844,v1:172.21.15.110:6825/2821892844] [v2:172.21.15.110:6826/2821892844,v1:172.21.15.110:6827/2821892844] exists,up 9700d2f7-ed56-4143-ba9d-6cebe524e877', 'osd.6 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.118:6816/273776428,v1:172.21.15.118:6818/273776428] [v2:172.21.15.118:6820/273776428,v1:172.21.15.118:6822/273776428] exists,up 49324695-d83b-41ad-92e1-5348e8e9149f', 'osd.7 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.155:6826/1006872173,v1:172.21.15.155:6827/1006872173] [v2:172.21.15.155:6828/1006872173,v1:172.21.15.155:6829/1006872173] exists,up afef160d-0911-4406-ad49-cd6d61ee8a2f', 'osd.8 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.33:6816/744651323,v1:172.21.15.33:6817/744651323] [v2:172.21.15.33:6818/744651323,v1:172.21.15.33:6819/744651323] exists,up 298396bf-31a2-4832-91a4-55fcb050130f', 'osd.9 up in weight 1 up_from 24 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.110:6800/872519654,v1:172.21.15.110:6802/872519654] [v2:172.21.15.110:6804/872519654,v1:172.21.15.110:6807/872519654] exists,up c151ae6b-279d-4174-aceb-4671fbdac3b3', 'osd.10 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.118:6817/4099141917,v1:172.21.15.118:6819/4099141917] [v2:172.21.15.118:6821/4099141917,v1:172.21.15.118:6823/4099141917] exists,up 0b292dfb-2029-4e1e-b206-ea7e0f6969ab', 'osd.11 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.155:6808/3981951559,v1:172.21.15.155:6811/3981951559] [v2:172.21.15.155:6814/3981951559,v1:172.21.15.155:6817/3981951559] exists,up 10c61619-ad11-40f0-b013-9ef52bcabf59', 'osd.12 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.33:6808/4133738119,v1:172.21.15.33:6809/4133738119] [v2:172.21.15.33:6810/4133738119,v1:172.21.15.33:6811/4133738119] exists,up 3f0d26e5-f3e2-47b6-91a8-0daa30d0544f', 'osd.13 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.110:6801/1583232449,v1:172.21.15.110:6803/1583232449] [v2:172.21.15.110:6806/1583232449,v1:172.21.15.110:6809/1583232449] exists,up c8d6213f-af5e-4646-9241-6f91589b0e1b', 'osd.14 up in weight 1 up_from 24 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.118:6800/506416463,v1:172.21.15.118:6801/506416463] [v2:172.21.15.118:6802/506416463,v1:172.21.15.118:6803/506416463] exists,up 7b3d56df-62bf-4877-b984-6f17fe1fab33', 'osd.15 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.155:6807/741132602,v1:172.21.15.155:6810/741132602] [v2:172.21.15.155:6812/741132602,v1:172.21.15.155:6815/741132602] exists,up 6d526edb-f1d8-415a-ad55-81e810ff4944'] 2024-04-19T01:03:58.348 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-04-19T01:03:58.469 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:03:58.470 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "mon.a", 2024-04-19T01:03:58.470 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster": "ceph", 2024-04-19T01:03:58.470 INFO:teuthology.orchestra.run.smithi033.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-04-19T01:03:58.470 INFO:teuthology.orchestra.run.smithi033.stdout: "admin_socket_mode": "", 2024-04-19T01:03:58.470 INFO:teuthology.orchestra.run.smithi033.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-04-19T01:03:58.470 INFO:teuthology.orchestra.run.smithi033.stdout: "auth_client_required": "cephx, none", 2024-04-19T01:03:58.470 INFO:teuthology.orchestra.run.smithi033.stdout: "auth_cluster_required": "cephx", 2024-04-19T01:03:58.470 INFO:teuthology.orchestra.run.smithi033.stdout: "auth_debug": "true", 2024-04-19T01:03:58.470 INFO:teuthology.orchestra.run.smithi033.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-04-19T01:03:58.470 INFO:teuthology.orchestra.run.smithi033.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-04-19T01:03:58.470 INFO:teuthology.orchestra.run.smithi033.stdout: "auth_service_required": "cephx", 2024-04-19T01:03:58.470 INFO:teuthology.orchestra.run.smithi033.stdout: "auth_service_ticket_ttl": "240.000000", 2024-04-19T01:03:58.471 INFO:teuthology.orchestra.run.smithi033.stdout: "auth_supported": "cephx", 2024-04-19T01:03:58.471 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_aio": "true", 2024-04-19T01:03:58.471 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_aio_max_queue_depth": "1024", 2024-04-19T01:03:58.471 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_aio_poll_ms": "250", 2024-04-19T01:03:58.471 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_aio_reap_max": "16", 2024-04-19T01:03:58.471 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_async_discard_threads": "0", 2024-04-19T01:03:58.471 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_block_size": "4096", 2024-04-19T01:03:58.471 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_debug_aio": "false", 2024-04-19T01:03:58.471 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-04-19T01:03:58.471 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-04-19T01:03:58.471 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_debug_inflight_ios": "false", 2024-04-19T01:03:58.471 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_enable_discard": "false", 2024-04-19T01:03:58.471 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_flock_retry": "3", 2024-04-19T01:03:58.471 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_flock_retry_interval": "0.100000", 2024-04-19T01:03:58.472 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_inject_crash": "0", 2024-04-19T01:03:58.472 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_inject_crash_flush_delay": "2", 2024-04-19T01:03:58.472 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_ioring": "false", 2024-04-19T01:03:58.472 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_ioring_hipri": "false", 2024-04-19T01:03:58.472 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_ioring_sqthread_poll": "false", 2024-04-19T01:03:58.472 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-04-19T01:03:58.472 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_read_buffer_alignment": "4096", 2024-04-19T01:03:58.472 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-04-19T01:03:58.472 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_type": "", 2024-04-19T01:03:58.472 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_alloc_size": "1048576", 2024-04-19T01:03:58.472 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_allocator": "hybrid", 2024-04-19T01:03:58.472 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_buffered_io": "true", 2024-04-19T01:03:58.472 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_check_for_zeros": "false", 2024-04-19T01:03:58.472 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_check_volume_selector_often": "false", 2024-04-19T01:03:58.473 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-04-19T01:03:58.474 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_compact_log_sync": "false", 2024-04-19T01:03:58.474 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-04-19T01:03:58.474 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-04-19T01:03:58.474 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_log_compact_min_size": "16777216", 2024-04-19T01:03:58.474 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_log_replay_check_allocations": "true", 2024-04-19T01:03:58.474 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_max_log_runway": "4194304", 2024-04-19T01:03:58.475 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_max_prefetch": "1048576", 2024-04-19T01:03:58.475 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_min_flush_size": "524288", 2024-04-19T01:03:58.475 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_min_log_runway": "1048576", 2024-04-19T01:03:58.475 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_replay_recovery": "false", 2024-04-19T01:03:58.475 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-04-19T01:03:58.475 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_shared_alloc_size": "65536", 2024-04-19T01:03:58.475 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_sync_write": "false", 2024-04-19T01:03:58.475 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-04-19T01:03:58.475 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-04-19T01:03:58.475 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-04-19T01:03:58.475 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_allocation_from_file": "true", 2024-04-19T01:03:58.475 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_allocator": "hybrid", 2024-04-19T01:03:58.475 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-04-19T01:03:58.475 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-04-19T01:03:58.476 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-04-19T01:03:58.476 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-04-19T01:03:58.476 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-04-19T01:03:58.476 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-04-19T01:03:58.476 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_blobid_prealloc": "10240", 2024-04-19T01:03:58.476 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_block_create": "true", 2024-04-19T01:03:58.476 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_block_db_create": "false", 2024-04-19T01:03:58.476 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_block_db_path": "", 2024-04-19T01:03:58.477 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_block_db_size": "0", 2024-04-19T01:03:58.477 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_block_path": "", 2024-04-19T01:03:58.477 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_block_preallocate_file": "false", 2024-04-19T01:03:58.477 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_block_size": "107374182400", 2024-04-19T01:03:58.477 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_block_wal_create": "false", 2024-04-19T01:03:58.477 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_block_wal_path": "", 2024-04-19T01:03:58.478 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_block_wal_size": "100663296", 2024-04-19T01:03:58.478 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_bluefs": "true", 2024-04-19T01:03:58.478 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-04-19T01:03:58.478 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_bluefs_env_mirror": "false", 2024-04-19T01:03:58.478 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-04-19T01:03:58.479 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-04-19T01:03:58.479 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-04-19T01:03:58.479 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-04-19T01:03:58.479 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-04-19T01:03:58.479 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-04-19T01:03:58.479 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_autotune": "true", 2024-04-19T01:03:58.479 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-04-19T01:03:58.480 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-04-19T01:03:58.480 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-04-19T01:03:58.480 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-04-19T01:03:58.480 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_size": "0", 2024-04-19T01:03:58.480 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-04-19T01:03:58.480 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-04-19T01:03:58.480 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-04-19T01:03:58.480 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-04-19T01:03:58.481 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_type": "2q", 2024-04-19T01:03:58.481 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-04-19T01:03:58.481 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_clone_cow": "true", 2024-04-19T01:03:58.481 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_compression_algorithm": "snappy", 2024-04-19T01:03:58.481 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_compression_max_blob_size": "0", 2024-04-19T01:03:58.481 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-04-19T01:03:58.481 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-04-19T01:03:58.481 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_compression_min_blob_size": "0", 2024-04-19T01:03:58.482 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-04-19T01:03:58.482 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-04-19T01:03:58.482 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_compression_mode": "none", 2024-04-19T01:03:58.483 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-04-19T01:03:58.483 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_csum_type": "crc32c", 2024-04-19T01:03:58.483 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-04-19T01:03:58.483 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_enforce_settings": "default", 2024-04-19T01:03:58.483 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_freelist": "false", 2024-04-19T01:03:58.483 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_fsck_abort": "false", 2024-04-19T01:03:58.483 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-04-19T01:03:58.483 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-04-19T01:03:58.483 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_inject_read_err": "false", 2024-04-19T01:03:58.484 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_legacy_omap": "false", 2024-04-19T01:03:58.484 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-04-19T01:03:58.484 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-04-19T01:03:58.484 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-04-19T01:03:58.485 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-04-19T01:03:58.485 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_prefill": "0.000000", 2024-04-19T01:03:58.485 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-04-19T01:03:58.485 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-04-19T01:03:58.485 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-04-19T01:03:58.485 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_small_allocations": "0", 2024-04-19T01:03:58.485 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-04-19T01:03:58.485 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_default_buffered_read": "true", 2024-04-19T01:03:58.485 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_default_buffered_write": "false", 2024-04-19T01:03:58.485 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_deferred_batch_ops": "0", 2024-04-19T01:03:58.485 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-04-19T01:03:58.485 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-04-19T01:03:58.485 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_elastic_shared_blobs": "true", 2024-04-19T01:03:58.485 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-04-19T01:03:58.486 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-04-19T01:03:58.486 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-04-19T01:03:58.486 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-04-19T01:03:58.486 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-04-19T01:03:58.486 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fail_eio": "false", 2024-04-19T01:03:58.486 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-04-19T01:03:58.486 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-04-19T01:03:58.486 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-04-19T01:03:58.486 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-04-19T01:03:58.486 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_on_mkfs": "true", 2024-04-19T01:03:58.486 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-04-19T01:03:58.486 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_on_mount": "false", 2024-04-19T01:03:58.486 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-04-19T01:03:58.487 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_on_umount": "false", 2024-04-19T01:03:58.487 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-04-19T01:03:58.487 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-04-19T01:03:58.487 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-04-19T01:03:58.487 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-04-19T01:03:58.487 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-04-19T01:03:58.488 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-04-19T01:03:58.488 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-04-19T01:03:58.488 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-04-19T01:03:58.488 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_ignore_data_csum": "false", 2024-04-19T01:03:58.488 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-04-19T01:03:58.488 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_kvbackend": "rocksdb", 2024-04-19T01:03:58.488 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-04-19T01:03:58.488 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-04-19T01:03:58.488 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_log_op_age": "5.000000", 2024-04-19T01:03:58.488 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_max_alloc_size": "0", 2024-04-19T01:03:58.488 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_max_blob_size": "0", 2024-04-19T01:03:58.488 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-04-19T01:03:58.489 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-04-19T01:03:58.489 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_max_defer_interval": "3.000000", 2024-04-19T01:03:58.489 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_max_deferred_txc": "32", 2024-04-19T01:03:58.489 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_min_alloc_size": "0", 2024-04-19T01:03:58.490 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-04-19T01:03:58.490 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-04-19T01:03:58.490 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_nid_prealloc": "1024", 2024-04-19T01:03:58.490 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_prefer_deferred_size": "0", 2024-04-19T01:03:58.490 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-04-19T01:03:58.490 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-04-19T01:03:58.491 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_qfsck_on_mount": "true", 2024-04-19T01:03:58.491 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_retry_disk_reads": "3", 2024-04-19T01:03:58.491 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_rocksdb_cf": "true", 2024-04-19T01:03:58.491 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-04-19T01:03:58.491 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-04-19T01:03:58.491 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_rocksdb_options_annex": "", 2024-04-19T01:03:58.491 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_spdk_coremask": "0x1", 2024-04-19T01:03:58.491 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_spdk_io_sleep": "5", 2024-04-19T01:03:58.491 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_spdk_max_io_completion": "0", 2024-04-19T01:03:58.491 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_spdk_mem": "512", 2024-04-19T01:03:58.491 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_sync_submit_transaction": "false", 2024-04-19T01:03:58.491 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_throttle_bytes": "67108864", 2024-04-19T01:03:58.492 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_throttle_cost_per_io": "0", 2024-04-19T01:03:58.492 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-04-19T01:03:58.492 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-04-19T01:03:58.492 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-04-19T01:03:58.492 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-04-19T01:03:58.492 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_tracing": "false", 2024-04-19T01:03:58.493 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-04-19T01:03:58.493 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-04-19T01:03:58.493 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_volume_selection_reserved": "0", 2024-04-19T01:03:58.493 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-04-19T01:03:58.493 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-04-19T01:03:58.493 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-04-19T01:03:58.493 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-04-19T01:03:58.493 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-04-19T01:03:58.493 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-04-19T01:03:58.493 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_zero_block_detection": "false", 2024-04-19T01:03:58.493 INFO:teuthology.orchestra.run.smithi033.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-04-19T01:03:58.493 INFO:teuthology.orchestra.run.smithi033.stdout: "cephfs_mirror_action_update_interval": "2", 2024-04-19T01:03:58.493 INFO:teuthology.orchestra.run.smithi033.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-04-19T01:03:58.494 INFO:teuthology.orchestra.run.smithi033.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-04-19T01:03:58.494 INFO:teuthology.orchestra.run.smithi033.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-04-19T01:03:58.494 INFO:teuthology.orchestra.run.smithi033.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-04-19T01:03:58.494 INFO:teuthology.orchestra.run.smithi033.stdout: "cephfs_mirror_mount_timeout": "10", 2024-04-19T01:03:58.494 INFO:teuthology.orchestra.run.smithi033.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-04-19T01:03:58.494 INFO:teuthology.orchestra.run.smithi033.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-04-19T01:03:58.494 INFO:teuthology.orchestra.run.smithi033.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-04-19T01:03:58.494 INFO:teuthology.orchestra.run.smithi033.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-04-19T01:03:58.494 INFO:teuthology.orchestra.run.smithi033.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-04-19T01:03:58.494 INFO:teuthology.orchestra.run.smithi033.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-04-19T01:03:58.494 INFO:teuthology.orchestra.run.smithi033.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-04-19T01:03:58.494 INFO:teuthology.orchestra.run.smithi033.stdout: "cephx_cluster_require_signatures": "false", 2024-04-19T01:03:58.495 INFO:teuthology.orchestra.run.smithi033.stdout: "cephx_cluster_require_version": "2", 2024-04-19T01:03:58.495 INFO:teuthology.orchestra.run.smithi033.stdout: "cephx_require_signatures": "false", 2024-04-19T01:03:58.495 INFO:teuthology.orchestra.run.smithi033.stdout: "cephx_require_version": "2", 2024-04-19T01:03:58.495 INFO:teuthology.orchestra.run.smithi033.stdout: "cephx_service_require_signatures": "false", 2024-04-19T01:03:58.495 INFO:teuthology.orchestra.run.smithi033.stdout: "cephx_service_require_version": "2", 2024-04-19T01:03:58.495 INFO:teuthology.orchestra.run.smithi033.stdout: "cephx_sign_messages": "true", 2024-04-19T01:03:58.495 INFO:teuthology.orchestra.run.smithi033.stdout: "chdir": "", 2024-04-19T01:03:58.495 INFO:teuthology.orchestra.run.smithi033.stdout: "client_acl_type": "", 2024-04-19T01:03:58.495 INFO:teuthology.orchestra.run.smithi033.stdout: "client_asio_thread_count": "2", 2024-04-19T01:03:58.496 INFO:teuthology.orchestra.run.smithi033.stdout: "client_cache_mid": "0.750000", 2024-04-19T01:03:58.496 INFO:teuthology.orchestra.run.smithi033.stdout: "client_cache_size": "16384", 2024-04-19T01:03:58.496 INFO:teuthology.orchestra.run.smithi033.stdout: "client_caps_release_delay": "5", 2024-04-19T01:03:58.496 INFO:teuthology.orchestra.run.smithi033.stdout: "client_check_pool_perm": "true", 2024-04-19T01:03:58.496 INFO:teuthology.orchestra.run.smithi033.stdout: "client_collect_and_send_global_metrics": "false", 2024-04-19T01:03:58.496 INFO:teuthology.orchestra.run.smithi033.stdout: "client_debug_force_sync_read": "false", 2024-04-19T01:03:58.496 INFO:teuthology.orchestra.run.smithi033.stdout: "client_debug_getattr_caps": "false", 2024-04-19T01:03:58.496 INFO:teuthology.orchestra.run.smithi033.stdout: "client_debug_inject_tick_delay": "0", 2024-04-19T01:03:58.496 INFO:teuthology.orchestra.run.smithi033.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-04-19T01:03:58.496 INFO:teuthology.orchestra.run.smithi033.stdout: "client_die_on_failed_remount": "false", 2024-04-19T01:03:58.496 INFO:teuthology.orchestra.run.smithi033.stdout: "client_dirsize_rbytes": "true", 2024-04-19T01:03:58.496 INFO:teuthology.orchestra.run.smithi033.stdout: "client_force_lazyio": "false", 2024-04-19T01:03:58.496 INFO:teuthology.orchestra.run.smithi033.stdout: "client_fs": "", 2024-04-19T01:03:58.496 INFO:teuthology.orchestra.run.smithi033.stdout: "client_inject_fixed_oldest_tid": "false", 2024-04-19T01:03:58.497 INFO:teuthology.orchestra.run.smithi033.stdout: "client_inject_release_failure": "false", 2024-04-19T01:03:58.497 INFO:teuthology.orchestra.run.smithi033.stdout: "client_max_inline_size": "4096", 2024-04-19T01:03:58.497 INFO:teuthology.orchestra.run.smithi033.stdout: "client_max_retries_on_remount_failure": "5", 2024-04-19T01:03:58.497 INFO:teuthology.orchestra.run.smithi033.stdout: "client_mds_namespace": "", 2024-04-19T01:03:58.497 INFO:teuthology.orchestra.run.smithi033.stdout: "client_metadata": "", 2024-04-19T01:03:58.497 INFO:teuthology.orchestra.run.smithi033.stdout: "client_mount_gid": "-1", 2024-04-19T01:03:58.498 INFO:teuthology.orchestra.run.smithi033.stdout: "client_mount_timeout": "300", 2024-04-19T01:03:58.498 INFO:teuthology.orchestra.run.smithi033.stdout: "client_mount_uid": "-1", 2024-04-19T01:03:58.498 INFO:teuthology.orchestra.run.smithi033.stdout: "client_mountpoint": "/", 2024-04-19T01:03:58.498 INFO:teuthology.orchestra.run.smithi033.stdout: "client_notify_timeout": "10", 2024-04-19T01:03:58.498 INFO:teuthology.orchestra.run.smithi033.stdout: "client_oc": "true", 2024-04-19T01:03:58.499 INFO:teuthology.orchestra.run.smithi033.stdout: "client_oc_max_dirty": "104857600", 2024-04-19T01:03:58.499 INFO:teuthology.orchestra.run.smithi033.stdout: "client_oc_max_dirty_age": "5.000000", 2024-04-19T01:03:58.499 INFO:teuthology.orchestra.run.smithi033.stdout: "client_oc_max_objects": "1000", 2024-04-19T01:03:58.499 INFO:teuthology.orchestra.run.smithi033.stdout: "client_oc_size": "209715200", 2024-04-19T01:03:58.499 INFO:teuthology.orchestra.run.smithi033.stdout: "client_oc_target_dirty": "8388608", 2024-04-19T01:03:58.499 INFO:teuthology.orchestra.run.smithi033.stdout: "client_permissions": "true", 2024-04-19T01:03:58.499 INFO:teuthology.orchestra.run.smithi033.stdout: "client_quota": "true", 2024-04-19T01:03:58.499 INFO:teuthology.orchestra.run.smithi033.stdout: "client_quota_df": "true", 2024-04-19T01:03:58.499 INFO:teuthology.orchestra.run.smithi033.stdout: "client_readahead_max_bytes": "0", 2024-04-19T01:03:58.499 INFO:teuthology.orchestra.run.smithi033.stdout: "client_readahead_max_periods": "4", 2024-04-19T01:03:58.499 INFO:teuthology.orchestra.run.smithi033.stdout: "client_readahead_min": "131072", 2024-04-19T01:03:58.499 INFO:teuthology.orchestra.run.smithi033.stdout: "client_reconnect_stale": "false", 2024-04-19T01:03:58.499 INFO:teuthology.orchestra.run.smithi033.stdout: "client_shutdown_timeout": "30", 2024-04-19T01:03:58.499 INFO:teuthology.orchestra.run.smithi033.stdout: "client_snapdir": ".snap", 2024-04-19T01:03:58.500 INFO:teuthology.orchestra.run.smithi033.stdout: "client_tick_interval": "1", 2024-04-19T01:03:58.500 INFO:teuthology.orchestra.run.smithi033.stdout: "client_trace": "", 2024-04-19T01:03:58.500 INFO:teuthology.orchestra.run.smithi033.stdout: "client_try_dentry_invalidate": "false", 2024-04-19T01:03:58.500 INFO:teuthology.orchestra.run.smithi033.stdout: "client_use_faked_inos": "false", 2024-04-19T01:03:58.500 INFO:teuthology.orchestra.run.smithi033.stdout: "client_use_random_mds": "false", 2024-04-19T01:03:58.500 INFO:teuthology.orchestra.run.smithi033.stdout: "clog_to_graylog": "false", 2024-04-19T01:03:58.500 INFO:teuthology.orchestra.run.smithi033.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-04-19T01:03:58.500 INFO:teuthology.orchestra.run.smithi033.stdout: "clog_to_graylog_port": "12201", 2024-04-19T01:03:58.500 INFO:teuthology.orchestra.run.smithi033.stdout: "clog_to_monitors": "default=true", 2024-04-19T01:03:58.501 INFO:teuthology.orchestra.run.smithi033.stdout: "clog_to_syslog": "false", 2024-04-19T01:03:58.501 INFO:teuthology.orchestra.run.smithi033.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-04-19T01:03:58.501 INFO:teuthology.orchestra.run.smithi033.stdout: "clog_to_syslog_level": "info", 2024-04-19T01:03:58.501 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "-", 2024-04-19T01:03:58.501 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_network": "", 2024-04-19T01:03:58.501 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_network_interface": "", 2024-04-19T01:03:58.501 INFO:teuthology.orchestra.run.smithi033.stdout: "compressor_zlib_isal": "false", 2024-04-19T01:03:58.501 INFO:teuthology.orchestra.run.smithi033.stdout: "compressor_zlib_level": "5", 2024-04-19T01:03:58.501 INFO:teuthology.orchestra.run.smithi033.stdout: "compressor_zlib_winsize": "-15", 2024-04-19T01:03:58.502 INFO:teuthology.orchestra.run.smithi033.stdout: "compressor_zstd_level": "1", 2024-04-19T01:03:58.502 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-04-19T01:03:58.502 INFO:teuthology.orchestra.run.smithi033.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-04-19T01:03:58.502 INFO:teuthology.orchestra.run.smithi033.stdout: "crimson_alien_op_num_threads": "6", 2024-04-19T01:03:58.502 INFO:teuthology.orchestra.run.smithi033.stdout: "crimson_alien_thread_cpu_cores": "", 2024-04-19T01:03:58.502 INFO:teuthology.orchestra.run.smithi033.stdout: "crimson_osd_obc_lru_size": "512", 2024-04-19T01:03:58.502 INFO:teuthology.orchestra.run.smithi033.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-04-19T01:03:58.502 INFO:teuthology.orchestra.run.smithi033.stdout: "crimson_osd_stat_interval": "0", 2024-04-19T01:03:58.502 INFO:teuthology.orchestra.run.smithi033.stdout: "crimson_seastar_cpu_cores": "", 2024-04-19T01:03:58.502 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_location": "", 2024-04-19T01:03:58.502 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_location_hook": "", 2024-04-19T01:03:58.502 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_location_hook_timeout": "10", 2024-04-19T01:03:58.502 INFO:teuthology.orchestra.run.smithi033.stdout: "daemonize": "false", 2024-04-19T01:03:58.502 INFO:teuthology.orchestra.run.smithi033.stdout: "daos_pool": "tank", 2024-04-19T01:03:58.503 INFO:teuthology.orchestra.run.smithi033.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-04-19T01:03:58.503 INFO:teuthology.orchestra.run.smithi033.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-04-19T01:03:58.503 INFO:teuthology.orchestra.run.smithi033.stdout: "dbstore_db_name_prefix": "dbstore", 2024-04-19T01:03:58.503 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_alienstore": "0/5", 2024-04-19T01:03:58.503 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_allow_any_pool_priority": "false", 2024-04-19T01:03:58.503 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_asok": "1/5", 2024-04-19T01:03:58.503 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_asok_assert_abort": "false", 2024-04-19T01:03:58.503 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_asserts_on_shutdown": "true", 2024-04-19T01:03:58.503 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_auth": "20/20", 2024-04-19T01:03:58.503 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_bdev": "1/3", 2024-04-19T01:03:58.503 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_bluefs": "1/5", 2024-04-19T01:03:58.503 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_bluestore": "1/5", 2024-04-19T01:03:58.503 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_buffer": "0/1", 2024-04-19T01:03:58.504 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_ceph_exporter": "1/5", 2024-04-19T01:03:58.504 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_cephfs_mirror": "0/5", 2024-04-19T01:03:58.504 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_cephsqlite": "0/5", 2024-04-19T01:03:58.504 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_client": "0/5", 2024-04-19T01:03:58.504 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_compressor": "1/5", 2024-04-19T01:03:58.504 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_context": "0/1", 2024-04-19T01:03:58.504 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_crimson_interrupt": "0/5", 2024-04-19T01:03:58.504 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_crush": "1/1", 2024-04-19T01:03:58.504 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_crypto": "1/5", 2024-04-19T01:03:58.504 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_cyanstore": "0/5", 2024-04-19T01:03:58.504 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_deliberately_leak_memory": "false", 2024-04-19T01:03:58.504 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_disable_randomized_ping": "false", 2024-04-19T01:03:58.504 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_dpdk": "1/5", 2024-04-19T01:03:58.504 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_eventtrace": "1/5", 2024-04-19T01:03:58.505 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_filer": "0/1", 2024-04-19T01:03:58.505 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_filestore": "1/3", 2024-04-19T01:03:58.505 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_finisher": "1/1", 2024-04-19T01:03:58.505 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_fuse": "1/5", 2024-04-19T01:03:58.505 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_heartbeat_testing_span": "0", 2024-04-19T01:03:58.505 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_heartbeatmap": "1/5", 2024-04-19T01:03:58.505 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_immutable_obj_cache": "0/5", 2024-04-19T01:03:58.506 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_javaclient": "1/5", 2024-04-19T01:03:58.506 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_journal": "1/3", 2024-04-19T01:03:58.506 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_journaler": "0/5", 2024-04-19T01:03:58.506 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_kstore": "1/5", 2024-04-19T01:03:58.506 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_lockdep": "0/1", 2024-04-19T01:03:58.506 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_mclock": "1/5", 2024-04-19T01:03:58.506 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_mds": "1/5", 2024-04-19T01:03:58.506 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_mds_balancer": "1/5", 2024-04-19T01:03:58.506 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_mds_locker": "1/5", 2024-04-19T01:03:58.506 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_mds_log": "1/5", 2024-04-19T01:03:58.506 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_mds_log_expire": "1/5", 2024-04-19T01:03:58.506 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_mds_migrator": "1/5", 2024-04-19T01:03:58.506 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_mds_quiesce": "3/5", 2024-04-19T01:03:58.506 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_memstore": "1/5", 2024-04-19T01:03:58.507 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_mgr": "2/5", 2024-04-19T01:03:58.507 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_mgrc": "1/5", 2024-04-19T01:03:58.507 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_mon": "20/20", 2024-04-19T01:03:58.507 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_monc": "0/10", 2024-04-19T01:03:58.507 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_ms": "1/1", 2024-04-19T01:03:58.507 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_none": "0/5", 2024-04-19T01:03:58.507 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_objclass": "0/5", 2024-04-19T01:03:58.507 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_objectcacher": "0/5", 2024-04-19T01:03:58.507 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_objecter": "0/1", 2024-04-19T01:03:58.507 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_optracker": "0/5", 2024-04-19T01:03:58.507 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_osd": "1/5", 2024-04-19T01:03:58.507 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_paxos": "20/20", 2024-04-19T01:03:58.509 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_perfcounter": "1/5", 2024-04-19T01:03:58.509 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_prioritycache": "1/5", 2024-04-19T01:03:58.509 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_rados": "0/5", 2024-04-19T01:03:58.509 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_rbd": "0/5", 2024-04-19T01:03:58.509 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_rbd_mirror": "0/5", 2024-04-19T01:03:58.509 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_rbd_pwl": "0/5", 2024-04-19T01:03:58.509 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_rbd_replay": "0/5", 2024-04-19T01:03:58.509 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_refs": "0/0", 2024-04-19T01:03:58.509 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_reserver": "1/1", 2024-04-19T01:03:58.509 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_rgw": "1/5", 2024-04-19T01:03:58.509 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_rgw_access": "1/5", 2024-04-19T01:03:58.509 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_rgw_datacache": "1/5", 2024-04-19T01:03:58.509 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_rgw_dbstore": "1/5", 2024-04-19T01:03:58.510 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_rgw_flight": "1/5", 2024-04-19T01:03:58.510 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_rgw_lifecycle": "1/5", 2024-04-19T01:03:58.510 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_rgw_notification": "1/5", 2024-04-19T01:03:58.510 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_rgw_sync": "1/5", 2024-04-19T01:03:58.510 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_rocksdb": "4/5", 2024-04-19T01:03:58.510 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_seastore": "0/5", 2024-04-19T01:03:58.510 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_seastore_backref": "0/5", 2024-04-19T01:03:58.510 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_seastore_cache": "0/5", 2024-04-19T01:03:58.510 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_seastore_cleaner": "0/5", 2024-04-19T01:03:58.510 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_seastore_device": "0/5", 2024-04-19T01:03:58.510 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_seastore_epm": "0/5", 2024-04-19T01:03:58.510 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-04-19T01:03:58.510 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_seastore_journal": "0/5", 2024-04-19T01:03:58.511 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_seastore_lba": "0/5", 2024-04-19T01:03:58.511 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_seastore_odata": "0/5", 2024-04-19T01:03:58.511 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_seastore_omap": "0/5", 2024-04-19T01:03:58.511 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_seastore_onode": "0/5", 2024-04-19T01:03:58.511 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_seastore_t": "0/5", 2024-04-19T01:03:58.511 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_seastore_tm": "0/5", 2024-04-19T01:03:58.511 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_striper": "0/1", 2024-04-19T01:03:58.512 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_test": "0/5", 2024-04-19T01:03:58.512 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_throttle": "1/1", 2024-04-19T01:03:58.512 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_timer": "0/1", 2024-04-19T01:03:58.512 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_tp": "0/5", 2024-04-19T01:03:58.512 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_trace": "1/5", 2024-04-19T01:03:58.512 INFO:teuthology.orchestra.run.smithi033.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-04-19T01:03:58.512 INFO:teuthology.orchestra.run.smithi033.stdout: "device_failure_prediction_mode": "none", 2024-04-19T01:03:58.512 INFO:teuthology.orchestra.run.smithi033.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-04-19T01:03:58.512 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_dir": "/usr/lib/ceph/erasure-code", 2024-04-19T01:03:58.512 INFO:teuthology.orchestra.run.smithi033.stdout: "err_to_graylog": "false", 2024-04-19T01:03:58.512 INFO:teuthology.orchestra.run.smithi033.stdout: "err_to_journald": "false", 2024-04-19T01:03:58.512 INFO:teuthology.orchestra.run.smithi033.stdout: "err_to_stderr": "true", 2024-04-19T01:03:58.513 INFO:teuthology.orchestra.run.smithi033.stdout: "err_to_syslog": "false", 2024-04-19T01:03:58.513 INFO:teuthology.orchestra.run.smithi033.stdout: "event_tracing": "false", 2024-04-19T01:03:58.513 INFO:teuthology.orchestra.run.smithi033.stdout: "exporter_addr": "0.0.0.0", 2024-04-19T01:03:58.513 INFO:teuthology.orchestra.run.smithi033.stdout: "exporter_http_port": "9926", 2024-04-19T01:03:58.513 INFO:teuthology.orchestra.run.smithi033.stdout: "exporter_prio_limit": "5", 2024-04-19T01:03:58.513 INFO:teuthology.orchestra.run.smithi033.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-04-19T01:03:58.513 INFO:teuthology.orchestra.run.smithi033.stdout: "exporter_sort_metrics": "true", 2024-04-19T01:03:58.513 INFO:teuthology.orchestra.run.smithi033.stdout: "exporter_stats_period": "5", 2024-04-19T01:03:58.513 INFO:teuthology.orchestra.run.smithi033.stdout: "fake_statfs_for_testing": "0", 2024-04-19T01:03:58.513 INFO:teuthology.orchestra.run.smithi033.stdout: "fatal_signal_handlers": "true", 2024-04-19T01:03:58.513 INFO:teuthology.orchestra.run.smithi033.stdout: "filer_max_purge_ops": "10", 2024-04-19T01:03:58.513 INFO:teuthology.orchestra.run.smithi033.stdout: "filer_max_truncate_ops": "128", 2024-04-19T01:03:58.513 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_apply_finisher_threads": "1", 2024-04-19T01:03:58.513 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_blackhole": "false", 2024-04-19T01:03:58.514 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_btrfs_clone_range": "true", 2024-04-19T01:03:58.514 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_btrfs_snap": "true", 2024-04-19T01:03:58.514 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_caller_concurrency": "10", 2024-04-19T01:03:58.514 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_collect_device_partition_information": "true", 2024-04-19T01:03:58.514 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_commit_timeout": "600.000000", 2024-04-19T01:03:58.514 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_debug_inject_read_err": "false", 2024-04-19T01:03:58.514 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_debug_omap_check": "false", 2024-04-19T01:03:58.514 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_debug_verify_split": "false", 2024-04-19T01:03:58.514 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_dump_file": "", 2024-04-19T01:03:58.514 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-04-19T01:03:58.514 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-04-19T01:03:58.514 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_fadvise": "true", 2024-04-19T01:03:58.514 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_fail_eio": "true", 2024-04-19T01:03:58.514 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_fd_cache_shards": "16", 2024-04-19T01:03:58.515 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_fd_cache_size": "128", 2024-04-19T01:03:58.515 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_fiemap": "false", 2024-04-19T01:03:58.515 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_fiemap_threshold": "4096", 2024-04-19T01:03:58.515 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-04-19T01:03:58.515 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_index_retry_probability": "0.000000", 2024-04-19T01:03:58.515 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_inject_stall": "0", 2024-04-19T01:03:58.515 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_journal_parallel": "false", 2024-04-19T01:03:58.515 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_journal_trailing": "false", 2024-04-19T01:03:58.515 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_journal_writeahead": "false", 2024-04-19T01:03:58.515 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_kill_at": "0", 2024-04-19T01:03:58.515 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-04-19T01:03:58.516 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_inline_xattr_size": "0", 2024-04-19T01:03:58.516 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-04-19T01:03:58.516 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-04-19T01:03:58.516 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-04-19T01:03:58.516 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_inline_xattrs": "0", 2024-04-19T01:03:58.516 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-04-19T01:03:58.516 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_inline_xattrs_other": "2", 2024-04-19T01:03:58.516 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-04-19T01:03:58.516 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_sync_interval": "5.000000", 2024-04-19T01:03:58.516 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_xattr_value_size": "0", 2024-04-19T01:03:58.517 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-04-19T01:03:58.517 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-04-19T01:03:58.517 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-04-19T01:03:58.517 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_merge_threshold": "-10", 2024-04-19T01:03:58.517 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_min_sync_interval": "0.010000", 2024-04-19T01:03:58.517 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_odsync_write": "false", 2024-04-19T01:03:58.517 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_omap_backend": "rocksdb", 2024-04-19T01:03:58.517 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_omap_backend_path": "", 2024-04-19T01:03:58.517 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_omap_header_cache_size": "1024", 2024-04-19T01:03:58.517 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_ondisk_finisher_threads": "1", 2024-04-19T01:03:58.517 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-04-19T01:03:58.517 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_op_thread_timeout": "60", 2024-04-19T01:03:58.517 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_op_threads": "2", 2024-04-19T01:03:58.518 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_punch_hole": "false", 2024-04-19T01:03:58.518 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-04-19T01:03:58.518 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-04-19T01:03:58.518 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-04-19T01:03:58.519 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-04-19T01:03:58.519 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-04-19T01:03:58.519 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_queue_max_bytes": "104857600", 2024-04-19T01:03:58.519 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-04-19T01:03:58.519 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-04-19T01:03:58.519 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-04-19T01:03:58.519 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_queue_max_ops": "50", 2024-04-19T01:03:58.519 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-04-19T01:03:58.520 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_seek_data_hole": "false", 2024-04-19T01:03:58.520 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_sloppy_crc": "false", 2024-04-19T01:03:58.520 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-04-19T01:03:58.520 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_splice": "false", 2024-04-19T01:03:58.520 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_split_multiple": "2", 2024-04-19T01:03:58.520 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_split_rand_factor": "20", 2024-04-19T01:03:58.520 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_update_to": "1000", 2024-04-19T01:03:58.520 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-04-19T01:03:58.520 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-04-19T01:03:58.520 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-04-19T01:03:58.520 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-04-19T01:03:58.520 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-04-19T01:03:58.520 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-04-19T01:03:58.520 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_wbthrottle_enable": "true", 2024-04-19T01:03:58.521 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-04-19T01:03:58.521 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-04-19T01:03:58.521 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-04-19T01:03:58.521 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-04-19T01:03:58.521 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-04-19T01:03:58.521 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-04-19T01:03:58.521 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_xfs_extsize": "false", 2024-04-19T01:03:58.521 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_zfs_snap": "false", 2024-04-19T01:03:58.521 INFO:teuthology.orchestra.run.smithi033.stdout: "fio_dir": "/tmp/fio", 2024-04-19T01:03:58.522 INFO:teuthology.orchestra.run.smithi033.stdout: "fsid": "f11bd2d5-4456-4ffa-83c0-722dddd22ad3", 2024-04-19T01:03:58.522 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_allow_other": "true", 2024-04-19T01:03:58.522 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_atomic_o_trunc": "true", 2024-04-19T01:03:58.522 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_big_writes": "true", 2024-04-19T01:03:58.522 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_debug": "false", 2024-04-19T01:03:58.522 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_default_permissions": "false", 2024-04-19T01:03:58.522 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_disable_pagecache": "false", 2024-04-19T01:03:58.522 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_max_write": "0", 2024-04-19T01:03:58.522 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_multithreaded": "true", 2024-04-19T01:03:58.522 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_require_active_mds": "true", 2024-04-19T01:03:58.522 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_set_user_groups": "true", 2024-04-19T01:03:58.523 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_splice_move": "true", 2024-04-19T01:03:58.523 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_splice_read": "true", 2024-04-19T01:03:58.523 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_splice_write": "true", 2024-04-19T01:03:58.523 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_syncfs_on_mksnap": "true", 2024-04-19T01:03:58.523 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_use_invalidate_cb": "true", 2024-04-19T01:03:58.523 INFO:teuthology.orchestra.run.smithi033.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-04-19T01:03:58.523 INFO:teuthology.orchestra.run.smithi033.stdout: "gss_target_name": "ceph", 2024-04-19T01:03:58.523 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_file": "", 2024-04-19T01:03:58.523 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_inject_failure": "0", 2024-04-19T01:03:58.523 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_interval": "5", 2024-04-19T01:03:58.523 INFO:teuthology.orchestra.run.smithi033.stdout: "host": "smithi033", 2024-04-19T01:03:58.523 INFO:teuthology.orchestra.run.smithi033.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-04-19T01:03:58.523 INFO:teuthology.orchestra.run.smithi033.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-04-19T01:03:58.523 INFO:teuthology.orchestra.run.smithi033.stdout: "immutable_object_cache_max_size": "1073741824", 2024-04-19T01:03:58.524 INFO:teuthology.orchestra.run.smithi033.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-04-19T01:03:58.524 INFO:teuthology.orchestra.run.smithi033.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-04-19T01:03:58.524 INFO:teuthology.orchestra.run.smithi033.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-04-19T01:03:58.524 INFO:teuthology.orchestra.run.smithi033.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-04-19T01:03:58.524 INFO:teuthology.orchestra.run.smithi033.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-04-19T01:03:58.524 INFO:teuthology.orchestra.run.smithi033.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-04-19T01:03:58.524 INFO:teuthology.orchestra.run.smithi033.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-04-19T01:03:58.524 INFO:teuthology.orchestra.run.smithi033.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-04-19T01:03:58.524 INFO:teuthology.orchestra.run.smithi033.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-04-19T01:03:58.524 INFO:teuthology.orchestra.run.smithi033.stdout: "immutable_object_cache_watermark": "0.900000", 2024-04-19T01:03:58.524 INFO:teuthology.orchestra.run.smithi033.stdout: "inject_early_sigterm": "false", 2024-04-19T01:03:58.524 INFO:teuthology.orchestra.run.smithi033.stdout: "jaeger_agent_port": "6799", 2024-04-19T01:03:58.524 INFO:teuthology.orchestra.run.smithi033.stdout: "jaeger_tracing_enable": "false", 2024-04-19T01:03:58.524 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_aio": "true", 2024-04-19T01:03:58.525 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_align_min_size": "65536", 2024-04-19T01:03:58.525 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_block_align": "true", 2024-04-19T01:03:58.525 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_block_size": "4096", 2024-04-19T01:03:58.525 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_dio": "true", 2024-04-19T01:03:58.525 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_discard": "false", 2024-04-19T01:03:58.525 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_force_aio": "false", 2024-04-19T01:03:58.525 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_ignore_corruption": "false", 2024-04-19T01:03:58.525 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_max_write_bytes": "10485760", 2024-04-19T01:03:58.525 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_max_write_entries": "100", 2024-04-19T01:03:58.525 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_replay_from": "0", 2024-04-19T01:03:58.525 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_throttle_high_multiple": "0.000000", 2024-04-19T01:03:58.525 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-04-19T01:03:58.525 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-04-19T01:03:58.525 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_throttle_max_multiple": "0.000000", 2024-04-19T01:03:58.526 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_write_header_frequency": "0", 2024-04-19T01:03:58.526 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_zero_on_create": "false", 2024-04-19T01:03:58.526 INFO:teuthology.orchestra.run.smithi033.stdout: "journaler_prefetch_periods": "10", 2024-04-19T01:03:58.526 INFO:teuthology.orchestra.run.smithi033.stdout: "journaler_prezero_periods": "5", 2024-04-19T01:03:58.526 INFO:teuthology.orchestra.run.smithi033.stdout: "journaler_write_head_interval": "15", 2024-04-19T01:03:58.526 INFO:teuthology.orchestra.run.smithi033.stdout: "key": "", 2024-04-19T01:03:58.527 INFO:teuthology.orchestra.run.smithi033.stdout: "keyfile": "", 2024-04-19T01:03:58.527 INFO:teuthology.orchestra.run.smithi033.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-04-19T01:03:58.527 INFO:teuthology.orchestra.run.smithi033.stdout: "kstore_backend": "rocksdb", 2024-04-19T01:03:58.527 INFO:teuthology.orchestra.run.smithi033.stdout: "kstore_default_stripe_size": "65536", 2024-04-19T01:03:58.527 INFO:teuthology.orchestra.run.smithi033.stdout: "kstore_fsck_on_mount": "false", 2024-04-19T01:03:58.527 INFO:teuthology.orchestra.run.smithi033.stdout: "kstore_fsck_on_mount_deep": "true", 2024-04-19T01:03:58.527 INFO:teuthology.orchestra.run.smithi033.stdout: "kstore_max_bytes": "67108864", 2024-04-19T01:03:58.527 INFO:teuthology.orchestra.run.smithi033.stdout: "kstore_max_ops": "512", 2024-04-19T01:03:58.527 INFO:teuthology.orchestra.run.smithi033.stdout: "kstore_nid_prealloc": "1024", 2024-04-19T01:03:58.527 INFO:teuthology.orchestra.run.smithi033.stdout: "kstore_onode_map_size": "1024", 2024-04-19T01:03:58.527 INFO:teuthology.orchestra.run.smithi033.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-04-19T01:03:58.527 INFO:teuthology.orchestra.run.smithi033.stdout: "kstore_sync_submit_transaction": "false", 2024-04-19T01:03:58.527 INFO:teuthology.orchestra.run.smithi033.stdout: "kstore_sync_transaction": "false", 2024-04-19T01:03:58.528 INFO:teuthology.orchestra.run.smithi033.stdout: "librados_thread_count": "2", 2024-04-19T01:03:58.528 INFO:teuthology.orchestra.run.smithi033.stdout: "lockdep": "false", 2024-04-19T01:03:58.528 INFO:teuthology.orchestra.run.smithi033.stdout: "lockdep_force_backtrace": "false", 2024-04-19T01:03:58.528 INFO:teuthology.orchestra.run.smithi033.stdout: "log_coarse_timestamps": "true", 2024-04-19T01:03:58.528 INFO:teuthology.orchestra.run.smithi033.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-04-19T01:03:58.528 INFO:teuthology.orchestra.run.smithi033.stdout: "log_flush_on_exit": "false", 2024-04-19T01:03:58.528 INFO:teuthology.orchestra.run.smithi033.stdout: "log_graylog_host": "127.0.0.1", 2024-04-19T01:03:58.528 INFO:teuthology.orchestra.run.smithi033.stdout: "log_graylog_port": "12201", 2024-04-19T01:03:58.528 INFO:teuthology.orchestra.run.smithi033.stdout: "log_max_new": "1000", 2024-04-19T01:03:58.529 INFO:teuthology.orchestra.run.smithi033.stdout: "log_max_recent": "10000", 2024-04-19T01:03:58.529 INFO:teuthology.orchestra.run.smithi033.stdout: "log_stderr_prefix": "", 2024-04-19T01:03:58.530 INFO:teuthology.orchestra.run.smithi033.stdout: "log_stop_at_utilization": "0.970000", 2024-04-19T01:03:58.530 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": "true", 2024-04-19T01:03:58.530 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_graylog": "false", 2024-04-19T01:03:58.530 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_journald": "false", 2024-04-19T01:03:58.530 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_stderr": "false", 2024-04-19T01:03:58.530 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_syslog": "false", 2024-04-19T01:03:58.530 INFO:teuthology.orchestra.run.smithi033.stdout: "max_rotating_auth_attempts": "10", 2024-04-19T01:03:58.530 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-04-19T01:03:58.530 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_action_on_write_error": "1", 2024-04-19T01:03:58.530 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_alternate_name_max": "8192", 2024-04-19T01:03:58.530 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_asio_thread_count": "2", 2024-04-19T01:03:58.530 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_export_pin": "true", 2024-04-19T01:03:58.531 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_fragment_dirs": "true", 2024-04-19T01:03:58.531 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-04-19T01:03:58.531 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_fragment_interval": "5", 2024-04-19T01:03:58.531 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_fragment_size_max": "100000", 2024-04-19T01:03:58.531 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_idle_threshold": "0.000000", 2024-04-19T01:03:58.531 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_interval": "10", 2024-04-19T01:03:58.531 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_max": "-1", 2024-04-19T01:03:58.531 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_max_until": "-1", 2024-04-19T01:03:58.531 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_merge_size": "50", 2024-04-19T01:03:58.532 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_midchunk": "0.300000", 2024-04-19T01:03:58.532 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_min_rebalance": "0.100000", 2024-04-19T01:03:58.532 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_min_start": "0.200000", 2024-04-19T01:03:58.532 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_minchunk": "0.001000", 2024-04-19T01:03:58.532 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_mode": "0", 2024-04-19T01:03:58.532 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_need_max": "1.200000", 2024-04-19T01:03:58.532 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_need_min": "0.800000", 2024-04-19T01:03:58.533 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_overload_epochs": "2", 2024-04-19T01:03:58.533 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-04-19T01:03:58.533 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_sample_interval": "3.000000", 2024-04-19T01:03:58.533 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_split_bits": "3", 2024-04-19T01:03:58.533 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_split_rd": "25000.000000", 2024-04-19T01:03:58.533 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_split_size": "10000", 2024-04-19T01:03:58.533 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_split_wr": "10000.000000", 2024-04-19T01:03:58.533 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_target_decay": "10.000000", 2024-04-19T01:03:58.533 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-04-19T01:03:58.533 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_beacon_grace": "15.000000", 2024-04-19T01:03:58.533 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_beacon_interval": "4.000000", 2024-04-19T01:03:58.533 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_beacon_mon_down_grace": "60", 2024-04-19T01:03:58.533 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_cache_memory_limit": "4294967296", 2024-04-19T01:03:58.533 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_cache_mid": "0.700000", 2024-04-19T01:03:58.534 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-04-19T01:03:58.534 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_cache_quiesce_delay": "0", 2024-04-19T01:03:58.534 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_cache_quiesce_sleep": "200", 2024-04-19T01:03:58.534 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_cache_quiesce_splitauth": "true", 2024-04-19T01:03:58.534 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_cache_quiesce_threshold": "524288", 2024-04-19T01:03:58.534 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_cache_release_free_interval": "10", 2024-04-19T01:03:58.534 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_cache_reservation": "0.050000", 2024-04-19T01:03:58.534 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-04-19T01:03:58.534 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_cache_trim_interval": "1", 2024-04-19T01:03:58.534 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_cache_trim_threshold": "262144", 2024-04-19T01:03:58.534 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-04-19T01:03:58.534 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-04-19T01:03:58.534 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_client_delegate_inos_pct": "50", 2024-04-19T01:03:58.534 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_client_prealloc_inos": "1000", 2024-04-19T01:03:58.535 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-04-19T01:03:58.535 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_connect_bootstrapping": "false", 2024-04-19T01:03:58.535 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_damage_table_max_entries": "10000", 2024-04-19T01:03:58.535 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-04-19T01:03:58.535 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_debug_auth_pins": "false", 2024-04-19T01:03:58.535 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_debug_frag": "false", 2024-04-19T01:03:58.535 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_debug_scatterstat": "false", 2024-04-19T01:03:58.535 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_debug_subtrees": "false", 2024-04-19T01:03:58.535 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_decay_halflife": "5.000000", 2024-04-19T01:03:58.535 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_default_dir_hash": "2", 2024-04-19T01:03:58.535 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_defer_session_stale": "true", 2024-04-19T01:03:58.535 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_deny_all_reconnect": "false", 2024-04-19T01:03:58.535 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_dir_keys_per_op": "16384", 2024-04-19T01:03:58.535 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_dir_max_commit_size": "10", 2024-04-19T01:03:58.536 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_dir_max_entries": "0", 2024-04-19T01:03:58.536 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_dir_prefetch": "true", 2024-04-19T01:03:58.536 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_dirstat_min_interval": "1.000000", 2024-04-19T01:03:58.536 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_dump_cache_after_rejoin": "false", 2024-04-19T01:03:58.536 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_dump_cache_on_map": "false", 2024-04-19T01:03:58.536 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_dump_cache_threshold_file": "0", 2024-04-19T01:03:58.536 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-04-19T01:03:58.536 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_early_reply": "true", 2024-04-19T01:03:58.536 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_enable_op_tracker": "true", 2024-04-19T01:03:58.536 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_enforce_unique_name": "true", 2024-04-19T01:03:58.537 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_export_ephemeral_distributed": "true", 2024-04-19T01:03:58.538 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-04-19T01:03:58.538 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_export_ephemeral_random": "true", 2024-04-19T01:03:58.538 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-04-19T01:03:58.538 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-04-19T01:03:58.538 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_forward_all_requests_to_auth": "false", 2024-04-19T01:03:58.538 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-04-19T01:03:58.538 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-04-19T01:03:58.538 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-04-19T01:03:58.538 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-04-19T01:03:58.538 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_health_cache_threshold": "1.500000", 2024-04-19T01:03:58.538 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_health_summarize_threshold": "10", 2024-04-19T01:03:58.538 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_heartbeat_grace": "15.000000", 2024-04-19T01:03:58.538 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_heartbeat_reset_grace": "1000", 2024-04-19T01:03:58.538 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_inject_health_dummy": "false", 2024-04-19T01:03:58.539 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-04-19T01:03:58.540 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_inject_migrator_session_race": "false", 2024-04-19T01:03:58.540 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-04-19T01:03:58.540 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-04-19T01:03:58.540 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-04-19T01:03:58.540 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_join_fs": "", 2024-04-19T01:03:58.540 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_journal_format": "1", 2024-04-19T01:03:58.540 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_kill_create_at": "0", 2024-04-19T01:03:58.540 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_kill_export_at": "0", 2024-04-19T01:03:58.541 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_kill_import_at": "0", 2024-04-19T01:03:58.541 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_kill_journal_at": "0", 2024-04-19T01:03:58.541 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_kill_journal_expire_at": "0", 2024-04-19T01:03:58.541 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_kill_journal_replay_at": "0", 2024-04-19T01:03:58.541 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_kill_link_at": "0", 2024-04-19T01:03:58.541 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_kill_mdstable_at": "0", 2024-04-19T01:03:58.541 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_kill_openc_at": "0", 2024-04-19T01:03:58.541 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_kill_rename_at": "0", 2024-04-19T01:03:58.541 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_kill_shutdown_at": "0", 2024-04-19T01:03:58.542 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-04-19T01:03:58.542 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_log_event_large_threshold": "524288", 2024-04-19T01:03:58.542 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_log_events_per_segment": "1024", 2024-04-19T01:03:58.542 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_log_major_segment_event_ratio": "12", 2024-04-19T01:03:58.542 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_log_max_events": "-1", 2024-04-19T01:03:58.542 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_log_max_segments": "128", 2024-04-19T01:03:58.542 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_log_pause": "false", 2024-04-19T01:03:58.542 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_log_segment_size": "0", 2024-04-19T01:03:58.542 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_log_skip_corrupt_events": "false", 2024-04-19T01:03:58.542 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_log_skip_unbounded_events": "false", 2024-04-19T01:03:58.542 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-04-19T01:03:58.542 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_log_trim_threshold": "128", 2024-04-19T01:03:58.542 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-04-19T01:03:58.543 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_log_warn_factor": "2.000000", 2024-04-19T01:03:58.543 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_max_caps_per_client": "1048576", 2024-04-19T01:03:58.543 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_max_completed_flushes": "100000", 2024-04-19T01:03:58.543 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_max_completed_requests": "100000", 2024-04-19T01:03:58.543 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_max_export_size": "20971520", 2024-04-19T01:03:58.543 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_max_file_recover": "32", 2024-04-19T01:03:58.543 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_max_purge_files": "64", 2024-04-19T01:03:58.543 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_max_purge_ops": "8192", 2024-04-19T01:03:58.543 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-04-19T01:03:58.543 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-04-19T01:03:58.543 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_max_snaps_per_dir": "100", 2024-04-19T01:03:58.543 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_metrics_update_interval": "2", 2024-04-19T01:03:58.543 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_min_caps_per_client": "100", 2024-04-19T01:03:58.543 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_min_caps_working_set": "10000", 2024-04-19T01:03:58.544 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-04-19T01:03:58.544 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_numa_node": "-1", 2024-04-19T01:03:58.544 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-04-19T01:03:58.544 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_op_complaint_time": "30.000000", 2024-04-19T01:03:58.544 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_op_history_duration": "600", 2024-04-19T01:03:58.544 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_op_history_size": "20", 2024-04-19T01:03:58.544 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_op_history_slow_op_size": "20", 2024-04-19T01:03:58.544 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_op_history_slow_op_threshold": "10", 2024-04-19T01:03:58.544 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_op_log_threshold": "5", 2024-04-19T01:03:58.544 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_ping_grace": "15", 2024-04-19T01:03:58.544 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_ping_interval": "5", 2024-04-19T01:03:58.544 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-04-19T01:03:58.544 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-04-19T01:03:58.544 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_recall_max_caps": "30000", 2024-04-19T01:03:58.544 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-04-19T01:03:58.545 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_recall_max_decay_threshold": "131072", 2024-04-19T01:03:58.545 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-04-19T01:03:58.545 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_recall_warning_threshold": "262144", 2024-04-19T01:03:58.545 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_reconnect_timeout": "45.000000", 2024-04-19T01:03:58.545 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_replay_interval": "1.000000", 2024-04-19T01:03:58.545 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-04-19T01:03:58.545 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-04-19T01:03:58.545 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_root_ino_gid": "0", 2024-04-19T01:03:58.545 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_root_ino_uid": "0", 2024-04-19T01:03:58.545 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-04-19T01:03:58.545 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_session_blocklist_on_evict": "true", 2024-04-19T01:03:58.545 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_session_blocklist_on_timeout": "true", 2024-04-19T01:03:58.545 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-04-19T01:03:58.545 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-04-19T01:03:58.545 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-04-19T01:03:58.546 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-04-19T01:03:58.546 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-04-19T01:03:58.546 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_session_metadata_threshold": "16777216", 2024-04-19T01:03:58.546 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-04-19T01:03:58.546 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_shutdown_check": "0", 2024-04-19T01:03:58.546 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_skip_ino": "0", 2024-04-19T01:03:58.546 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_sleep_rank_change": "0.000000", 2024-04-19T01:03:58.546 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_snap_max_uid": "4294967294", 2024-04-19T01:03:58.546 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_snap_min_uid": "0", 2024-04-19T01:03:58.546 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_snap_rstat": "false", 2024-04-19T01:03:58.546 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_standby_replay_damaged": "false", 2024-04-19T01:03:58.546 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_symlink_recovery": "true", 2024-04-19T01:03:58.546 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_task_status_update_interval": "2.000000", 2024-04-19T01:03:58.547 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_thrash_exports": "0", 2024-04-19T01:03:58.547 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_thrash_fragments": "0", 2024-04-19T01:03:58.547 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_tick_interval": "5.000000", 2024-04-19T01:03:58.547 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_valgrind_exit": "false", 2024-04-19T01:03:58.547 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_verify_backtrace": "1", 2024-04-19T01:03:58.547 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_verify_scatter": "false", 2024-04-19T01:03:58.547 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_wipe_ino_prealloc": "false", 2024-04-19T01:03:58.547 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_wipe_sessions": "false", 2024-04-19T01:03:58.547 INFO:teuthology.orchestra.run.smithi033.stdout: "mempool_debug": "false", 2024-04-19T01:03:58.547 INFO:teuthology.orchestra.run.smithi033.stdout: "memstore_debug_omit_block_device_write": "false", 2024-04-19T01:03:58.547 INFO:teuthology.orchestra.run.smithi033.stdout: "memstore_device_bytes": "1073741824", 2024-04-19T01:03:58.547 INFO:teuthology.orchestra.run.smithi033.stdout: "memstore_page_set": "false", 2024-04-19T01:03:58.548 INFO:teuthology.orchestra.run.smithi033.stdout: "memstore_page_size": "65536", 2024-04-19T01:03:58.548 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_client_bytes": "134217728", 2024-04-19T01:03:58.548 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_client_messages": "512", 2024-04-19T01:03:58.548 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-04-19T01:03:58.548 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_connect_retry_interval": "1.000000", 2024-04-19T01:03:58.548 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-04-19T01:03:58.548 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-04-19T01:03:58.548 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_disabled_modules": "", 2024-04-19T01:03:58.548 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-04-19T01:03:58.549 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_max_pg_creating": "1024", 2024-04-19T01:03:58.549 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_max_pg_num_change": "128", 2024-04-19T01:03:58.549 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_mds_bytes": "134217728", 2024-04-19T01:03:58.549 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_mds_messages": "128", 2024-04-19T01:03:58.549 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-04-19T01:03:58.550 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_mon_bytes": "134217728", 2024-04-19T01:03:58.550 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_mon_messages": "128", 2024-04-19T01:03:58.550 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_osd_bytes": "536870912", 2024-04-19T01:03:58.550 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_osd_messages": "8192", 2024-04-19T01:03:58.550 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_pool": "true", 2024-04-19T01:03:58.550 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_service_beacon_grace": "60.000000", 2024-04-19T01:03:58.551 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_standby_modules": "true", 2024-04-19T01:03:58.551 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_stats_period": "5", 2024-04-19T01:03:58.551 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_stats_threshold": "5", 2024-04-19T01:03:58.551 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_tick_period": "2", 2024-04-19T01:03:58.551 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-04-19T01:03:58.551 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_accept_timeout_factor": "2.000000", 2024-04-19T01:03:58.551 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_allow_pool_delete": "true", 2024-04-19T01:03:58.551 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_allow_pool_size_one": "true", 2024-04-19T01:03:58.551 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_auth_validate_all_caps": "true", 2024-04-19T01:03:58.551 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-04-19T01:03:58.551 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-04-19T01:03:58.551 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_client_bytes": "104857600", 2024-04-19T01:03:58.552 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_client_directed_command_retry": "5", 2024-04-19T01:03:58.552 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_client_hunt_interval": "3.000000", 2024-04-19T01:03:58.552 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-04-19T01:03:58.552 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-04-19T01:03:58.552 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-04-19T01:03:58.552 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_client_hunt_parallel": "3", 2024-04-19T01:03:58.552 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_client_log_interval": "1.000000", 2024-04-19T01:03:58.552 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-04-19T01:03:58.552 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_client_ping_interval": "10.000000", 2024-04-19T01:03:58.552 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_client_ping_timeout": "30.000000", 2024-04-19T01:03:58.552 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_client_target_rank": "-1", 2024-04-19T01:03:58.552 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_clock_drift_allowed": "1.000000", 2024-04-19T01:03:58.553 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-04-19T01:03:58.553 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-04-19T01:03:58.553 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_cluster_log_level": "debug", 2024-04-19T01:03:58.553 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_cluster_log_to_file": "true", 2024-04-19T01:03:58.553 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_cluster_log_to_graylog": "false", 2024-04-19T01:03:58.553 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-04-19T01:03:58.553 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-04-19T01:03:58.553 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_cluster_log_to_journald": "false", 2024-04-19T01:03:58.553 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_cluster_log_to_stderr": "false", 2024-04-19T01:03:58.553 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-04-19T01:03:58.553 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-04-19T01:03:58.553 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_compact_on_bootstrap": "false", 2024-04-19T01:03:58.553 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_compact_on_start": "false", 2024-04-19T01:03:58.553 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_compact_on_trim": "true", 2024-04-19T01:03:58.553 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_con_tracker_persist_interval": "10", 2024-04-19T01:03:58.554 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_con_tracker_score_halflife": "43200", 2024-04-19T01:03:58.554 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_config_key_max_entry_size": "65536", 2024-04-19T01:03:58.554 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_cpu_threads": "4", 2024-04-19T01:03:58.554 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_crush_min_required_version": "hammer", 2024-04-19T01:03:58.554 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_daemon_bytes": "419430400", 2024-04-19T01:03:58.554 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-04-19T01:03:58.554 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_data_avail_crit": "5", 2024-04-19T01:03:58.554 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_data_avail_warn": "5", 2024-04-19T01:03:58.554 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_data_size_warn": "16106127360", 2024-04-19T01:03:58.554 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_debug_block_osdmap_trim": "false", 2024-04-19T01:03:58.554 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-04-19T01:03:58.554 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_debug_dump_json": "false", 2024-04-19T01:03:58.555 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-04-19T01:03:58.555 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_debug_dump_transactions": "false", 2024-04-19T01:03:58.555 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_debug_extra_checks": "false", 2024-04-19T01:03:58.555 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-04-19T01:03:58.555 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-04-19T01:03:58.555 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_debug_no_require_reef": "false", 2024-04-19T01:03:58.555 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_debug_no_require_squid": "false", 2024-04-19T01:03:58.555 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-04-19T01:03:58.555 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_delta_reset_interval": "10.000000", 2024-04-19T01:03:58.555 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_dns_srv_name": "ceph-mon", 2024-04-19T01:03:58.555 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_down_mkfs_grace": "120", 2024-04-19T01:03:58.555 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_down_uptime_grace": "60", 2024-04-19T01:03:58.555 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_election_default_strategy": "1", 2024-04-19T01:03:58.555 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_election_timeout": "5.000000", 2024-04-19T01:03:58.556 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-04-19T01:03:58.556 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_elector_ping_divisor": "2", 2024-04-19T01:03:58.556 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_elector_ping_timeout": "2.000000", 2024-04-19T01:03:58.556 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_enable_op_tracker": "true", 2024-04-19T01:03:58.556 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_fake_pool_delete": "false", 2024-04-19T01:03:58.556 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_force_quorum_join": "false", 2024-04-19T01:03:58.556 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_globalid_prealloc": "10000", 2024-04-19T01:03:58.556 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_health_detail_to_clog": "false", 2024-04-19T01:03:58.556 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_health_log_update_period": "5", 2024-04-19T01:03:58.556 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_health_max_detail": "50", 2024-04-19T01:03:58.556 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_health_to_clog": "true", 2024-04-19T01:03:58.556 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_health_to_clog_interval": "600", 2024-04-19T01:03:58.556 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-04-19T01:03:58.557 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_host": "172.21.15.33,172.21.15.110,172.21.15.118", 2024-04-19T01:03:58.557 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_host_override": "", 2024-04-19T01:03:58.557 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_initial_members": "", 2024-04-19T01:03:58.557 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-04-19T01:03:58.558 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-04-19T01:03:58.559 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-04-19T01:03:58.559 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-04-19T01:03:58.559 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_keyvaluedb": "rocksdb", 2024-04-19T01:03:58.559 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_lease": "5.000000", 2024-04-19T01:03:58.559 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-04-19T01:03:58.559 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-04-19T01:03:58.559 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_log_full_interval": "50", 2024-04-19T01:03:58.559 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_log_max": "10000", 2024-04-19T01:03:58.559 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_log_max_summary": "50", 2024-04-19T01:03:58.559 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_max_log_entries_per_event": "4096", 2024-04-19T01:03:58.559 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_max_log_epochs": "500", 2024-04-19T01:03:58.559 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_max_mdsmap_epochs": "500", 2024-04-19T01:03:58.559 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_max_mgrmap_epochs": "500", 2024-04-19T01:03:58.559 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_max_osd": "10000", 2024-04-19T01:03:58.560 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_max_pg_per_osd": "10000", 2024-04-19T01:03:58.560 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_max_pool_pg_num": "65536", 2024-04-19T01:03:58.560 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-04-19T01:03:58.560 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-04-19T01:03:58.560 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_mds_force_trim_to": "0", 2024-04-19T01:03:58.560 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_mds_skip_sanity": "false", 2024-04-19T01:03:58.560 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_memory_autotune": "true", 2024-04-19T01:03:58.560 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_memory_target": "2147483648", 2024-04-19T01:03:58.560 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_mgr_beacon_grace": "30", 2024-04-19T01:03:58.560 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-04-19T01:03:58.560 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_mgr_digest_period": "5", 2024-04-19T01:03:58.560 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_mgr_inactive_grace": "60", 2024-04-19T01:03:58.560 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_mgr_mkfs_grace": "240", 2024-04-19T01:03:58.560 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-04-19T01:03:58.560 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_min_osdmap_epochs": "500", 2024-04-19T01:03:58.561 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_op_complaint_time": "30", 2024-04-19T01:03:58.561 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_op_history_duration": "600", 2024-04-19T01:03:58.561 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_op_history_size": "20", 2024-04-19T01:03:58.561 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_op_history_slow_op_size": "20", 2024-04-19T01:03:58.561 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_op_history_slow_op_threshold": "10", 2024-04-19T01:03:58.561 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_op_log_threshold": "5", 2024-04-19T01:03:58.561 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-04-19T01:03:58.561 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-04-19T01:03:58.561 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-04-19T01:03:58.561 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_auto_mark_in": "false", 2024-04-19T01:03:58.561 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_auto_mark_new_in": "true", 2024-04-19T01:03:58.561 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-04-19T01:03:58.561 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-04-19T01:03:58.561 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_cache_size": "500", 2024-04-19T01:03:58.561 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_cache_size_min": "134217728", 2024-04-19T01:03:58.562 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_crush_smoke_test": "true", 2024-04-19T01:03:58.562 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_destroyed_out_interval": "600", 2024-04-19T01:03:58.562 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_down_out_interval": "600", 2024-04-19T01:03:58.562 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-04-19T01:03:58.562 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-04-19T01:03:58.562 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_force_trim_to": "0", 2024-04-19T01:03:58.563 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_full_ratio": "0.950000", 2024-04-19T01:03:58.563 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-04-19T01:03:58.563 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_laggy_halflife": "3600", 2024-04-19T01:03:58.563 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_laggy_max_interval": "300", 2024-04-19T01:03:58.563 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_laggy_weight": "0.300000", 2024-04-19T01:03:58.563 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-04-19T01:03:58.563 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_max_initial_pgs": "1024", 2024-04-19T01:03:58.563 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_min_down_reporters": "2", 2024-04-19T01:03:58.563 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-04-19T01:03:58.563 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-04-19T01:03:58.563 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-04-19T01:03:58.563 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_prime_pg_temp": "true", 2024-04-19T01:03:58.563 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-04-19T01:03:58.564 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-04-19T01:03:58.564 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_report_timeout": "900", 2024-04-19T01:03:58.564 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-04-19T01:03:58.564 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-04-19T01:03:58.564 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_warn_num_repaired": "10", 2024-04-19T01:03:58.564 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_warn_op_age": "32.000000", 2024-04-19T01:03:58.564 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-04-19T01:03:58.564 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osdmap_full_prune_interval": "10", 2024-04-19T01:03:58.564 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osdmap_full_prune_min": "10000", 2024-04-19T01:03:58.564 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-04-19T01:03:58.564 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-04-19T01:03:58.564 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_pg_stuck_threshold": "60", 2024-04-19T01:03:58.564 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-04-19T01:03:58.564 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_pg_warn_min_objects": "10000", 2024-04-19T01:03:58.564 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_pg_warn_min_per_osd": "0", 2024-04-19T01:03:58.565 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-04-19T01:03:58.565 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_pool_quota_crit_threshold": "0", 2024-04-19T01:03:58.565 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_pool_quota_warn_threshold": "0", 2024-04-19T01:03:58.565 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_probe_timeout": "2.000000", 2024-04-19T01:03:58.565 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_reweight_max_change": "0.050000", 2024-04-19T01:03:58.565 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_reweight_max_osds": "4", 2024-04-19T01:03:58.565 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-04-19T01:03:58.565 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-04-19T01:03:58.565 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-04-19T01:03:58.565 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-04-19T01:03:58.565 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-04-19T01:03:58.565 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_scrub_interval": "300", 2024-04-19T01:03:58.565 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_scrub_max_keys": "100", 2024-04-19T01:03:58.565 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_scrub_timeout": "300", 2024-04-19T01:03:58.565 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_session_timeout": "300", 2024-04-19T01:03:58.566 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_smart_report_timeout": "5", 2024-04-19T01:03:58.566 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_stat_smooth_intervals": "6", 2024-04-19T01:03:58.566 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-04-19T01:03:58.566 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_stretch_pool_min_size": "2", 2024-04-19T01:03:58.566 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_stretch_pool_size": "4", 2024-04-19T01:03:58.567 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-04-19T01:03:58.567 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_subscribe_interval": "86400.000000", 2024-04-19T01:03:58.567 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_sync_debug": "false", 2024-04-19T01:03:58.567 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_sync_max_payload_keys": "2000", 2024-04-19T01:03:58.567 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_sync_max_payload_size": "1048576", 2024-04-19T01:03:58.567 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_sync_provider_kill_at": "0", 2024-04-19T01:03:58.567 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_sync_requester_kill_at": "0", 2024-04-19T01:03:58.568 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_sync_timeout": "60.000000", 2024-04-19T01:03:58.568 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_target_pg_per_osd": "100", 2024-04-19T01:03:58.568 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_tick_interval": "5", 2024-04-19T01:03:58.568 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_timecheck_interval": "300.000000", 2024-04-19T01:03:58.568 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-04-19T01:03:58.568 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_use_min_delay_socket": "false", 2024-04-19T01:03:58.568 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_older_version_delay": "604800", 2024-04-19T01:03:58.568 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-04-19T01:03:58.568 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-04-19T01:03:58.568 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-04-19T01:03:58.568 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_filestore_osds": "false", 2024-04-19T01:03:58.569 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-04-19T01:03:58.569 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-04-19T01:03:58.569 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-04-19T01:03:58.569 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_misplaced": "false", 2024-04-19T01:03:58.569 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-04-19T01:03:58.569 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_older_version": "true", 2024-04-19T01:03:58.569 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-04-19T01:03:58.569 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_pool_no_app": "true", 2024-04-19T01:03:58.569 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-04-19T01:03:58.569 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-04-19T01:03:58.569 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-04-19T01:03:58.569 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-04-19T01:03:58.569 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_too_few_osds": "false", 2024-04-19T01:03:58.569 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-04-19T01:03:58.570 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-04-19T01:03:58.570 INFO:teuthology.orchestra.run.smithi033.stdout: "monmap": "", 2024-04-19T01:03:58.570 INFO:teuthology.orchestra.run.smithi033.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-19T01:03:58.570 INFO:teuthology.orchestra.run.smithi033.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-04-19T01:03:58.570 INFO:teuthology.orchestra.run.smithi033.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-04-19T01:03:58.570 INFO:teuthology.orchestra.run.smithi033.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-19T01:03:58.570 INFO:teuthology.orchestra.run.smithi033.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-04-19T01:03:58.570 INFO:teuthology.orchestra.run.smithi033.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-04-19T01:03:58.570 INFO:teuthology.orchestra.run.smithi033.stdout: "motr_tracing_enabled": "false", 2024-04-19T01:03:58.570 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_op_threads": "3", 2024-04-19T01:03:58.570 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_buffer_size": "131072", 2024-04-19T01:03:58.570 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_cm": "false", 2024-04-19T01:03:58.570 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_device_name": "", 2024-04-19T01:03:58.570 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_dscp": "96", 2024-04-19T01:03:58.571 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-04-19T01:03:58.571 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_gid_idx": "0", 2024-04-19T01:03:58.571 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_local_gid": "", 2024-04-19T01:03:58.571 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_polling_us": "1000", 2024-04-19T01:03:58.571 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_port_num": "1", 2024-04-19T01:03:58.571 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-04-19T01:03:58.571 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-04-19T01:03:58.571 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_roce_ver": "1", 2024-04-19T01:03:58.571 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_send_buffers": "1024", 2024-04-19T01:03:58.571 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_sl": "3", 2024-04-19T01:03:58.571 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_support_srq": "true", 2024-04-19T01:03:58.571 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_type": "ib", 2024-04-19T01:03:58.571 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_reap_threshold": "5", 2024-04-19T01:03:58.571 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_bind_before_connect": "false", 2024-04-19T01:03:58.571 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_bind_ipv4": "true", 2024-04-19T01:03:58.572 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_bind_ipv6": "false", 2024-04-19T01:03:58.572 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_bind_msgr1": "true", 2024-04-19T01:03:58.572 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_bind_msgr2": "true", 2024-04-19T01:03:58.572 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_bind_port_max": "7568", 2024-04-19T01:03:58.572 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_bind_port_min": "6800", 2024-04-19T01:03:58.572 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_bind_prefer_ipv4": "false", 2024-04-19T01:03:58.572 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_bind_retry_count": "3", 2024-04-19T01:03:58.572 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_bind_retry_delay": "5", 2024-04-19T01:03:58.572 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_blackhole_client": "false", 2024-04-19T01:03:58.573 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_blackhole_mds": "false", 2024-04-19T01:03:58.573 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_blackhole_mgr": "false", 2024-04-19T01:03:58.573 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_blackhole_mon": "false", 2024-04-19T01:03:58.573 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_blackhole_osd": "false", 2024-04-19T01:03:58.573 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_client_mode": "crc secure", 2024-04-19T01:03:58.573 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_cluster_mode": "crc secure", 2024-04-19T01:03:58.573 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_cluster_type": "", 2024-04-19T01:03:58.573 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_compress_secure": "false", 2024-04-19T01:03:58.573 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_connection_idle_timeout": "900", 2024-04-19T01:03:58.573 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_connection_ready_timeout": "10", 2024-04-19T01:03:58.573 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_crc_data": "true", 2024-04-19T01:03:58.573 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_crc_header": "true", 2024-04-19T01:03:58.573 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_die_on_bad_msg": "false", 2024-04-19T01:03:58.573 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_die_on_bug": "true", 2024-04-19T01:03:58.574 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_die_on_old_message": "true", 2024-04-19T01:03:58.574 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_die_on_skipped_message": "false", 2024-04-19T01:03:58.574 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_die_on_unhandled_msg": "false", 2024-04-19T01:03:58.574 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-04-19T01:03:58.574 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_coremask": "0xF", 2024-04-19T01:03:58.574 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-04-19T01:03:58.574 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_devs_allowlist": "", 2024-04-19T01:03:58.574 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_enable_tso": "true", 2024-04-19T01:03:58.574 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-04-19T01:03:58.574 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-04-19T01:03:58.574 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_hugepages": "", 2024-04-19T01:03:58.574 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_hw_flow_control": "true", 2024-04-19T01:03:58.574 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-04-19T01:03:58.574 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_lro": "true", 2024-04-19T01:03:58.574 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_memory_channel": "4", 2024-04-19T01:03:58.575 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-04-19T01:03:58.575 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_pmd": "", 2024-04-19T01:03:58.575 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_port_id": "0", 2024-04-19T01:03:58.575 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-04-19T01:03:58.575 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dump_corrupt_message_level": "1", 2024-04-19T01:03:58.575 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dump_on_send": "false", 2024-04-19T01:03:58.575 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_initial_backoff": "0.200000", 2024-04-19T01:03:58.575 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_inject_delay_max": "1.000000", 2024-04-19T01:03:58.575 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_inject_delay_probability": "0.000000", 2024-04-19T01:03:58.575 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_inject_delay_type": "", 2024-04-19T01:03:58.575 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_inject_internal_delays": "0.000000", 2024-04-19T01:03:58.575 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_inject_network_congestion": "0", 2024-04-19T01:03:58.575 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_inject_socket_failures": "2500", 2024-04-19T01:03:58.575 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_learn_addr_from_peer": "true", 2024-04-19T01:03:58.575 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_max_accept_failures": "4", 2024-04-19T01:03:58.576 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_max_backoff": "15.000000", 2024-04-19T01:03:58.576 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_mon_client_mode": "secure crc", 2024-04-19T01:03:58.576 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_mon_cluster_mode": "secure crc", 2024-04-19T01:03:58.576 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_mon_service_mode": "secure crc", 2024-04-19T01:03:58.576 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_osd_compress_min_size": "1024", 2024-04-19T01:03:58.576 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_osd_compress_mode": "none", 2024-04-19T01:03:58.576 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_osd_compression_algorithm": "snappy", 2024-04-19T01:03:58.576 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-04-19T01:03:58.576 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_pq_min_cost": "65536", 2024-04-19T01:03:58.576 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_public_type": "", 2024-04-19T01:03:58.576 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_service_mode": "crc secure", 2024-04-19T01:03:58.576 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_tcp_listen_backlog": "512", 2024-04-19T01:03:58.576 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_tcp_nodelay": "true", 2024-04-19T01:03:58.576 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-04-19T01:03:58.576 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_tcp_rcvbuf": "0", 2024-04-19T01:03:58.577 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_type": "async+posix", 2024-04-19T01:03:58.577 INFO:teuthology.orchestra.run.smithi033.stdout: "no_config_file": "false", 2024-04-19T01:03:58.577 INFO:teuthology.orchestra.run.smithi033.stdout: "objecter_completion_locks_per_session": "32", 2024-04-19T01:03:58.577 INFO:teuthology.orchestra.run.smithi033.stdout: "objecter_debug_inject_relock_delay": "false", 2024-04-19T01:03:58.577 INFO:teuthology.orchestra.run.smithi033.stdout: "objecter_inflight_op_bytes": "104857600", 2024-04-19T01:03:58.577 INFO:teuthology.orchestra.run.smithi033.stdout: "objecter_inflight_ops": "1024", 2024-04-19T01:03:58.577 INFO:teuthology.orchestra.run.smithi033.stdout: "objecter_inject_no_watch_ping": "false", 2024-04-19T01:03:58.577 INFO:teuthology.orchestra.run.smithi033.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-04-19T01:03:58.577 INFO:teuthology.orchestra.run.smithi033.stdout: "objecter_tick_interval": "5.000000", 2024-04-19T01:03:58.577 INFO:teuthology.orchestra.run.smithi033.stdout: "objecter_timeout": "10.000000", 2024-04-19T01:03:58.577 INFO:teuthology.orchestra.run.smithi033.stdout: "objectstore_blackhole": "false", 2024-04-19T01:03:58.577 INFO:teuthology.orchestra.run.smithi033.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-04-19T01:03:58.577 INFO:teuthology.orchestra.run.smithi033.stdout: "openssl_engine_opts": "", 2024-04-19T01:03:58.577 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_agent_delay_time": "5.000000", 2024-04-19T01:03:58.577 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_agent_hist_halflife": "1000", 2024-04-19T01:03:58.578 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_agent_max_low_ops": "2", 2024-04-19T01:03:58.578 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_agent_max_ops": "4", 2024-04-19T01:03:58.578 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-04-19T01:03:58.578 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_agent_quantize_effort": "0.100000", 2024-04-19T01:03:58.578 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_agent_slop": "0.020000", 2024-04-19T01:03:58.578 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-04-19T01:03:58.578 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_allow_recovery_below_min_size": "true", 2024-04-19T01:03:58.578 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_asio_thread_count": "2", 2024-04-19T01:03:58.578 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_async_recovery_min_cost": "1", 2024-04-19T01:03:58.578 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_auto_mark_unfound_lost": "false", 2024-04-19T01:03:58.578 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_backfill_retry_interval": "30.000000", 2024-04-19T01:03:58.578 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_backfill_scan_max": "512", 2024-04-19T01:03:58.578 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_backfill_scan_min": "64", 2024-04-19T01:03:58.578 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_backoff_on_degraded": "false", 2024-04-19T01:03:58.579 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_backoff_on_peering": "false", 2024-04-19T01:03:58.579 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_backoff_on_unfound": "true", 2024-04-19T01:03:58.579 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_beacon_report_interval": "300", 2024-04-19T01:03:58.579 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_bench_duration": "30", 2024-04-19T01:03:58.579 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-04-19T01:03:58.579 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_bench_max_block_size": "67108864", 2024-04-19T01:03:58.579 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_bench_small_size_max_iops": "100", 2024-04-19T01:03:58.579 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_blkin_trace_all": "false", 2024-04-19T01:03:58.579 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_blocked_scrub_grace_period": "120", 2024-04-19T01:03:58.579 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-04-19T01:03:58.579 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-04-19T01:03:58.579 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-04-19T01:03:58.579 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_check_for_log_corruption": "false", 2024-04-19T01:03:58.579 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-04-19T01:03:58.579 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-19T01:03:58.580 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_class_dir": "/usr/lib/rados-classes", 2024-04-19T01:03:58.580 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-19T01:03:58.580 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_class_update_on_start": "true", 2024-04-19T01:03:58.580 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_client_message_cap": "256", 2024-04-19T01:03:58.580 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_client_message_size_cap": "524288000", 2024-04-19T01:03:58.580 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_client_op_priority": "63", 2024-04-19T01:03:58.580 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_client_watch_timeout": "30", 2024-04-19T01:03:58.580 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_command_max_records": "256", 2024-04-19T01:03:58.580 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_compact_on_start": "false", 2024-04-19T01:03:58.580 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-04-19T01:03:58.580 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_crush_chooseleaf_type": "0", 2024-04-19T01:03:58.580 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_crush_initial_weight": "-1.000000", 2024-04-19T01:03:58.580 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_crush_update_on_start": "true", 2024-04-19T01:03:58.580 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_crush_update_weight_set": "true", 2024-04-19T01:03:58.580 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-04-19T01:03:58.581 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-04-19T01:03:58.581 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-04-19T01:03:58.581 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_drop_ping_duration": "0", 2024-04-19T01:03:58.581 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-04-19T01:03:58.581 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_feed_pullee": "-1", 2024-04-19T01:03:58.581 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-04-19T01:03:58.581 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-04-19T01:03:58.581 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-04-19T01:03:58.581 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_misdirected_ops": "false", 2024-04-19T01:03:58.581 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_no_acting_change": "false", 2024-04-19T01:03:58.581 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_no_purge_strays": "false", 2024-04-19T01:03:58.581 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_op_order": "false", 2024-04-19T01:03:58.581 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_pg_log_writeout": "false", 2024-04-19T01:03:58.581 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_pretend_recovery_active": "false", 2024-04-19T01:03:58.581 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-04-19T01:03:58.582 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-04-19T01:03:58.582 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_shutdown": "false", 2024-04-19T01:03:58.582 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-04-19T01:03:58.582 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-04-19T01:03:58.582 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_trim_objects": "false", 2024-04-19T01:03:58.582 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_verify_cached_snaps": "false", 2024-04-19T01:03:58.582 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_verify_missing_on_start": "false", 2024-04-19T01:03:58.582 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_verify_snaps": "false", 2024-04-19T01:03:58.582 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-04-19T01:03:58.582 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-04-19T01:03:58.582 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_deep_scrub_keys": "1024", 2024-04-19T01:03:58.582 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-04-19T01:03:58.582 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-04-19T01:03:58.582 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-04-19T01:03:58.583 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_deep_scrub_stride": "524288", 2024-04-19T01:03:58.583 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-04-19T01:03:58.583 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_default_data_pool_replay_window": "5", 2024-04-19T01:03:58.583 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_default_notify_timeout": "30", 2024-04-19T01:03:58.583 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_delete_sleep": "0.000000", 2024-04-19T01:03:58.583 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-04-19T01:03:58.583 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-04-19T01:03:58.583 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-04-19T01:03:58.584 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_discard_disconnected_ops": "true", 2024-04-19T01:03:58.584 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_enable_op_tracker": "true", 2024-04-19T01:03:58.584 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-04-19T01:03:58.584 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_extblkdev_plugins": "vdo", 2024-04-19T01:03:58.584 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-04-19T01:03:58.584 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-04-19T01:03:58.584 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_fast_info": "true", 2024-04-19T01:03:58.584 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_fast_shutdown": "true", 2024-04-19T01:03:58.584 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-04-19T01:03:58.584 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_fast_shutdown_timeout": "15", 2024-04-19T01:03:58.584 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-04-19T01:03:58.584 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-04-19T01:03:58.584 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-04-19T01:03:58.584 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_function_tracing": "false", 2024-04-19T01:03:58.584 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_heartbeat_grace": "20", 2024-04-19T01:03:58.585 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_heartbeat_interval": "6", 2024-04-19T01:03:58.585 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-04-19T01:03:58.585 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_heartbeat_min_peers": "10", 2024-04-19T01:03:58.585 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_heartbeat_min_size": "2000", 2024-04-19T01:03:58.585 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_heartbeat_stale": "600", 2024-04-19T01:03:58.585 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-04-19T01:03:58.585 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_hit_set_max_size": "100000", 2024-04-19T01:03:58.585 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_hit_set_min_size": "1000", 2024-04-19T01:03:58.585 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-04-19T01:03:58.585 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-04-19T01:03:58.585 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-04-19T01:03:58.585 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-04-19T01:03:58.585 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-04-19T01:03:58.585 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_journal_flush_on_shutdown": "true", 2024-04-19T01:03:58.585 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_journal_size": "5120", 2024-04-19T01:03:58.586 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_kill_backfill_at": "0", 2024-04-19T01:03:58.586 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_loop_before_reset_tphandle": "64", 2024-04-19T01:03:58.586 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_map_cache_size": "50", 2024-04-19T01:03:58.586 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_map_dedup": "true", 2024-04-19T01:03:58.586 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_map_message_max": "40", 2024-04-19T01:03:58.586 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_map_message_max_bytes": "10485760", 2024-04-19T01:03:58.586 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_map_share_max_epochs": "40", 2024-04-19T01:03:58.586 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_attr_name_len": "100", 2024-04-19T01:03:58.586 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_attr_size": "0", 2024-04-19T01:03:58.586 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_backfills": "1", 2024-04-19T01:03:58.586 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_markdown_count": "5", 2024-04-19T01:03:58.586 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_markdown_period": "600", 2024-04-19T01:03:58.586 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_object_name_len": "2048", 2024-04-19T01:03:58.586 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_object_namespace_len": "256", 2024-04-19T01:03:58.587 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_object_size": "134217728", 2024-04-19T01:03:58.587 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-04-19T01:03:58.587 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_omap_entries_per_request": "1024", 2024-04-19T01:03:58.587 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_pg_blocked_by": "16", 2024-04-19T01:03:58.587 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_pg_log_entries": "10000", 2024-04-19T01:03:58.587 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-04-19T01:03:58.587 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_pgls": "1024", 2024-04-19T01:03:58.587 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_push_cost": "8388608", 2024-04-19T01:03:58.587 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_push_objects": "10", 2024-04-19T01:03:58.587 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_scrubs": "3", 2024-04-19T01:03:58.587 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-04-19T01:03:58.587 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_trimming_pgs": "2", 2024-04-19T01:03:58.587 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_write_op_reply_len": "64", 2024-04-19T01:03:58.587 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_write_size": "90", 2024-04-19T01:03:58.587 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-04-19T01:03:58.588 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-04-19T01:03:58.588 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-04-19T01:03:58.588 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-04-19T01:03:58.588 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-04-19T01:03:58.588 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-04-19T01:03:58.588 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-04-19T01:03:58.588 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_override_recovery_settings": "false", 2024-04-19T01:03:58.588 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_profile": "balanced", 2024-04-19T01:03:58.588 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-04-19T01:03:58.588 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-04-19T01:03:58.588 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-04-19T01:03:58.588 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-04-19T01:03:58.588 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-04-19T01:03:58.588 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-04-19T01:03:58.588 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-04-19T01:03:58.589 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-04-19T01:03:58.589 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-04-19T01:03:58.589 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-04-19T01:03:58.589 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_skip_benchmark": "false", 2024-04-19T01:03:58.589 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_memory_base": "805306368", 2024-04-19T01:03:58.589 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_memory_cache_min": "134217728", 2024-04-19T01:03:58.589 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-04-19T01:03:58.589 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-04-19T01:03:58.589 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_memory_target": "4294967296", 2024-04-19T01:03:58.589 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_memory_target_autotune": "false", 2024-04-19T01:03:58.589 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-04-19T01:03:58.589 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_min_pg_log_entries": "250", 2024-04-19T01:03:58.589 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_min_recovery_priority": "0", 2024-04-19T01:03:58.589 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mon_heartbeat_interval": "30", 2024-04-19T01:03:58.589 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-04-19T01:03:58.590 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mon_report_interval": "5", 2024-04-19T01:03:58.590 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mon_report_max_in_flight": "2", 2024-04-19T01:03:58.590 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-04-19T01:03:58.590 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_num_cache_shards": "32", 2024-04-19T01:03:58.590 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_num_op_tracker_shard": "32", 2024-04-19T01:03:58.590 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_numa_auto_affinity": "true", 2024-04-19T01:03:58.590 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_numa_node": "-1", 2024-04-19T01:03:58.590 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_numa_prefer_iface": "true", 2024-04-19T01:03:58.590 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_object_clean_region_max_num_intervals": "1000", 2024-04-19T01:03:58.590 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_objecter_finishers": "1", 2024-04-19T01:03:58.590 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_objectstore": "bluestore", 2024-04-19T01:03:58.590 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_objectstore_fuse": "false", 2024-04-19T01:03:58.590 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_objectstore_tracing": "false", 2024-04-19T01:03:58.590 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_complaint_time": "30.000000", 2024-04-19T01:03:58.591 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_history_duration": "600", 2024-04-19T01:03:58.591 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_history_size": "20", 2024-04-19T01:03:58.591 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_history_slow_op_size": "20", 2024-04-19T01:03:58.591 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-04-19T01:03:58.591 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_log_threshold": "5", 2024-04-19T01:03:58.591 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_num_shards": "0", 2024-04-19T01:03:58.591 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_num_shards_hdd": "5", 2024-04-19T01:03:58.591 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_num_shards_ssd": "8", 2024-04-19T01:03:58.591 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_num_threads_per_shard": "0", 2024-04-19T01:03:58.591 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-04-19T01:03:58.591 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-04-19T01:03:58.591 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-04-19T01:03:58.591 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_pq_min_cost": "65536", 2024-04-19T01:03:58.591 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_queue": "mclock_scheduler", 2024-04-19T01:03:58.591 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_queue_cut_off": "high", 2024-04-19T01:03:58.591 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_thread_suicide_timeout": "150", 2024-04-19T01:03:58.592 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_thread_timeout": "15", 2024-04-19T01:03:58.592 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_open_classes_on_start": "true", 2024-04-19T01:03:58.592 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_os_flags": "0", 2024-04-19T01:03:58.592 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_peering_op_priority": "255", 2024-04-19T01:03:58.592 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pg_delete_cost": "1048576", 2024-04-19T01:03:58.592 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pg_delete_priority": "5", 2024-04-19T01:03:58.592 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-04-19T01:03:58.592 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-04-19T01:03:58.592 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pg_log_dups_tracked": "3000", 2024-04-19T01:03:58.592 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pg_log_trim_max": "10000", 2024-04-19T01:03:58.592 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pg_log_trim_min": "100", 2024-04-19T01:03:58.592 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-04-19T01:03:58.592 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pg_object_context_cache_count": "64", 2024-04-19T01:03:58.592 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-04-19T01:03:58.592 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-04-19T01:03:58.593 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-04-19T01:03:58.593 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-04-19T01:03:58.593 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-04-19T01:03:58.593 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-04-19T01:03:58.593 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-04-19T01:03:58.593 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-04-19T01:03:58.593 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_crimson": "false", 2024-04-19T01:03:58.593 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_crush_rule": "-1", 2024-04-19T01:03:58.594 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_ec_fast_read": "false", 2024-04-19T01:03:58.594 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-04-19T01:03:58.594 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_flag_bulk": "false", 2024-04-19T01:03:58.594 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-04-19T01:03:58.594 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_flag_nodelete": "false", 2024-04-19T01:03:58.594 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-04-19T01:03:58.594 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-04-19T01:03:58.594 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_flags": "0", 2024-04-19T01:03:58.594 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-04-19T01:03:58.594 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_min_size": "0", 2024-04-19T01:03:58.594 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-04-19T01:03:58.594 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_pg_num": "32", 2024-04-19T01:03:58.594 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_pgp_num": "0", 2024-04-19T01:03:58.595 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-04-19T01:03:58.595 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_read_ratio": "70", 2024-04-19T01:03:58.595 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_size": "2", 2024-04-19T01:03:58.595 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_type": "replicated", 2024-04-19T01:03:58.595 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-04-19T01:03:58.595 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_use_gmt_hitset": "true", 2024-04-19T01:03:58.595 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_push_per_object_cost": "1000", 2024-04-19T01:03:58.595 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_read_ec_check_for_errors": "false", 2024-04-19T01:03:58.595 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recover_clone_overlap": "true", 2024-04-19T01:03:58.595 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recover_clone_overlap_limit": "10", 2024-04-19T01:03:58.595 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_cost": "20971520", 2024-04-19T01:03:58.595 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_delay_start": "0.000000", 2024-04-19T01:03:58.595 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_max_active": "0", 2024-04-19T01:03:58.595 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_max_active_hdd": "3", 2024-04-19T01:03:58.596 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_max_active_ssd": "10", 2024-04-19T01:03:58.596 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_max_chunk": "8388608", 2024-04-19T01:03:58.596 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-04-19T01:03:58.596 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_max_single_start": "1", 2024-04-19T01:03:58.596 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_op_priority": "3", 2024-04-19T01:03:58.596 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_op_warn_multiple": "16", 2024-04-19T01:03:58.596 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_priority": "5", 2024-04-19T01:03:58.596 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_retry_interval": "30.000000", 2024-04-19T01:03:58.597 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_sleep": "0.000000", 2024-04-19T01:03:58.597 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-04-19T01:03:58.597 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-04-19T01:03:58.597 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-04-19T01:03:58.597 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_repair_during_recovery": "false", 2024-04-19T01:03:58.597 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_requested_scrub_priority": "120", 2024-04-19T01:03:58.597 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-04-19T01:03:58.597 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_rollback_to_cluster_snap": "", 2024-04-19T01:03:58.597 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_auto_repair": "false", 2024-04-19T01:03:58.597 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-04-19T01:03:58.597 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-04-19T01:03:58.597 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_begin_hour": "0", 2024-04-19T01:03:58.597 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_begin_week_day": "0", 2024-04-19T01:03:58.597 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_chunk_max": "25", 2024-04-19T01:03:58.598 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_chunk_min": "5", 2024-04-19T01:03:58.598 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_cost": "52428800", 2024-04-19T01:03:58.598 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-04-19T01:03:58.598 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_during_recovery": "false", 2024-04-19T01:03:58.598 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_end_hour": "0", 2024-04-19T01:03:58.598 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_end_week_day": "0", 2024-04-19T01:03:58.598 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_event_cost": "4096", 2024-04-19T01:03:58.598 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-04-19T01:03:58.598 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-04-19T01:03:58.598 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_invalid_stats": "true", 2024-04-19T01:03:58.598 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_load_threshold": "0.500000", 2024-04-19T01:03:58.598 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_max_interval": "604800.000000", 2024-04-19T01:03:58.598 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_max_preemptions": "5", 2024-04-19T01:03:58.598 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_min_interval": "86400.000000", 2024-04-19T01:03:58.599 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_priority": "5", 2024-04-19T01:03:58.599 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_reservation_timeout": "300000", 2024-04-19T01:03:58.599 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_sleep": "0.000000", 2024-04-19T01:03:58.599 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-04-19T01:03:58.599 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-04-19T01:03:58.599 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_shutdown_pgref_assert": "false", 2024-04-19T01:03:58.599 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-04-19T01:03:58.599 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_skip_data_digest": "false", 2024-04-19T01:03:58.599 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_smart_report_timeout": "5", 2024-04-19T01:03:58.599 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_snap_trim_cost": "1048576", 2024-04-19T01:03:58.599 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_snap_trim_priority": "5", 2024-04-19T01:03:58.599 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_snap_trim_sleep": "0.000000", 2024-04-19T01:03:58.599 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-04-19T01:03:58.599 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-04-19T01:03:58.599 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-04-19T01:03:58.600 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-04-19T01:03:58.600 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_stats_update_period_scrubbing": "15", 2024-04-19T01:03:58.600 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-04-19T01:03:58.600 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_target_transaction_size": "30", 2024-04-19T01:03:58.600 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-04-19T01:03:58.600 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-04-19T01:03:58.600 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-04-19T01:03:58.600 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-04-19T01:03:58.600 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-04-19T01:03:58.600 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-04-19T01:03:58.600 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-04-19T01:03:58.600 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_tier_default_cache_mode": "writeback", 2024-04-19T01:03:58.600 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-04-19T01:03:58.600 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-04-19T01:03:58.601 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_tracing": "false", 2024-04-19T01:03:58.601 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_use_stale_snap": "false", 2024-04-19T01:03:58.601 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-04-19T01:03:58.601 INFO:teuthology.orchestra.run.smithi033.stdout: "osdc_blkin_trace_all": "false", 2024-04-19T01:03:58.601 INFO:teuthology.orchestra.run.smithi033.stdout: "paxos_kill_at": "0", 2024-04-19T01:03:58.601 INFO:teuthology.orchestra.run.smithi033.stdout: "paxos_max_join_drift": "10", 2024-04-19T01:03:58.601 INFO:teuthology.orchestra.run.smithi033.stdout: "paxos_min": "500", 2024-04-19T01:03:58.601 INFO:teuthology.orchestra.run.smithi033.stdout: "paxos_min_wait": "0.050000", 2024-04-19T01:03:58.601 INFO:teuthology.orchestra.run.smithi033.stdout: "paxos_propose_interval": "1.000000", 2024-04-19T01:03:58.601 INFO:teuthology.orchestra.run.smithi033.stdout: "paxos_service_trim_max": "500", 2024-04-19T01:03:58.601 INFO:teuthology.orchestra.run.smithi033.stdout: "paxos_service_trim_max_multiplier": "20", 2024-04-19T01:03:58.601 INFO:teuthology.orchestra.run.smithi033.stdout: "paxos_service_trim_min": "250", 2024-04-19T01:03:58.601 INFO:teuthology.orchestra.run.smithi033.stdout: "paxos_stash_full_interval": "25", 2024-04-19T01:03:58.601 INFO:teuthology.orchestra.run.smithi033.stdout: "paxos_trim_max": "500", 2024-04-19T01:03:58.602 INFO:teuthology.orchestra.run.smithi033.stdout: "paxos_trim_min": "250", 2024-04-19T01:03:58.602 INFO:teuthology.orchestra.run.smithi033.stdout: "perf": "true", 2024-04-19T01:03:58.602 INFO:teuthology.orchestra.run.smithi033.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-04-19T01:03:58.602 INFO:teuthology.orchestra.run.smithi033.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-04-19T01:03:58.602 INFO:teuthology.orchestra.run.smithi033.stdout: "plugin_dir": "/usr/lib/ceph", 2024-04-19T01:03:58.602 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "-", 2024-04-19T01:03:58.602 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrv": "", 2024-04-19T01:03:58.602 INFO:teuthology.orchestra.run.smithi033.stdout: "public_bind_addr": "-", 2024-04-19T01:03:58.602 INFO:teuthology.orchestra.run.smithi033.stdout: "public_network": "", 2024-04-19T01:03:58.602 INFO:teuthology.orchestra.run.smithi033.stdout: "public_network_interface": "", 2024-04-19T01:03:58.602 INFO:teuthology.orchestra.run.smithi033.stdout: "qat_compressor_enabled": "false", 2024-04-19T01:03:58.602 INFO:teuthology.orchestra.run.smithi033.stdout: "qat_compressor_session_max_number": "256", 2024-04-19T01:03:58.602 INFO:teuthology.orchestra.run.smithi033.stdout: "rados_mon_op_timeout": "0", 2024-04-19T01:03:58.602 INFO:teuthology.orchestra.run.smithi033.stdout: "rados_osd_op_timeout": "0", 2024-04-19T01:03:58.602 INFO:teuthology.orchestra.run.smithi033.stdout: "rados_tracing": "false", 2024-04-19T01:03:58.603 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_atime_update_interval": "60", 2024-04-19T01:03:58.603 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-04-19T01:03:58.603 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_balance_parent_reads": "false", 2024-04-19T01:03:58.603 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_balance_snap_reads": "false", 2024-04-19T01:03:58.603 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_blkin_trace_all": "false", 2024-04-19T01:03:58.603 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_blocklist_expire_seconds": "0", 2024-04-19T01:03:58.603 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_blocklist_on_break_lock": "true", 2024-04-19T01:03:58.603 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_cache": "true", 2024-04-19T01:03:58.603 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_cache_block_writes_upfront": "false", 2024-04-19T01:03:58.603 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_cache_max_dirty": "25165824", 2024-04-19T01:03:58.603 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-04-19T01:03:58.603 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_cache_max_dirty_object": "0", 2024-04-19T01:03:58.603 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_cache_policy": "writearound", 2024-04-19T01:03:58.603 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_cache_size": "33554432", 2024-04-19T01:03:58.603 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_cache_target_dirty": "16777216", 2024-04-19T01:03:58.604 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-04-19T01:03:58.604 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_clone_copy_on_read": "false", 2024-04-19T01:03:58.604 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_compression_hint": "none", 2024-04-19T01:03:58.604 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_concurrent_management_ops": "10", 2024-04-19T01:03:58.604 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-04-19T01:03:58.604 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_default_clone_format": "auto", 2024-04-19T01:03:58.604 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_default_data_pool": "", 2024-04-19T01:03:58.604 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_default_features": "61", 2024-04-19T01:03:58.604 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_default_format": "2", 2024-04-19T01:03:58.604 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_default_map_options": "", 2024-04-19T01:03:58.604 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_default_order": "22", 2024-04-19T01:03:58.604 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_default_pool": "rbd", 2024-04-19T01:03:58.605 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-04-19T01:03:58.605 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_default_stripe_count": "0", 2024-04-19T01:03:58.605 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_default_stripe_unit": "0", 2024-04-19T01:03:58.605 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_disable_zero_copy_writes": "true", 2024-04-19T01:03:58.605 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_discard_granularity_bytes": "65536", 2024-04-19T01:03:58.605 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-04-19T01:03:58.605 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_enable_alloc_hint": "true", 2024-04-19T01:03:58.605 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-04-19T01:03:58.605 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_io_scheduler": "simple", 2024-04-19T01:03:58.605 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-04-19T01:03:58.605 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_journal_commit_age": "5.000000", 2024-04-19T01:03:58.605 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-04-19T01:03:58.605 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-04-19T01:03:58.605 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-04-19T01:03:58.605 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-04-19T01:03:58.606 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_journal_object_flush_interval": "0", 2024-04-19T01:03:58.606 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-04-19T01:03:58.606 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-04-19T01:03:58.606 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_journal_order": "24", 2024-04-19T01:03:58.606 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_journal_pool": "", 2024-04-19T01:03:58.606 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_journal_splay_width": "4", 2024-04-19T01:03:58.606 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_localize_parent_reads": "false", 2024-04-19T01:03:58.606 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_localize_snap_reads": "false", 2024-04-19T01:03:58.606 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-04-19T01:03:58.606 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-04-19T01:03:58.606 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-04-19T01:03:58.606 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-04-19T01:03:58.606 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-04-19T01:03:58.606 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-04-19T01:03:58.607 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_image_policy_type": "simple", 2024-04-19T01:03:58.607 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-04-19T01:03:58.607 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-04-19T01:03:58.607 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-04-19T01:03:58.607 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-04-19T01:03:58.607 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-04-19T01:03:58.607 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-04-19T01:03:58.607 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-04-19T01:03:58.607 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_memory_autotune": "true", 2024-04-19T01:03:58.607 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_memory_base": "805306368", 2024-04-19T01:03:58.607 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-04-19T01:03:58.607 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-04-19T01:03:58.607 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-04-19T01:03:58.607 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-04-19T01:03:58.607 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_memory_target": "4294967296", 2024-04-19T01:03:58.608 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-04-19T01:03:58.608 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-04-19T01:03:58.608 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-04-19T01:03:58.608 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirroring_delete_delay": "0", 2024-04-19T01:03:58.608 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-04-19T01:03:58.608 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirroring_replay_delay": "0", 2024-04-19T01:03:58.608 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-04-19T01:03:58.608 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-04-19T01:03:58.608 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_move_to_trash_on_remove": "false", 2024-04-19T01:03:58.608 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-04-19T01:03:58.608 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mtime_update_interval": "60", 2024-04-19T01:03:58.608 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_non_blocking_aio": "true", 2024-04-19T01:03:58.608 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_op_thread_timeout": "60", 2024-04-19T01:03:58.608 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_op_threads": "1", 2024-04-19T01:03:58.608 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_parent_cache_enabled": "false", 2024-04-19T01:03:58.608 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_persistent_cache_mode": "disabled", 2024-04-19T01:03:58.609 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_persistent_cache_path": "/tmp", 2024-04-19T01:03:58.609 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_persistent_cache_size": "1073741824", 2024-04-19T01:03:58.609 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_plugins": "", 2024-04-19T01:03:58.609 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_bps_burst": "0", 2024-04-19T01:03:58.609 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-04-19T01:03:58.609 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_bps_limit": "0", 2024-04-19T01:03:58.609 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_exclude_ops": "0", 2024-04-19T01:03:58.609 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_iops_burst": "0", 2024-04-19T01:03:58.609 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-04-19T01:03:58.609 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_iops_limit": "0", 2024-04-19T01:03:58.609 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_read_bps_burst": "0", 2024-04-19T01:03:58.609 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-04-19T01:03:58.609 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_read_bps_limit": "0", 2024-04-19T01:03:58.609 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_read_iops_burst": "0", 2024-04-19T01:03:58.610 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-04-19T01:03:58.610 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_read_iops_limit": "0", 2024-04-19T01:03:58.610 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_schedule_tick_min": "50", 2024-04-19T01:03:58.610 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_write_bps_burst": "0", 2024-04-19T01:03:58.610 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-04-19T01:03:58.610 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_write_bps_limit": "0", 2024-04-19T01:03:58.610 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_write_iops_burst": "0", 2024-04-19T01:03:58.610 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-04-19T01:03:58.610 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_write_iops_limit": "0", 2024-04-19T01:03:58.610 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_quiesce_notification_attempts": "10", 2024-04-19T01:03:58.610 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_read_from_replica_policy": "default", 2024-04-19T01:03:58.610 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-04-19T01:03:58.610 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_readahead_max_bytes": "524288", 2024-04-19T01:03:58.611 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_readahead_trigger_requests": "10", 2024-04-19T01:03:58.611 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_request_timed_out_seconds": "30", 2024-04-19T01:03:58.611 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_skip_partial_discard": "true", 2024-04-19T01:03:58.611 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-04-19T01:03:58.611 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_tracing": "false", 2024-04-19T01:03:58.611 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_validate_names": "true", 2024-04-19T01:03:58.611 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_validate_pool": "true", 2024-04-19T01:03:58.611 INFO:teuthology.orchestra.run.smithi033.stdout: "restapi_base_url": "", 2024-04-19T01:03:58.611 INFO:teuthology.orchestra.run.smithi033.stdout: "restapi_log_level": "", 2024-04-19T01:03:58.611 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_acl_grants_max_num": "100", 2024-04-19T01:03:58.611 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_admin_entry": "admin", 2024-04-19T01:03:58.611 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-04-19T01:03:58.611 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_backend_store": "rados", 2024-04-19T01:03:58.611 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_barbican_url": "", 2024-04-19T01:03:58.611 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_beast_enable_async": "true", 2024-04-19T01:03:58.612 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_bucket_counters_cache": "false", 2024-04-19T01:03:58.612 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-04-19T01:03:58.612 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-04-19T01:03:58.612 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-04-19T01:03:58.612 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_bucket_index_max_aio": "128", 2024-04-19T01:03:58.612 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-04-19T01:03:58.612 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-04-19T01:03:58.612 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_bucket_quota_ttl": "600", 2024-04-19T01:03:58.612 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-04-19T01:03:58.612 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_cache_enabled": "true", 2024-04-19T01:03:58.612 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_cache_expiry_interval": "900", 2024-04-19T01:03:58.612 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_cache_lru_size": "10000", 2024-04-19T01:03:58.612 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_config_store": "rados", 2024-04-19T01:03:58.612 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_content_length_compat": "false", 2024-04-19T01:03:58.612 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_copy_obj_progress": "true", 2024-04-19T01:03:58.613 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-04-19T01:03:58.613 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_cors_rules_max_num": "100", 2024-04-19T01:03:58.613 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_cross_domain_policy": "", 2024-04-19T01:03:58.613 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_default_encryption_key": "", 2024-04-19T01:03:58.613 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_kmip_addr": "", 2024-04-19T01:03:58.613 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_kmip_ca_path": "", 2024-04-19T01:03:58.613 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_kmip_client_cert": "", 2024-04-19T01:03:58.613 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_kmip_client_key": "", 2024-04-19T01:03:58.613 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-04-19T01:03:58.613 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_kmip_password": "", 2024-04-19T01:03:58.613 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-04-19T01:03:58.613 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_kmip_username": "", 2024-04-19T01:03:58.613 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_require_ssl": "true", 2024-04-19T01:03:58.613 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-04-19T01:03:58.614 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-04-19T01:03:58.614 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-04-19T01:03:58.614 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-04-19T01:03:58.614 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-04-19T01:03:58.614 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-04-19T01:03:58.614 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-04-19T01:03:58.614 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-04-19T01:03:58.614 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-04-19T01:03:58.614 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-04-19T01:03:58.614 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-04-19T01:03:58.614 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-04-19T01:03:58.614 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-04-19T01:03:58.614 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-04-19T01:03:58.614 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_suppress_logs": "true", 2024-04-19T01:03:58.615 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_vault_addr": "", 2024-04-19T01:03:58.615 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_vault_auth": "token", 2024-04-19T01:03:58.615 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_vault_namespace": "", 2024-04-19T01:03:58.615 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_vault_prefix": "", 2024-04-19T01:03:58.615 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-04-19T01:03:58.615 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-04-19T01:03:58.615 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-04-19T01:03:58.615 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-04-19T01:03:58.615 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_vault_token_file": "", 2024-04-19T01:03:58.615 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-04-19T01:03:58.615 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_curl_buffersize": "524288", 2024-04-19T01:03:58.615 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_curl_low_speed_limit": "1024", 2024-04-19T01:03:58.615 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_curl_low_speed_time": "300", 2024-04-19T01:03:58.615 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_curl_tcp_keepalive": "0", 2024-04-19T01:03:58.615 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-04-19T01:03:58.616 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-04-19T01:03:58.616 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-04-19T01:03:58.616 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-04-19T01:03:58.616 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-04-19T01:03:58.616 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_d3n_l1_fadvise": "4", 2024-04-19T01:03:58.616 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-04-19T01:03:58.616 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-04-19T01:03:58.616 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-04-19T01:03:58.616 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_d4n_host": "127.0.0.1", 2024-04-19T01:03:58.616 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-04-19T01:03:58.616 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-04-19T01:03:58.616 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-04-19T01:03:58.616 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_d4n_l1_fadvise": "4", 2024-04-19T01:03:58.616 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-04-19T01:03:58.616 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-04-19T01:03:58.617 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_d4n_port": "6379", 2024-04-19T01:03:58.617 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-04-19T01:03:58.617 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_data_log_changes_size": "1000", 2024-04-19T01:03:58.617 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_data_log_num_shards": "128", 2024-04-19T01:03:58.617 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-04-19T01:03:58.617 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_data_log_window": "30", 2024-04-19T01:03:58.617 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_data_notify_interval_msec": "0", 2024-04-19T01:03:58.617 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_data_sync_poll_interval": "20", 2024-04-19T01:03:58.617 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_data_sync_spawn_window": "20", 2024-04-19T01:03:58.617 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-04-19T01:03:58.617 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-04-19T01:03:58.617 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-04-19T01:03:58.617 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_default_data_log_backing": "fifo", 2024-04-19T01:03:58.617 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-04-19T01:03:58.618 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_default_region_info_oid": "default.region", 2024-04-19T01:03:58.618 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-04-19T01:03:58.618 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-04-19T01:03:58.618 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_defer_to_bucket_acls": "", 2024-04-19T01:03:58.618 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-04-19T01:03:58.618 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-04-19T01:03:58.618 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-04-19T01:03:58.618 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-04-19T01:03:58.618 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-04-19T01:03:58.618 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-04-19T01:03:58.618 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-04-19T01:03:58.618 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-04-19T01:03:58.618 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dmclock_data_res": "500.000000", 2024-04-19T01:03:58.618 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-04-19T01:03:58.618 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-04-19T01:03:58.619 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-04-19T01:03:58.619 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-04-19T01:03:58.619 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dns_name": "", 2024-04-19T01:03:58.619 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dns_s3website_name": "", 2024-04-19T01:03:58.619 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dynamic_resharding": "true", 2024-04-19T01:03:58.619 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-04-19T01:03:58.619 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_enable_gc_threads": "true", 2024-04-19T01:03:58.619 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_enable_lc_threads": "true", 2024-04-19T01:03:58.619 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_enable_ops_log": "false", 2024-04-19T01:03:58.619 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_enable_quota_threads": "true", 2024-04-19T01:03:58.619 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_enable_static_website": "false", 2024-04-19T01:03:58.619 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_enable_usage_log": "false", 2024-04-19T01:03:58.619 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_enforce_swift_acls": "true", 2024-04-19T01:03:58.619 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_exit_timeout_secs": "120", 2024-04-19T01:03:58.619 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_expose_bucket": "false", 2024-04-19T01:03:58.620 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_extended_http_attrs": "", 2024-04-19T01:03:58.620 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_filter": "none", 2024-04-19T01:03:58.620 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-04-19T01:03:58.620 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_frontends": "beast port=7480", 2024-04-19T01:03:58.620 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_gc_max_concurrent_io": "10", 2024-04-19T01:03:58.620 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_gc_max_deferred": "50", 2024-04-19T01:03:58.620 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-04-19T01:03:58.620 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_gc_max_objs": "32", 2024-04-19T01:03:58.620 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_gc_max_queue_size": "134213632", 2024-04-19T01:03:58.620 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_gc_max_trim_chunk": "16", 2024-04-19T01:03:58.620 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_gc_obj_min_wait": "7200", 2024-04-19T01:03:58.620 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_gc_processor_max_time": "3600", 2024-04-19T01:03:58.620 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_gc_processor_period": "3600", 2024-04-19T01:03:58.620 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-04-19T01:03:58.621 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_get_obj_window_size": "16777216", 2024-04-19T01:03:58.621 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_healthcheck_disabling_path": "", 2024-04-19T01:03:58.621 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_ignore_get_invalid_range": "false", 2024-04-19T01:03:58.621 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_init_timeout": "300", 2024-04-19T01:03:58.621 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-04-19T01:03:58.621 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-04-19T01:03:58.621 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_kafka_connection_idle": "300", 2024-04-19T01:03:58.621 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_kafka_message_timeout": "5000", 2024-04-19T01:03:58.621 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_kafka_sleep_timeout": "10", 2024-04-19T01:03:58.621 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-04-19T01:03:58.621 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-04-19T01:03:58.621 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-04-19T01:03:58.621 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_admin_domain": "", 2024-04-19T01:03:58.621 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_admin_password": "", 2024-04-19T01:03:58.622 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_admin_password_path": "", 2024-04-19T01:03:58.622 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_admin_project": "", 2024-04-19T01:03:58.622 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_admin_tenant": "", 2024-04-19T01:03:58.622 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_admin_token": "", 2024-04-19T01:03:58.622 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_admin_token_path": "", 2024-04-19T01:03:58.622 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_admin_user": "", 2024-04-19T01:03:58.622 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_api_version": "2", 2024-04-19T01:03:58.622 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_barbican_domain": "", 2024-04-19T01:03:58.622 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_barbican_password": "", 2024-04-19T01:03:58.622 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_barbican_project": "", 2024-04-19T01:03:58.622 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_barbican_tenant": "", 2024-04-19T01:03:58.622 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_barbican_user": "", 2024-04-19T01:03:58.622 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-04-19T01:03:58.623 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_implicit_tenants": "false", 2024-04-19T01:03:58.623 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-04-19T01:03:58.623 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_service_token_enabled": "false", 2024-04-19T01:03:58.623 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_token_cache_size": "10000", 2024-04-19T01:03:58.623 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_url": "", 2024-04-19T01:03:58.623 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_verify_ssl": "true", 2024-04-19T01:03:58.623 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_lc_debug_interval": "-1", 2024-04-19T01:03:58.623 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_lc_lock_max_time": "90", 2024-04-19T01:03:58.623 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_lc_max_objs": "32", 2024-04-19T01:03:58.623 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_lc_max_rules": "1000", 2024-04-19T01:03:58.623 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_lc_max_worker": "3", 2024-04-19T01:03:58.623 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_lc_max_wp_worker": "3", 2024-04-19T01:03:58.623 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_lc_thread_delay": "0", 2024-04-19T01:03:58.623 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-04-19T01:03:58.624 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_ldap_dnattr": "uid", 2024-04-19T01:03:58.624 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-04-19T01:03:58.624 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_ldap_searchfilter": "", 2024-04-19T01:03:58.624 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-04-19T01:03:58.624 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_ldap_uri": "", 2024-04-19T01:03:58.624 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_lfuda_sync_frequency": "60", 2024-04-19T01:03:58.624 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-04-19T01:03:58.624 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-04-19T01:03:58.624 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-04-19T01:03:58.624 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_local_cache_address": "127.0.0.1:6379", 2024-04-19T01:03:58.624 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_log_http_headers": "", 2024-04-19T01:03:58.624 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_log_nonexistent_bucket": "false", 2024-04-19T01:03:58.624 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-04-19T01:03:58.624 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_log_object_name_utc": "false", 2024-04-19T01:03:58.624 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-04-19T01:03:58.625 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-04-19T01:03:58.625 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_max_attr_name_len": "0", 2024-04-19T01:03:58.625 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_max_attr_size": "0", 2024-04-19T01:03:58.625 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_max_attrs_num_in_req": "0", 2024-04-19T01:03:58.625 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_max_chunk_size": "4194304", 2024-04-19T01:03:58.625 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_max_concurrent_requests": "1024", 2024-04-19T01:03:58.625 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-04-19T01:03:58.625 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_max_dynamic_shards": "1999", 2024-04-19T01:03:58.625 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_max_listing_results": "1000", 2024-04-19T01:03:58.625 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_max_notify_retries": "10", 2024-04-19T01:03:58.625 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_max_objs_per_shard": "100000", 2024-04-19T01:03:58.625 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_max_put_param_size": "1048576", 2024-04-19T01:03:58.625 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_max_put_size": "5368709120", 2024-04-19T01:03:58.625 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_max_slo_entries": "1000", 2024-04-19T01:03:58.625 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_md_log_max_shards": "64", 2024-04-19T01:03:58.626 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_md_notify_interval_msec": "200", 2024-04-19T01:03:58.626 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_meta_sync_poll_interval": "20", 2024-04-19T01:03:58.626 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_meta_sync_spawn_window": "20", 2024-04-19T01:03:58.626 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-04-19T01:03:58.626 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_mp_lock_max_time": "600", 2024-04-19T01:03:58.626 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-04-19T01:03:58.626 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_multipart_min_part_size": "5242880", 2024-04-19T01:03:58.626 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-04-19T01:03:58.626 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-04-19T01:03:58.626 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_nfs_fhcache_size": "2017", 2024-04-19T01:03:58.626 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-04-19T01:03:58.626 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-04-19T01:03:58.626 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_nfs_lru_lanes": "5", 2024-04-19T01:03:58.626 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_nfs_max_gc": "300", 2024-04-19T01:03:58.627 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-04-19T01:03:58.627 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_nfs_run_gc_threads": "false", 2024-04-19T01:03:58.627 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_nfs_run_lc_threads": "false", 2024-04-19T01:03:58.627 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_nfs_run_quota_threads": "false", 2024-04-19T01:03:58.627 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_nfs_run_sync_thread": "false", 2024-04-19T01:03:58.627 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-04-19T01:03:58.627 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-04-19T01:03:58.627 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_num_async_rados_threads": "32", 2024-04-19T01:03:58.627 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_num_control_oids": "8", 2024-04-19T01:03:58.627 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_numa_node": "-1", 2024-04-19T01:03:58.627 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_obj_stripe_size": "4194304", 2024-04-19T01:03:58.627 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-04-19T01:03:58.627 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_objexp_chunk_size": "100", 2024-04-19T01:03:58.628 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_objexp_gc_interval": "600", 2024-04-19T01:03:58.628 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_objexp_hints_num_shards": "127", 2024-04-19T01:03:58.628 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-04-19T01:03:58.628 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-04-19T01:03:58.628 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_op_thread_timeout": "600", 2024-04-19T01:03:58.628 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_op_tracing": "false", 2024-04-19T01:03:58.628 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_opa_token": "", 2024-04-19T01:03:58.629 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_opa_url": "", 2024-04-19T01:03:58.629 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_opa_verify_ssl": "true", 2024-04-19T01:03:58.629 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-04-19T01:03:58.629 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-04-19T01:03:58.629 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_ops_log_rados": "false", 2024-04-19T01:03:58.629 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_ops_log_socket_path": "", 2024-04-19T01:03:58.629 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-04-19T01:03:58.629 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-04-19T01:03:58.629 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-04-19T01:03:58.629 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_period_push_interval": "2.000000", 2024-04-19T01:03:58.629 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_period_push_interval_max": "30.000000", 2024-04-19T01:03:58.630 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_period_root_pool": ".rgw.root", 2024-04-19T01:03:58.630 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-04-19T01:03:58.630 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-04-19T01:03:58.630 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_posix_cache_lanes": "3", 2024-04-19T01:03:58.630 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-04-19T01:03:58.630 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_posix_cache_max_buckets": "100", 2024-04-19T01:03:58.630 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_posix_cache_partitions": "3", 2024-04-19T01:03:58.630 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-04-19T01:03:58.630 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_print_continue": "true", 2024-04-19T01:03:58.630 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_print_prohibited_content_length": "false", 2024-04-19T01:03:58.630 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-04-19T01:03:58.630 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-04-19T01:03:58.630 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-04-19T01:03:58.630 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-04-19T01:03:58.631 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_rados_tracing": "false", 2024-04-19T01:03:58.631 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_realm": "", 2024-04-19T01:03:58.631 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_realm_id": "", 2024-04-19T01:03:58.631 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-04-19T01:03:58.631 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_region": "", 2024-04-19T01:03:58.631 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_region_root_pool": ".rgw.root", 2024-04-19T01:03:58.631 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_relaxed_region_enforcement": "false", 2024-04-19T01:03:58.631 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-04-19T01:03:58.631 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-04-19T01:03:58.631 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_request_uri": "", 2024-04-19T01:03:58.631 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_reshard_batch_size": "64", 2024-04-19T01:03:58.631 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-04-19T01:03:58.631 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_reshard_max_aio": "128", 2024-04-19T01:03:58.631 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_reshard_num_logs": "16", 2024-04-19T01:03:58.631 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_reshard_thread_interval": "600", 2024-04-19T01:03:58.632 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_resolve_cname": "false", 2024-04-19T01:03:58.632 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_rest_getusage_op_compat": "false", 2024-04-19T01:03:58.632 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_run_sync_thread": "true", 2024-04-19T01:03:58.633 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-04-19T01:03:58.633 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-04-19T01:03:58.633 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_s3_auth_use_keystone": "false", 2024-04-19T01:03:58.633 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_s3_auth_use_ldap": "false", 2024-04-19T01:03:58.633 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_s3_auth_use_rados": "true", 2024-04-19T01:03:58.634 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_s3_auth_use_sts": "false", 2024-04-19T01:03:58.634 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-04-19T01:03:58.634 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_s3_success_create_obj_status": "0", 2024-04-19T01:03:58.634 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-04-19T01:03:58.634 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_scheduler_type": "throttler", 2024-04-19T01:03:58.634 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_script_uri": "", 2024-04-19T01:03:58.634 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_service_provider_name": "", 2024-04-19T01:03:58.634 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-04-19T01:03:58.634 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sts_client_id": "", 2024-04-19T01:03:58.634 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sts_client_secret": "", 2024-04-19T01:03:58.634 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sts_entry": "sts", 2024-04-19T01:03:58.634 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sts_key": "sts", 2024-04-19T01:03:58.634 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sts_max_session_duration": "43200", 2024-04-19T01:03:58.634 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sts_min_session_duration": "900", 2024-04-19T01:03:58.635 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sts_token_introspection_url": "", 2024-04-19T01:03:58.635 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_swift_account_in_url": "false", 2024-04-19T01:03:58.635 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_swift_auth_entry": "auth", 2024-04-19T01:03:58.635 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_swift_auth_url": "", 2024-04-19T01:03:58.635 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_swift_custom_header": "", 2024-04-19T01:03:58.635 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_swift_enforce_content_length": "false", 2024-04-19T01:03:58.635 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_swift_need_stats": "true", 2024-04-19T01:03:58.635 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_swift_tenant_name": "", 2024-04-19T01:03:58.635 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_swift_token_expiration": "86400", 2024-04-19T01:03:58.635 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_swift_url": "", 2024-04-19T01:03:58.635 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_swift_url_prefix": "swift", 2024-04-19T01:03:58.635 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_swift_versioning_enabled": "false", 2024-04-19T01:03:58.635 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-04-19T01:03:58.635 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-04-19T01:03:58.635 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sync_lease_period": "120", 2024-04-19T01:03:58.635 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-04-19T01:03:58.636 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sync_log_trim_interval": "1200", 2024-04-19T01:03:58.636 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-04-19T01:03:58.636 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-04-19T01:03:58.636 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-04-19T01:03:58.636 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sync_obj_etag_verify": "false", 2024-04-19T01:03:58.636 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sync_trace_history_size": "4096", 2024-04-19T01:03:58.636 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-04-19T01:03:58.636 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-04-19T01:03:58.636 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_thread_pool_size": "512", 2024-04-19T01:03:58.636 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_topic_persistency_max_retries": "0", 2024-04-19T01:03:58.636 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-04-19T01:03:58.636 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-04-19T01:03:58.636 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_topic_require_publish_policy": "false", 2024-04-19T01:03:58.636 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_torrent_comment": "", 2024-04-19T01:03:58.636 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_torrent_createby": "", 2024-04-19T01:03:58.636 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_torrent_encoding": "", 2024-04-19T01:03:58.637 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_torrent_flag": "false", 2024-04-19T01:03:58.637 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_torrent_max_size": "5368709120", 2024-04-19T01:03:58.637 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_torrent_origin": "", 2024-04-19T01:03:58.637 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_torrent_sha_unit": "524288", 2024-04-19T01:03:58.637 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_torrent_tracker": "", 2024-04-19T01:03:58.637 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_trust_forwarded_https": "false", 2024-04-19T01:03:58.637 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-04-19T01:03:58.637 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_usage_log_tick_interval": "30", 2024-04-19T01:03:58.637 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_usage_max_shards": "32", 2024-04-19T01:03:58.637 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_usage_max_user_shards": "1", 2024-04-19T01:03:58.637 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_use_opa_authz": "false", 2024-04-19T01:03:58.637 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_user_counters_cache": "false", 2024-04-19T01:03:58.637 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_user_counters_cache_size": "10000", 2024-04-19T01:03:58.637 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-04-19T01:03:58.638 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_user_default_quota_max_size": "-1", 2024-04-19T01:03:58.638 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_user_max_buckets": "1000", 2024-04-19T01:03:58.638 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_user_policies_max_num": "100", 2024-04-19T01:03:58.638 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-04-19T01:03:58.638 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-04-19T01:03:58.638 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_user_quota_sync_interval": "86400", 2024-04-19T01:03:58.638 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-04-19T01:03:58.638 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_user_unique_email": "true", 2024-04-19T01:03:58.638 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_verify_ssl": "true", 2024-04-19T01:03:58.638 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_website_routing_rules_max_num": "50", 2024-04-19T01:03:58.638 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_zone": "", 2024-04-19T01:03:58.638 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_zone_id": "", 2024-04-19T01:03:58.638 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-04-19T01:03:58.638 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_zonegroup": "", 2024-04-19T01:03:58.639 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_zonegroup_id": "", 2024-04-19T01:03:58.639 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-04-19T01:03:58.639 INFO:teuthology.orchestra.run.smithi033.stdout: "rgwlc_auto_session_clear": "true", 2024-04-19T01:03:58.639 INFO:teuthology.orchestra.run.smithi033.stdout: "rgwlc_skip_bucket_step": "false", 2024-04-19T01:03:58.639 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_block_size": "4096", 2024-04-19T01:03:58.639 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-04-19T01:03:58.639 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-04-19T01:03:58.639 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-04-19T01:03:58.639 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-04-19T01:03:58.639 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_cache_shard_bits": "4", 2024-04-19T01:03:58.639 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_cache_size": "536870912", 2024-04-19T01:03:58.639 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_cache_type": "binned_lru", 2024-04-19T01:03:58.639 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-04-19T01:03:58.640 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-04-19T01:03:58.640 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-04-19T01:03:58.640 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_collect_compaction_stats": "false", 2024-04-19T01:03:58.640 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_collect_extended_stats": "false", 2024-04-19T01:03:58.640 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_collect_memory_stats": "false", 2024-04-19T01:03:58.640 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-04-19T01:03:58.640 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_index_type": "binary_search", 2024-04-19T01:03:58.640 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_log_to_ceph_log": "true", 2024-04-19T01:03:58.640 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_metadata_block_size": "4096", 2024-04-19T01:03:58.640 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_partition_filters": "false", 2024-04-19T01:03:58.640 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_perf": "false", 2024-04-19T01:03:58.640 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-04-19T01:03:58.640 INFO:teuthology.orchestra.run.smithi033.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-04-19T01:03:58.640 INFO:teuthology.orchestra.run.smithi033.stdout: "rotating_keys_renewal_timeout": "10", 2024-04-19T01:03:58.640 INFO:teuthology.orchestra.run.smithi033.stdout: "run_dir": "/var/run/ceph", 2024-04-19T01:03:58.641 INFO:teuthology.orchestra.run.smithi033.stdout: "seastore_block_create": "true", 2024-04-19T01:03:58.641 INFO:teuthology.orchestra.run.smithi033.stdout: "seastore_cache_lru_size": "67108864", 2024-04-19T01:03:58.641 INFO:teuthology.orchestra.run.smithi033.stdout: "seastore_cbjournal_size": "5368709120", 2024-04-19T01:03:58.641 INFO:teuthology.orchestra.run.smithi033.stdout: "seastore_data_delta_based_overwrite": "0", 2024-04-19T01:03:58.641 INFO:teuthology.orchestra.run.smithi033.stdout: "seastore_default_max_object_size": "16777216", 2024-04-19T01:03:58.641 INFO:teuthology.orchestra.run.smithi033.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-04-19T01:03:58.641 INFO:teuthology.orchestra.run.smithi033.stdout: "seastore_device_size": "53687091200", 2024-04-19T01:03:58.641 INFO:teuthology.orchestra.run.smithi033.stdout: "seastore_journal_batch_capacity": "16", 2024-04-19T01:03:58.641 INFO:teuthology.orchestra.run.smithi033.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-04-19T01:03:58.641 INFO:teuthology.orchestra.run.smithi033.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-04-19T01:03:58.641 INFO:teuthology.orchestra.run.smithi033.stdout: "seastore_journal_iodepth_limit": "5", 2024-04-19T01:03:58.641 INFO:teuthology.orchestra.run.smithi033.stdout: "seastore_main_device_type": "SSD", 2024-04-19T01:03:58.641 INFO:teuthology.orchestra.run.smithi033.stdout: "seastore_max_concurrent_transactions": "8", 2024-04-19T01:03:58.641 INFO:teuthology.orchestra.run.smithi033.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-04-19T01:03:58.642 INFO:teuthology.orchestra.run.smithi033.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-04-19T01:03:58.642 INFO:teuthology.orchestra.run.smithi033.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-04-19T01:03:58.642 INFO:teuthology.orchestra.run.smithi033.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-04-19T01:03:58.642 INFO:teuthology.orchestra.run.smithi033.stdout: "seastore_segment_size": "67108864", 2024-04-19T01:03:58.642 INFO:teuthology.orchestra.run.smithi033.stdout: "set_keepcaps": "false", 2024-04-19T01:03:58.642 INFO:teuthology.orchestra.run.smithi033.stdout: "setgroup": "", 2024-04-19T01:03:58.642 INFO:teuthology.orchestra.run.smithi033.stdout: "setuser": "", 2024-04-19T01:03:58.642 INFO:teuthology.orchestra.run.smithi033.stdout: "setuser_match_path": "", 2024-04-19T01:03:58.642 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_misplaced_ratio": "0.050000", 2024-04-19T01:03:58.642 INFO:teuthology.orchestra.run.smithi033.stdout: "thp": "false", 2024-04-19T01:03:58.642 INFO:teuthology.orchestra.run.smithi033.stdout: "threadpool_default_timeout": "60", 2024-04-19T01:03:58.642 INFO:teuthology.orchestra.run.smithi033.stdout: "threadpool_empty_queue_max_wait": "2", 2024-04-19T01:03:58.642 INFO:teuthology.orchestra.run.smithi033.stdout: "throttler_perf_counter": "true" 2024-04-19T01:03:58.642 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:03:58.645 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-04-19T01:03:58.788 INFO:teuthology.orchestra.run.smithi033.stdout:mon.a: {} 2024-04-19T01:03:58.789 INFO:teuthology.orchestra.run.smithi033.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-19T01:03:58.801 INFO:teuthology.orchestra.run.smithi033.stdout:mon.b: {} 2024-04-19T01:03:58.802 INFO:teuthology.orchestra.run.smithi033.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-19T01:03:58.815 INFO:teuthology.orchestra.run.smithi033.stdout:mon.c: {} 2024-04-19T01:03:58.815 INFO:teuthology.orchestra.run.smithi033.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-19T01:03:58.835 DEBUG:teuthology.orchestra.run.smithi155:> type ceph-objectstore-tool 2024-04-19T01:03:58.843 INFO:teuthology.orchestra.run.smithi155.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-19T01:03:58.843 DEBUG:teuthology.orchestra.run.smithi118:> type ceph-objectstore-tool 2024-04-19T01:03:58.849 INFO:teuthology.orchestra.run.smithi118.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-19T01:03:58.850 DEBUG:teuthology.orchestra.run.smithi033:> type ceph-objectstore-tool 2024-04-19T01:03:58.854 INFO:teuthology.orchestra.run.smithi033.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-19T01:03:58.855 DEBUG:teuthology.orchestra.run.smithi110:> type ceph-objectstore-tool 2024-04-19T01:03:58.861 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-19T01:03:58.862 INFO:teuthology.run_tasks:Running task rados... 2024-04-19T01:03:58.875 INFO:tasks.rados:Beginning rados... 2024-04-19T01:03:58.875 INFO:tasks.rados:config is {'clients': ['client.0'], 'ec_pool': True, 'erasure_code_profile': {'c': 2, 'crush-failure-domain': 'osd', 'k': 4, 'm': 3, 'name': 'shecprofile', 'plugin': 'shec'}, 'objects': 50, 'op_weights': {'append': 100, 'copy_from': 50, 'delete': 50, 'read': 100, 'rmattr': 25, 'rollback': 50, 'setattr': 25, 'snap_create': 50, 'snap_remove': 50, 'write': 0}, 'ops': 400, 'write_append_excl': False} 2024-04-19T01:03:58.875 INFO:tasks.rados:overrides is {'admin_socket': {'branch': 'wip-yuri5-testing-2024-04-17-1400'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': '8b4ab94f9c30da63487fa2f241ce12f483378860'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8b4ab94f9c30da63487fa2f241ce12f483378860'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri5-testing-2024-04-17-1400', 'sha1': '8b4ab94f9c30da63487fa2f241ce12f483378860'}} 2024-04-19T01:03:58.876 INFO:tasks.rados:config is {'clients': ['client.0'], 'ec_pool': True, 'erasure_code_profile': {'c': 2, 'crush-failure-domain': 'osd', 'k': 4, 'm': 3, 'name': 'shecprofile', 'plugin': 'shec'}, 'objects': 50, 'op_weights': {'append': 100, 'copy_from': 50, 'delete': 50, 'read': 100, 'rmattr': 25, 'rollback': 50, 'setattr': 25, 'snap_create': 50, 'snap_remove': 50, 'write': 0}, 'ops': 400, 'write_append_excl': False} 2024-04-19T01:03:58.876 DEBUG:teuthology.run_tasks:Unwinding manager rados 2024-04-19T01:03:58.887 INFO:tasks.rados:joining rados 2024-04-19T01:03:58.887 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-04-19T01:03:58.887 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-04-19T01:03:58.887 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-19T01:03:58.887 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-19T01:03:58.887 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-19T01:03:58.888 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-04-19T01:03:58.889 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-04-19T01:03:58.889 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-19T01:03:58.889 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-04-19T01:03:58.889 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-19T01:03:58.890 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-04-19T01:03:58.890 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-19T01:03:58.890 INFO:tasks.rados:clients are ['client.0'] 2024-04-19T01:03:58.890 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set shecprofile c=2 crush-failure-domain=osd k=4 m=3 name=shecprofile plugin=shec 2024-04-19T01:03:58.905 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:03:58.908+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:03:58.990 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:03:58.992+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:03:59.022 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:03:59.022 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:03:59.022 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2024-04-19T01:03:59.023 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:03:59.047 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-19T01:03:59.092 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:03:59.093+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:03:59.138 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-04-19T01:03:59.139 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T01:03:59.178 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-04-19T01:03:59.179 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T01:03:59.192 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:03:59.193+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:03:59.196 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:03:59.196 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:03:59.209 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-19T01:03:59.210 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-04-19T01:03:59.210 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T01:03:59.245 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-04-19T01:03:59.246 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T01:03:59.266 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-04-19T01:03:59.267 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T01:03:59.274 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:03:59.274 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:03:59.274 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 28, 2024-04-19T01:03:59.274 INFO:teuthology.orchestra.run.smithi033.stdout: "fsid": "f11bd2d5-4456-4ffa-83c0-722dddd22ad3", 2024-04-19T01:03:59.274 INFO:teuthology.orchestra.run.smithi033.stdout: "created": "2024-04-19T01:03:36.766089+0000", 2024-04-19T01:03:59.274 INFO:teuthology.orchestra.run.smithi033.stdout: "modified": "2024-04-19T01:03:54.645990+0000", 2024-04-19T01:03:59.274 INFO:teuthology.orchestra.run.smithi033.stdout: "last_up_change": "2024-04-19T01:03:50.591452+0000", 2024-04-19T01:03:59.275 INFO:teuthology.orchestra.run.smithi033.stdout: "last_in_change": "2024-04-19T01:03:41.227644+0000", 2024-04-19T01:03:59.275 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-19T01:03:59.275 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_num": 5799936, 2024-04-19T01:03:59.275 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_set": [ 2024-04-19T01:03:59.275 INFO:teuthology.orchestra.run.smithi033.stdout: "pglog_hardlimit", 2024-04-19T01:03:59.275 INFO:teuthology.orchestra.run.smithi033.stdout: "purged_snapdirs", 2024-04-19T01:03:59.275 INFO:teuthology.orchestra.run.smithi033.stdout: "recovery_deletes", 2024-04-19T01:03:59.275 INFO:teuthology.orchestra.run.smithi033.stdout: "sortbitwise" 2024-04-19T01:03:59.275 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-04-19T01:03:59.275 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_version": 5, 2024-04-19T01:03:59.275 INFO:teuthology.orchestra.run.smithi033.stdout: "full_ratio": 0.94999998807907104, 2024-04-19T01:03:59.275 INFO:teuthology.orchestra.run.smithi033.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-19T01:03:59.275 INFO:teuthology.orchestra.run.smithi033.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-19T01:03:59.275 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_snapshot": "", 2024-04-19T01:03:59.276 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_max": 2, 2024-04-19T01:03:59.276 INFO:teuthology.orchestra.run.smithi033.stdout: "max_osd": 16, 2024-04-19T01:03:59.276 INFO:teuthology.orchestra.run.smithi033.stdout: "require_min_compat_client": "luminous", 2024-04-19T01:03:59.276 INFO:teuthology.orchestra.run.smithi033.stdout: "min_compat_client": "jewel", 2024-04-19T01:03:59.276 INFO:teuthology.orchestra.run.smithi033.stdout: "require_osd_release": "squid", 2024-04-19T01:03:59.276 INFO:teuthology.orchestra.run.smithi033.stdout: "allow_crimson": false, 2024-04-19T01:03:59.276 INFO:teuthology.orchestra.run.smithi033.stdout: "pools": [ 2024-04-19T01:03:59.276 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.276 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 1, 2024-04-19T01:03:59.276 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": ".mgr", 2024-04-19T01:03:59.276 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-04-19T01:03:49.571609+0000", 2024-04-19T01:03:59.276 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-04-19T01:03:59.276 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-04-19T01:03:59.276 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-04-19T01:03:59.276 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 2, 2024-04-19T01:03:59.277 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 1, 2024-04-19T01:03:59.277 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-04-19T01:03:59.277 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-04-19T01:03:59.277 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-04-19T01:03:59.277 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-04-19T01:03:59.277 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-19T01:03:59.277 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-04-19T01:03:59.277 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-04-19T01:03:59.277 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 1, 2024-04-19T01:03:59.277 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 1, 2024-04-19T01:03:59.277 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 1, 2024-04-19T01:03:59.277 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 1, 2024-04-19T01:03:59.277 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 1, 2024-04-19T01:03:59.279 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-04-19T01:03:59.279 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2024-04-19T01:03:59.279 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2024-04-19T01:03:59.279 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2024-04-19T01:03:59.280 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2024-04-19T01:03:59.280 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-04-19T01:03:59.280 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-04-19T01:03:59.280 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.280 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "25", 2024-04-19T01:03:59.280 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-04-19T01:03:59.280 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-19T01:03:59.280 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2024-04-19T01:03:59.280 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-04-19T01:03:59.280 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-04-19T01:03:59.281 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-04-19T01:03:59.281 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-04-19T01:03:59.281 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-04-19T01:03:59.281 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-04-19T01:03:59.281 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-04-19T01:03:59.281 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-04-19T01:03:59.281 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-04-19T01:03:59.281 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-04-19T01:03:59.281 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-04-19T01:03:59.281 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-04-19T01:03:59.281 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-04-19T01:03:59.281 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-04-19T01:03:59.281 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-04-19T01:03:59.282 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-19T01:03:59.282 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-19T01:03:59.282 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-19T01:03:59.282 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-04-19T01:03:59.282 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-04-19T01:03:59.282 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-04-19T01:03:59.282 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-04-19T01:03:59.282 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-04-19T01:03:59.282 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.282 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-04-19T01:03:59.282 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-04-19T01:03:59.282 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-04-19T01:03:59.282 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-04-19T01:03:59.282 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-04-19T01:03:59.283 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-04-19T01:03:59.283 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-04-19T01:03:59.283 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-04-19T01:03:59.283 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-04-19T01:03:59.283 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-04-19T01:03:59.283 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-04-19T01:03:59.283 INFO:teuthology.orchestra.run.smithi033.stdout: "options": { 2024-04-19T01:03:59.283 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_max": 32, 2024-04-19T01:03:59.283 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_min": 1 2024-04-19T01:03:59.283 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.283 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-04-19T01:03:59.283 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr": {} 2024-04-19T01:03:59.283 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.283 INFO:teuthology.orchestra.run.smithi033.stdout: "read_balance": { 2024-04-19T01:03:59.284 INFO:teuthology.orchestra.run.smithi033.stdout: "score_type": "Fair distribution", 2024-04-19T01:03:59.284 INFO:teuthology.orchestra.run.smithi033.stdout: "score_acting": 15.380000114440918, 2024-04-19T01:03:59.284 INFO:teuthology.orchestra.run.smithi033.stdout: "score_stable": 15.380000114440918, 2024-04-19T01:03:59.284 INFO:teuthology.orchestra.run.smithi033.stdout: "optimal_score": 0.12999999523162842, 2024-04-19T01:03:59.284 INFO:teuthology.orchestra.run.smithi033.stdout: "raw_score_acting": 2, 2024-04-19T01:03:59.284 INFO:teuthology.orchestra.run.smithi033.stdout: "raw_score_stable": 2, 2024-04-19T01:03:59.284 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity_weighted": 1, 2024-04-19T01:03:59.284 INFO:teuthology.orchestra.run.smithi033.stdout: "average_primary_affinity": 1, 2024-04-19T01:03:59.284 INFO:teuthology.orchestra.run.smithi033.stdout: "average_primary_affinity_weighted": 1 2024-04-19T01:03:59.284 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.284 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.284 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.284 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 2, 2024-04-19T01:03:59.284 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "rbd", 2024-04-19T01:03:59.285 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2024-04-19T01:03:51.913782+0000", 2024-04-19T01:03:59.285 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-04-19T01:03:59.285 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2024-04-19T01:03:59.285 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2024-04-19T01:03:59.285 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 2, 2024-04-19T01:03:59.285 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 1, 2024-04-19T01:03:59.285 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2024-04-19T01:03:59.285 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2024-04-19T01:03:59.285 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2024-04-19T01:03:59.285 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2024-04-19T01:03:59.285 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-19T01:03:59.285 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2024-04-19T01:03:59.285 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2024-04-19T01:03:59.285 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 8, 2024-04-19T01:03:59.285 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 8, 2024-04-19T01:03:59.286 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 8, 2024-04-19T01:03:59.286 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 8, 2024-04-19T01:03:59.286 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 8, 2024-04-19T01:03:59.286 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2024-04-19T01:03:59.286 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2024-04-19T01:03:59.286 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2024-04-19T01:03:59.286 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2024-04-19T01:03:59.286 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2024-04-19T01:03:59.286 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2024-04-19T01:03:59.286 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2024-04-19T01:03:59.286 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.286 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "28", 2024-04-19T01:03:59.286 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2024-04-19T01:03:59.286 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-19T01:03:59.287 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2024-04-19T01:03:59.287 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2024-04-19T01:03:59.287 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2024-04-19T01:03:59.287 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2024-04-19T01:03:59.287 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2024-04-19T01:03:59.287 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2024-04-19T01:03:59.287 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2024-04-19T01:03:59.287 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2024-04-19T01:03:59.287 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2024-04-19T01:03:59.287 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2024-04-19T01:03:59.287 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2024-04-19T01:03:59.287 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2024-04-19T01:03:59.287 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2024-04-19T01:03:59.287 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2024-04-19T01:03:59.288 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2024-04-19T01:03:59.288 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2024-04-19T01:03:59.288 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-19T01:03:59.288 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-19T01:03:59.288 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-19T01:03:59.288 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2024-04-19T01:03:59.288 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2024-04-19T01:03:59.288 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2024-04-19T01:03:59.288 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2024-04-19T01:03:59.288 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2024-04-19T01:03:59.288 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.288 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2024-04-19T01:03:59.288 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2024-04-19T01:03:59.288 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2024-04-19T01:03:59.289 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2024-04-19T01:03:59.289 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2024-04-19T01:03:59.289 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2024-04-19T01:03:59.289 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2024-04-19T01:03:59.289 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2024-04-19T01:03:59.289 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2024-04-19T01:03:59.289 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2024-04-19T01:03:59.289 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2024-04-19T01:03:59.289 INFO:teuthology.orchestra.run.smithi033.stdout: "options": {}, 2024-04-19T01:03:59.289 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2024-04-19T01:03:59.289 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd": {} 2024-04-19T01:03:59.289 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.289 INFO:teuthology.orchestra.run.smithi033.stdout: "read_balance": { 2024-04-19T01:03:59.289 INFO:teuthology.orchestra.run.smithi033.stdout: "score_type": "Fair distribution", 2024-04-19T01:03:59.289 INFO:teuthology.orchestra.run.smithi033.stdout: "score_acting": 6.0199999809265137, 2024-04-19T01:03:59.290 INFO:teuthology.orchestra.run.smithi033.stdout: "score_stable": 6.0199999809265137, 2024-04-19T01:03:59.290 INFO:teuthology.orchestra.run.smithi033.stdout: "optimal_score": 0.81000000238418579, 2024-04-19T01:03:59.290 INFO:teuthology.orchestra.run.smithi033.stdout: "raw_score_acting": 4.880000114440918, 2024-04-19T01:03:59.290 INFO:teuthology.orchestra.run.smithi033.stdout: "raw_score_stable": 4.880000114440918, 2024-04-19T01:03:59.290 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity_weighted": 1, 2024-04-19T01:03:59.290 INFO:teuthology.orchestra.run.smithi033.stdout: "average_primary_affinity": 1, 2024-04-19T01:03:59.290 INFO:teuthology.orchestra.run.smithi033.stdout: "average_primary_affinity_weighted": 1 2024-04-19T01:03:59.290 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.290 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.290 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-04-19T01:03:59.290 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2024-04-19T01:03:59.290 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.290 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 0, 2024-04-19T01:03:59.290 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "296899fd-2b2b-4071-8b1f-0f92c3d0bbc0", 2024-04-19T01:03:59.291 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-04-19T01:03:59.291 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-04-19T01:03:59.291 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-04-19T01:03:59.291 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-04-19T01:03:59.291 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-04-19T01:03:59.291 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-04-19T01:03:59.291 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 24, 2024-04-19T01:03:59.291 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 0, 2024-04-19T01:03:59.291 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-04-19T01:03:59.291 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-04-19T01:03:59.291 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-04-19T01:03:59.291 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.291 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.291 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.292 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6824", 2024-04-19T01:03:59.292 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1633616806 2024-04-19T01:03:59.292 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.292 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.292 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.292 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6825", 2024-04-19T01:03:59.292 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1633616806 2024-04-19T01:03:59.292 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.292 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.292 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.292 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-04-19T01:03:59.292 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.292 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.292 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.292 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6826", 2024-04-19T01:03:59.293 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1633616806 2024-04-19T01:03:59.293 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.293 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.293 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.293 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6827", 2024-04-19T01:03:59.293 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1633616806 2024-04-19T01:03:59.293 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.293 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.293 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.293 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-04-19T01:03:59.293 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.293 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.293 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.294 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6830", 2024-04-19T01:03:59.294 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1633616806 2024-04-19T01:03:59.294 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.294 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.294 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.294 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6831", 2024-04-19T01:03:59.294 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1633616806 2024-04-19T01:03:59.294 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.294 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.294 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.294 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-04-19T01:03:59.294 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.294 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.294 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.295 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6828", 2024-04-19T01:03:59.295 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1633616806 2024-04-19T01:03:59.295 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.295 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.295 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.295 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6829", 2024-04-19T01:03:59.295 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1633616806 2024-04-19T01:03:59.295 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.295 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.295 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.295 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.33:6825/1633616806", 2024-04-19T01:03:59.295 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.33:6827/1633616806", 2024-04-19T01:03:59.295 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.33:6831/1633616806", 2024-04-19T01:03:59.295 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.33:6829/1633616806", 2024-04-19T01:03:59.296 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-04-19T01:03:59.296 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-04-19T01:03:59.296 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-04-19T01:03:59.296 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.296 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.296 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.296 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 1, 2024-04-19T01:03:59.296 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "021f7190-0ec3-4553-a44f-15095765cccb", 2024-04-19T01:03:59.296 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-04-19T01:03:59.296 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-04-19T01:03:59.296 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-04-19T01:03:59.296 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-04-19T01:03:59.296 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-04-19T01:03:59.296 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-04-19T01:03:59.296 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 22, 2024-04-19T01:03:59.297 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 0, 2024-04-19T01:03:59.297 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-04-19T01:03:59.297 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-04-19T01:03:59.297 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-04-19T01:03:59.297 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.297 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.297 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.297 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.110:6805", 2024-04-19T01:03:59.297 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 4270148213 2024-04-19T01:03:59.297 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.297 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.297 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.297 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.110:6808", 2024-04-19T01:03:59.297 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 4270148213 2024-04-19T01:03:59.298 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.298 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.298 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.298 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-04-19T01:03:59.298 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.298 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.298 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.298 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.110:6811", 2024-04-19T01:03:59.298 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 4270148213 2024-04-19T01:03:59.298 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.298 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.298 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.298 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.110:6814", 2024-04-19T01:03:59.298 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 4270148213 2024-04-19T01:03:59.299 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.299 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.299 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.299 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-04-19T01:03:59.299 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.299 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.299 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.299 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.110:6822", 2024-04-19T01:03:59.299 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 4270148213 2024-04-19T01:03:59.299 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.299 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.299 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.299 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.110:6823", 2024-04-19T01:03:59.299 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 4270148213 2024-04-19T01:03:59.300 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.300 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.300 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.300 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-04-19T01:03:59.300 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.300 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.300 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.300 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.110:6817", 2024-04-19T01:03:59.300 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 4270148213 2024-04-19T01:03:59.300 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.300 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.300 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.300 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.110:6820", 2024-04-19T01:03:59.300 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 4270148213 2024-04-19T01:03:59.300 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.301 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.301 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.301 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.110:6808/4270148213", 2024-04-19T01:03:59.301 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.110:6814/4270148213", 2024-04-19T01:03:59.301 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.110:6823/4270148213", 2024-04-19T01:03:59.301 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.110:6820/4270148213", 2024-04-19T01:03:59.301 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-04-19T01:03:59.301 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-04-19T01:03:59.301 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-04-19T01:03:59.301 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.301 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.301 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.301 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 2, 2024-04-19T01:03:59.301 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "c283d0bc-5e34-4763-9019-328ee9b6c07a", 2024-04-19T01:03:59.302 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-04-19T01:03:59.302 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-04-19T01:03:59.302 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-04-19T01:03:59.302 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-04-19T01:03:59.302 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-04-19T01:03:59.302 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-04-19T01:03:59.302 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 24, 2024-04-19T01:03:59.302 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 0, 2024-04-19T01:03:59.302 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-04-19T01:03:59.302 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-04-19T01:03:59.302 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-04-19T01:03:59.302 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.302 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.303 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.303 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.118:6808", 2024-04-19T01:03:59.303 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 13248188 2024-04-19T01:03:59.303 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.303 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.303 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.303 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.118:6809", 2024-04-19T01:03:59.303 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 13248188 2024-04-19T01:03:59.303 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.303 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.303 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.303 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-04-19T01:03:59.303 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.303 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.304 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.304 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.118:6810", 2024-04-19T01:03:59.304 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 13248188 2024-04-19T01:03:59.304 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.304 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.304 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.304 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.118:6811", 2024-04-19T01:03:59.304 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 13248188 2024-04-19T01:03:59.304 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.304 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.304 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.304 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-04-19T01:03:59.304 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.304 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.304 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.305 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.118:6814", 2024-04-19T01:03:59.305 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 13248188 2024-04-19T01:03:59.305 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.305 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.305 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.305 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.118:6815", 2024-04-19T01:03:59.305 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 13248188 2024-04-19T01:03:59.305 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.305 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.305 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.305 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-04-19T01:03:59.305 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.305 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.305 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.306 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.118:6812", 2024-04-19T01:03:59.306 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 13248188 2024-04-19T01:03:59.306 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.306 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.306 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.306 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.118:6813", 2024-04-19T01:03:59.306 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 13248188 2024-04-19T01:03:59.306 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.306 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.306 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.306 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.118:6809/13248188", 2024-04-19T01:03:59.306 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.118:6811/13248188", 2024-04-19T01:03:59.306 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.118:6815/13248188", 2024-04-19T01:03:59.306 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.118:6813/13248188", 2024-04-19T01:03:59.307 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-04-19T01:03:59.307 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-04-19T01:03:59.307 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-04-19T01:03:59.307 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.307 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.307 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.307 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 3, 2024-04-19T01:03:59.307 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "d45888f2-eecd-480b-80f5-c0731a06ff3d", 2024-04-19T01:03:59.307 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-04-19T01:03:59.307 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-04-19T01:03:59.307 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-04-19T01:03:59.307 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-04-19T01:03:59.307 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-04-19T01:03:59.307 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-04-19T01:03:59.307 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 22, 2024-04-19T01:03:59.308 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 0, 2024-04-19T01:03:59.308 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-04-19T01:03:59.308 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-04-19T01:03:59.308 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-04-19T01:03:59.308 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.308 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.308 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.308 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.155:6802", 2024-04-19T01:03:59.308 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2658385904 2024-04-19T01:03:59.308 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.308 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.308 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.308 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.155:6803", 2024-04-19T01:03:59.308 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2658385904 2024-04-19T01:03:59.309 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.309 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.309 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.309 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-04-19T01:03:59.309 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.309 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.309 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.309 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.155:6804", 2024-04-19T01:03:59.309 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2658385904 2024-04-19T01:03:59.309 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.309 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.309 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.309 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.155:6805", 2024-04-19T01:03:59.309 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2658385904 2024-04-19T01:03:59.310 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.310 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.310 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.310 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-04-19T01:03:59.310 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.310 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.310 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.310 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.155:6813", 2024-04-19T01:03:59.310 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2658385904 2024-04-19T01:03:59.310 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.310 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.310 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.310 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.155:6816", 2024-04-19T01:03:59.310 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2658385904 2024-04-19T01:03:59.311 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.311 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.311 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.311 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-04-19T01:03:59.311 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.311 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.311 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.311 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.155:6806", 2024-04-19T01:03:59.311 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2658385904 2024-04-19T01:03:59.311 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.311 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.311 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.311 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.155:6809", 2024-04-19T01:03:59.311 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2658385904 2024-04-19T01:03:59.312 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.312 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.312 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.312 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.155:6803/2658385904", 2024-04-19T01:03:59.312 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.155:6805/2658385904", 2024-04-19T01:03:59.312 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.155:6816/2658385904", 2024-04-19T01:03:59.312 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.155:6809/2658385904", 2024-04-19T01:03:59.312 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-04-19T01:03:59.312 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-04-19T01:03:59.312 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-04-19T01:03:59.312 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.312 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.312 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.312 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 4, 2024-04-19T01:03:59.313 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "d060d939-8c94-4a07-9fd5-42af0044a16f", 2024-04-19T01:03:59.313 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-04-19T01:03:59.313 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-04-19T01:03:59.313 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-04-19T01:03:59.313 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-04-19T01:03:59.313 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-04-19T01:03:59.313 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-04-19T01:03:59.313 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 22, 2024-04-19T01:03:59.313 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 0, 2024-04-19T01:03:59.313 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-04-19T01:03:59.313 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-04-19T01:03:59.313 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-04-19T01:03:59.313 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.313 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.314 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.314 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6800", 2024-04-19T01:03:59.314 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2389566155 2024-04-19T01:03:59.314 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.314 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.314 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.314 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6801", 2024-04-19T01:03:59.314 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2389566155 2024-04-19T01:03:59.314 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.314 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.314 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.314 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-04-19T01:03:59.314 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.314 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.315 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.315 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6802", 2024-04-19T01:03:59.315 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2389566155 2024-04-19T01:03:59.315 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.315 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.315 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.315 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6803", 2024-04-19T01:03:59.315 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2389566155 2024-04-19T01:03:59.315 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.315 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.315 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.315 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-04-19T01:03:59.315 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.315 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.316 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.316 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6806", 2024-04-19T01:03:59.316 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2389566155 2024-04-19T01:03:59.316 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.316 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.316 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.316 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6807", 2024-04-19T01:03:59.316 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2389566155 2024-04-19T01:03:59.316 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.316 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.316 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.316 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-04-19T01:03:59.316 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.316 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.316 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.317 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6804", 2024-04-19T01:03:59.317 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2389566155 2024-04-19T01:03:59.317 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.317 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.317 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.317 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6805", 2024-04-19T01:03:59.317 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2389566155 2024-04-19T01:03:59.317 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.317 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.317 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.318 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.33:6801/2389566155", 2024-04-19T01:03:59.318 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.33:6803/2389566155", 2024-04-19T01:03:59.318 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.33:6807/2389566155", 2024-04-19T01:03:59.318 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.33:6805/2389566155", 2024-04-19T01:03:59.318 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-04-19T01:03:59.318 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-04-19T01:03:59.318 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-04-19T01:03:59.318 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.318 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.318 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.318 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 5, 2024-04-19T01:03:59.318 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "9700d2f7-ed56-4143-ba9d-6cebe524e877", 2024-04-19T01:03:59.318 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-04-19T01:03:59.319 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-04-19T01:03:59.319 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-04-19T01:03:59.319 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-04-19T01:03:59.319 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-04-19T01:03:59.319 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-04-19T01:03:59.319 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 22, 2024-04-19T01:03:59.319 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 0, 2024-04-19T01:03:59.319 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-04-19T01:03:59.319 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-04-19T01:03:59.319 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-04-19T01:03:59.319 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.319 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.319 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.319 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.110:6824", 2024-04-19T01:03:59.320 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2821892844 2024-04-19T01:03:59.320 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.320 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.320 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.320 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.110:6825", 2024-04-19T01:03:59.320 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2821892844 2024-04-19T01:03:59.320 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.320 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.320 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.320 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-04-19T01:03:59.320 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.320 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.320 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.321 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.110:6826", 2024-04-19T01:03:59.321 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2821892844 2024-04-19T01:03:59.321 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.321 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.321 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.321 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.110:6827", 2024-04-19T01:03:59.321 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2821892844 2024-04-19T01:03:59.321 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.321 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.321 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.321 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-04-19T01:03:59.321 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.321 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.321 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.322 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.110:6830", 2024-04-19T01:03:59.322 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2821892844 2024-04-19T01:03:59.322 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.322 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.322 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.322 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.110:6831", 2024-04-19T01:03:59.322 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2821892844 2024-04-19T01:03:59.322 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.322 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.322 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.322 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-04-19T01:03:59.322 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.322 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.322 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.323 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.110:6828", 2024-04-19T01:03:59.323 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2821892844 2024-04-19T01:03:59.323 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.323 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.323 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.323 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.110:6829", 2024-04-19T01:03:59.323 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2821892844 2024-04-19T01:03:59.323 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.323 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.323 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.323 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.110:6825/2821892844", 2024-04-19T01:03:59.323 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.110:6827/2821892844", 2024-04-19T01:03:59.323 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.110:6831/2821892844", 2024-04-19T01:03:59.323 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.110:6829/2821892844", 2024-04-19T01:03:59.323 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-04-19T01:03:59.324 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-04-19T01:03:59.324 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-04-19T01:03:59.324 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.324 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.324 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.324 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 6, 2024-04-19T01:03:59.324 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "49324695-d83b-41ad-92e1-5348e8e9149f", 2024-04-19T01:03:59.324 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-04-19T01:03:59.324 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-04-19T01:03:59.324 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-04-19T01:03:59.324 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-04-19T01:03:59.324 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-04-19T01:03:59.324 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-04-19T01:03:59.324 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 22, 2024-04-19T01:03:59.325 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 26, 2024-04-19T01:03:59.325 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-04-19T01:03:59.325 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-04-19T01:03:59.325 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-04-19T01:03:59.325 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.325 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.325 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.325 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.118:6816", 2024-04-19T01:03:59.325 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 273776428 2024-04-19T01:03:59.325 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.325 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.325 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.325 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.118:6818", 2024-04-19T01:03:59.325 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 273776428 2024-04-19T01:03:59.325 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.326 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.326 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.326 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-04-19T01:03:59.326 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.326 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.326 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.326 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.118:6820", 2024-04-19T01:03:59.326 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 273776428 2024-04-19T01:03:59.326 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.326 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.326 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.326 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.118:6822", 2024-04-19T01:03:59.326 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 273776428 2024-04-19T01:03:59.326 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.327 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.327 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.327 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-04-19T01:03:59.327 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.327 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.327 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.327 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.118:6828", 2024-04-19T01:03:59.327 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 273776428 2024-04-19T01:03:59.327 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.327 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.327 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.327 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.118:6830", 2024-04-19T01:03:59.327 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 273776428 2024-04-19T01:03:59.327 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.328 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.328 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.328 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-04-19T01:03:59.328 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.328 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.328 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.328 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.118:6824", 2024-04-19T01:03:59.328 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 273776428 2024-04-19T01:03:59.328 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.328 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.328 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.328 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.118:6826", 2024-04-19T01:03:59.328 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 273776428 2024-04-19T01:03:59.328 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.329 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.329 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.329 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.118:6818/273776428", 2024-04-19T01:03:59.329 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.118:6822/273776428", 2024-04-19T01:03:59.329 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.118:6830/273776428", 2024-04-19T01:03:59.329 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.118:6826/273776428", 2024-04-19T01:03:59.329 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-04-19T01:03:59.329 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-04-19T01:03:59.329 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-04-19T01:03:59.329 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.329 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.329 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.329 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 7, 2024-04-19T01:03:59.329 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "afef160d-0911-4406-ad49-cd6d61ee8a2f", 2024-04-19T01:03:59.330 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-04-19T01:03:59.330 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-04-19T01:03:59.330 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-04-19T01:03:59.330 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-04-19T01:03:59.330 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-04-19T01:03:59.330 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-04-19T01:03:59.330 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 24, 2024-04-19T01:03:59.330 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 0, 2024-04-19T01:03:59.330 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-04-19T01:03:59.330 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-04-19T01:03:59.330 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-04-19T01:03:59.330 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.330 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.330 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.331 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.155:6826", 2024-04-19T01:03:59.331 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1006872173 2024-04-19T01:03:59.331 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.331 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.331 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.331 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.155:6827", 2024-04-19T01:03:59.331 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1006872173 2024-04-19T01:03:59.331 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.331 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.331 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.331 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-04-19T01:03:59.331 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.331 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.331 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.332 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.155:6828", 2024-04-19T01:03:59.332 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1006872173 2024-04-19T01:03:59.332 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.332 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.332 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.332 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.155:6829", 2024-04-19T01:03:59.332 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1006872173 2024-04-19T01:03:59.332 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.332 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.332 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.332 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-04-19T01:03:59.332 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.332 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.332 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.333 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.155:6832", 2024-04-19T01:03:59.333 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1006872173 2024-04-19T01:03:59.333 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.333 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.333 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.333 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.155:6833", 2024-04-19T01:03:59.333 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1006872173 2024-04-19T01:03:59.333 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.333 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.333 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.333 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-04-19T01:03:59.333 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.333 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.333 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.333 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.155:6830", 2024-04-19T01:03:59.334 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1006872173 2024-04-19T01:03:59.334 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.334 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.334 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.334 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.155:6831", 2024-04-19T01:03:59.334 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1006872173 2024-04-19T01:03:59.334 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.334 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.334 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.334 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.155:6827/1006872173", 2024-04-19T01:03:59.334 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.155:6829/1006872173", 2024-04-19T01:03:59.334 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.155:6833/1006872173", 2024-04-19T01:03:59.334 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.155:6831/1006872173", 2024-04-19T01:03:59.334 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-04-19T01:03:59.335 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-04-19T01:03:59.335 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-04-19T01:03:59.335 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.335 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.335 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.335 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 8, 2024-04-19T01:03:59.335 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "298396bf-31a2-4832-91a4-55fcb050130f", 2024-04-19T01:03:59.335 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-04-19T01:03:59.335 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-04-19T01:03:59.335 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-04-19T01:03:59.335 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-04-19T01:03:59.335 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-04-19T01:03:59.335 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-04-19T01:03:59.335 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 22, 2024-04-19T01:03:59.335 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 26, 2024-04-19T01:03:59.336 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-04-19T01:03:59.336 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-04-19T01:03:59.336 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-04-19T01:03:59.336 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.336 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.336 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.336 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6816", 2024-04-19T01:03:59.336 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 744651323 2024-04-19T01:03:59.336 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.336 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.336 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.336 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6817", 2024-04-19T01:03:59.336 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 744651323 2024-04-19T01:03:59.336 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.337 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.337 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.337 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-04-19T01:03:59.337 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.337 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.337 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.337 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6818", 2024-04-19T01:03:59.337 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 744651323 2024-04-19T01:03:59.337 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.337 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.337 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.337 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6819", 2024-04-19T01:03:59.337 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 744651323 2024-04-19T01:03:59.337 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.338 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.338 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.338 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-04-19T01:03:59.338 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.338 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.338 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.338 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6822", 2024-04-19T01:03:59.338 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 744651323 2024-04-19T01:03:59.338 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.338 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.338 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.339 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6823", 2024-04-19T01:03:59.339 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 744651323 2024-04-19T01:03:59.339 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.339 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.339 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.339 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-04-19T01:03:59.339 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.339 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.339 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.339 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6820", 2024-04-19T01:03:59.339 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 744651323 2024-04-19T01:03:59.339 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.339 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.339 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.340 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6821", 2024-04-19T01:03:59.340 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 744651323 2024-04-19T01:03:59.340 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.340 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.340 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.340 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.33:6817/744651323", 2024-04-19T01:03:59.340 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.33:6819/744651323", 2024-04-19T01:03:59.340 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.33:6823/744651323", 2024-04-19T01:03:59.340 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.33:6821/744651323", 2024-04-19T01:03:59.340 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-04-19T01:03:59.340 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-04-19T01:03:59.340 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-04-19T01:03:59.340 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.340 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.341 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.341 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 9, 2024-04-19T01:03:59.341 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "c151ae6b-279d-4174-aceb-4671fbdac3b3", 2024-04-19T01:03:59.341 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-04-19T01:03:59.341 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-04-19T01:03:59.341 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-04-19T01:03:59.341 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-04-19T01:03:59.341 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-04-19T01:03:59.341 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-04-19T01:03:59.341 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 24, 2024-04-19T01:03:59.341 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 26, 2024-04-19T01:03:59.341 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-04-19T01:03:59.341 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-04-19T01:03:59.341 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-04-19T01:03:59.341 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.342 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.342 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.342 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.110:6800", 2024-04-19T01:03:59.342 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 872519654 2024-04-19T01:03:59.342 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.342 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.342 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.342 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.110:6802", 2024-04-19T01:03:59.342 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 872519654 2024-04-19T01:03:59.342 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.342 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.342 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.342 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-04-19T01:03:59.342 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.343 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.343 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.343 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.110:6804", 2024-04-19T01:03:59.343 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 872519654 2024-04-19T01:03:59.343 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.343 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.343 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.343 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.110:6807", 2024-04-19T01:03:59.343 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 872519654 2024-04-19T01:03:59.343 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.343 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.343 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.343 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-04-19T01:03:59.343 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.343 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.344 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.344 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.110:6816", 2024-04-19T01:03:59.344 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 872519654 2024-04-19T01:03:59.344 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.344 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.344 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.344 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.110:6819", 2024-04-19T01:03:59.344 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 872519654 2024-04-19T01:03:59.344 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.344 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.344 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.344 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-04-19T01:03:59.344 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.344 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.345 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.345 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.110:6810", 2024-04-19T01:03:59.345 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 872519654 2024-04-19T01:03:59.345 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.345 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.345 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.345 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.110:6813", 2024-04-19T01:03:59.345 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 872519654 2024-04-19T01:03:59.345 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.345 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.345 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.345 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.110:6802/872519654", 2024-04-19T01:03:59.345 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.110:6807/872519654", 2024-04-19T01:03:59.345 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.110:6819/872519654", 2024-04-19T01:03:59.345 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.110:6813/872519654", 2024-04-19T01:03:59.346 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-04-19T01:03:59.346 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-04-19T01:03:59.346 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-04-19T01:03:59.346 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.346 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.346 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.346 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 10, 2024-04-19T01:03:59.346 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "0b292dfb-2029-4e1e-b206-ea7e0f6969ab", 2024-04-19T01:03:59.346 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-04-19T01:03:59.346 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-04-19T01:03:59.346 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-04-19T01:03:59.346 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-04-19T01:03:59.346 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-04-19T01:03:59.347 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-04-19T01:03:59.347 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 21, 2024-04-19T01:03:59.347 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 0, 2024-04-19T01:03:59.347 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-04-19T01:03:59.347 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-04-19T01:03:59.347 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-04-19T01:03:59.347 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.347 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.347 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.347 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.118:6817", 2024-04-19T01:03:59.347 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 4099141917 2024-04-19T01:03:59.347 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.347 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.347 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.347 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.118:6819", 2024-04-19T01:03:59.348 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 4099141917 2024-04-19T01:03:59.348 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.348 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.348 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.348 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-04-19T01:03:59.348 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.348 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.348 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.348 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.118:6821", 2024-04-19T01:03:59.348 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 4099141917 2024-04-19T01:03:59.348 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.348 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.348 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.348 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.118:6823", 2024-04-19T01:03:59.349 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 4099141917 2024-04-19T01:03:59.349 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.349 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.349 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.349 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-04-19T01:03:59.349 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.349 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.349 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.349 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.118:6829", 2024-04-19T01:03:59.349 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 4099141917 2024-04-19T01:03:59.349 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.349 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.349 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.349 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.118:6831", 2024-04-19T01:03:59.349 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 4099141917 2024-04-19T01:03:59.350 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.350 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.350 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.350 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-04-19T01:03:59.350 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.350 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.350 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.350 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.118:6825", 2024-04-19T01:03:59.350 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 4099141917 2024-04-19T01:03:59.350 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.350 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.350 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.350 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.118:6827", 2024-04-19T01:03:59.350 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 4099141917 2024-04-19T01:03:59.351 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.351 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.351 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.351 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.118:6819/4099141917", 2024-04-19T01:03:59.351 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.118:6823/4099141917", 2024-04-19T01:03:59.351 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.118:6831/4099141917", 2024-04-19T01:03:59.351 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.118:6827/4099141917", 2024-04-19T01:03:59.351 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-04-19T01:03:59.351 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-04-19T01:03:59.351 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-04-19T01:03:59.351 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.351 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.351 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.351 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 11, 2024-04-19T01:03:59.352 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "10c61619-ad11-40f0-b013-9ef52bcabf59", 2024-04-19T01:03:59.352 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-04-19T01:03:59.352 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-04-19T01:03:59.352 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-04-19T01:03:59.352 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-04-19T01:03:59.352 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-04-19T01:03:59.352 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-04-19T01:03:59.352 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 22, 2024-04-19T01:03:59.352 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 26, 2024-04-19T01:03:59.352 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-04-19T01:03:59.352 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-04-19T01:03:59.352 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-04-19T01:03:59.352 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.352 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.353 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.353 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.155:6808", 2024-04-19T01:03:59.353 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3981951559 2024-04-19T01:03:59.353 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.353 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.353 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.353 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.155:6811", 2024-04-19T01:03:59.353 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3981951559 2024-04-19T01:03:59.353 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.353 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.353 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.353 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-04-19T01:03:59.353 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.353 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.353 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.354 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.155:6814", 2024-04-19T01:03:59.354 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3981951559 2024-04-19T01:03:59.354 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.354 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.354 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.354 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.155:6817", 2024-04-19T01:03:59.354 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3981951559 2024-04-19T01:03:59.354 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.354 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.354 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.354 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-04-19T01:03:59.354 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.354 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.354 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.355 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.155:6824", 2024-04-19T01:03:59.355 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3981951559 2024-04-19T01:03:59.355 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.355 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.355 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.355 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.155:6825", 2024-04-19T01:03:59.355 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3981951559 2024-04-19T01:03:59.355 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.355 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.355 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.355 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-04-19T01:03:59.355 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.355 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.355 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.356 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.155:6820", 2024-04-19T01:03:59.356 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3981951559 2024-04-19T01:03:59.356 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.356 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.356 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.356 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.155:6823", 2024-04-19T01:03:59.356 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3981951559 2024-04-19T01:03:59.356 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.356 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.356 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.356 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.155:6811/3981951559", 2024-04-19T01:03:59.356 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.155:6817/3981951559", 2024-04-19T01:03:59.357 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.155:6825/3981951559", 2024-04-19T01:03:59.357 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.155:6823/3981951559", 2024-04-19T01:03:59.357 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-04-19T01:03:59.357 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-04-19T01:03:59.357 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-04-19T01:03:59.357 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.357 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.357 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.357 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 12, 2024-04-19T01:03:59.357 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "3f0d26e5-f3e2-47b6-91a8-0daa30d0544f", 2024-04-19T01:03:59.357 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-04-19T01:03:59.357 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-04-19T01:03:59.357 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-04-19T01:03:59.357 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-04-19T01:03:59.358 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-04-19T01:03:59.358 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-04-19T01:03:59.358 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 22, 2024-04-19T01:03:59.358 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 0, 2024-04-19T01:03:59.358 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-04-19T01:03:59.358 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-04-19T01:03:59.358 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-04-19T01:03:59.358 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.358 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.358 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.358 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6808", 2024-04-19T01:03:59.358 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 4133738119 2024-04-19T01:03:59.358 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.358 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.359 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.359 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6809", 2024-04-19T01:03:59.359 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 4133738119 2024-04-19T01:03:59.359 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.359 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.359 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.359 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-04-19T01:03:59.359 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.359 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.359 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.359 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6810", 2024-04-19T01:03:59.359 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 4133738119 2024-04-19T01:03:59.359 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.359 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.359 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.360 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6811", 2024-04-19T01:03:59.360 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 4133738119 2024-04-19T01:03:59.360 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.360 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.360 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.360 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-04-19T01:03:59.360 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.360 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.360 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.360 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6814", 2024-04-19T01:03:59.360 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 4133738119 2024-04-19T01:03:59.360 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.360 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.361 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.361 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6815", 2024-04-19T01:03:59.361 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 4133738119 2024-04-19T01:03:59.361 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.361 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.361 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.361 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-04-19T01:03:59.361 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.361 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.361 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.361 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6812", 2024-04-19T01:03:59.361 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 4133738119 2024-04-19T01:03:59.361 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.361 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.361 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.362 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6813", 2024-04-19T01:03:59.362 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 4133738119 2024-04-19T01:03:59.362 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.362 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.362 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.362 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.33:6809/4133738119", 2024-04-19T01:03:59.362 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.33:6811/4133738119", 2024-04-19T01:03:59.362 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.33:6815/4133738119", 2024-04-19T01:03:59.362 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.33:6813/4133738119", 2024-04-19T01:03:59.362 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-04-19T01:03:59.362 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-04-19T01:03:59.363 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-04-19T01:03:59.363 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.363 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.363 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.363 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 13, 2024-04-19T01:03:59.363 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "c8d6213f-af5e-4646-9241-6f91589b0e1b", 2024-04-19T01:03:59.363 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-04-19T01:03:59.363 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-04-19T01:03:59.363 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-04-19T01:03:59.363 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-04-19T01:03:59.363 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-04-19T01:03:59.363 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-04-19T01:03:59.363 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 24, 2024-04-19T01:03:59.363 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 0, 2024-04-19T01:03:59.363 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-04-19T01:03:59.364 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-04-19T01:03:59.364 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-04-19T01:03:59.364 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.364 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.364 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.364 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.110:6801", 2024-04-19T01:03:59.364 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1583232449 2024-04-19T01:03:59.364 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.364 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.364 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.364 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.110:6803", 2024-04-19T01:03:59.364 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1583232449 2024-04-19T01:03:59.364 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.364 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.365 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.365 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-04-19T01:03:59.365 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.365 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.365 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.365 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.110:6806", 2024-04-19T01:03:59.365 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1583232449 2024-04-19T01:03:59.365 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.365 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.365 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.365 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.110:6809", 2024-04-19T01:03:59.365 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1583232449 2024-04-19T01:03:59.365 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.365 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.366 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.366 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-04-19T01:03:59.366 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.366 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.366 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.366 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.110:6818", 2024-04-19T01:03:59.366 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1583232449 2024-04-19T01:03:59.366 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.366 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.366 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.366 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.110:6821", 2024-04-19T01:03:59.366 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1583232449 2024-04-19T01:03:59.366 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.366 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.367 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.367 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-04-19T01:03:59.367 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.367 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.367 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.367 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.110:6812", 2024-04-19T01:03:59.367 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1583232449 2024-04-19T01:03:59.367 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.367 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.367 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.367 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.110:6815", 2024-04-19T01:03:59.367 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1583232449 2024-04-19T01:03:59.367 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.367 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.368 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.368 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.110:6803/1583232449", 2024-04-19T01:03:59.368 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.110:6809/1583232449", 2024-04-19T01:03:59.368 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.110:6821/1583232449", 2024-04-19T01:03:59.368 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.110:6815/1583232449", 2024-04-19T01:03:59.368 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-04-19T01:03:59.368 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-04-19T01:03:59.368 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-04-19T01:03:59.368 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.368 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.368 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.368 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 14, 2024-04-19T01:03:59.368 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "7b3d56df-62bf-4877-b984-6f17fe1fab33", 2024-04-19T01:03:59.368 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-04-19T01:03:59.369 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-04-19T01:03:59.369 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-04-19T01:03:59.369 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-04-19T01:03:59.369 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-04-19T01:03:59.369 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-04-19T01:03:59.369 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 24, 2024-04-19T01:03:59.369 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 26, 2024-04-19T01:03:59.369 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-04-19T01:03:59.369 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-04-19T01:03:59.369 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-04-19T01:03:59.369 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.369 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.369 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.369 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.118:6800", 2024-04-19T01:03:59.370 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 506416463 2024-04-19T01:03:59.370 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.370 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.370 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.370 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.118:6801", 2024-04-19T01:03:59.370 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 506416463 2024-04-19T01:03:59.370 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.370 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.370 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.370 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-04-19T01:03:59.370 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.370 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.370 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.370 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.118:6802", 2024-04-19T01:03:59.371 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 506416463 2024-04-19T01:03:59.371 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.371 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.371 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.371 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.118:6803", 2024-04-19T01:03:59.371 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 506416463 2024-04-19T01:03:59.371 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.371 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.371 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.371 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-04-19T01:03:59.371 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.371 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.371 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.371 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.118:6806", 2024-04-19T01:03:59.372 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 506416463 2024-04-19T01:03:59.372 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.372 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.372 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.372 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.118:6807", 2024-04-19T01:03:59.372 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 506416463 2024-04-19T01:03:59.372 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.372 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.372 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.372 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-04-19T01:03:59.372 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.372 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.372 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.372 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.118:6804", 2024-04-19T01:03:59.372 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 506416463 2024-04-19T01:03:59.373 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.373 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.373 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.373 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.118:6805", 2024-04-19T01:03:59.373 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 506416463 2024-04-19T01:03:59.373 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.373 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.373 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.373 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.118:6801/506416463", 2024-04-19T01:03:59.373 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.118:6803/506416463", 2024-04-19T01:03:59.373 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.118:6807/506416463", 2024-04-19T01:03:59.373 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.118:6805/506416463", 2024-04-19T01:03:59.373 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-04-19T01:03:59.373 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-04-19T01:03:59.374 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-04-19T01:03:59.374 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.374 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.374 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.374 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 15, 2024-04-19T01:03:59.374 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "6d526edb-f1d8-415a-ad55-81e810ff4944", 2024-04-19T01:03:59.374 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2024-04-19T01:03:59.374 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2024-04-19T01:03:59.374 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2024-04-19T01:03:59.374 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2024-04-19T01:03:59.374 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2024-04-19T01:03:59.374 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2024-04-19T01:03:59.374 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 22, 2024-04-19T01:03:59.374 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 26, 2024-04-19T01:03:59.375 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2024-04-19T01:03:59.375 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2024-04-19T01:03:59.375 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2024-04-19T01:03:59.375 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.375 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.375 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.375 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.155:6807", 2024-04-19T01:03:59.375 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 741132602 2024-04-19T01:03:59.375 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.375 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.375 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.375 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.155:6810", 2024-04-19T01:03:59.375 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 741132602 2024-04-19T01:03:59.376 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.376 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.376 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.376 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2024-04-19T01:03:59.376 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.376 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.376 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.376 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.155:6812", 2024-04-19T01:03:59.376 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 741132602 2024-04-19T01:03:59.376 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.376 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.376 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.377 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.155:6815", 2024-04-19T01:03:59.377 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 741132602 2024-04-19T01:03:59.377 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.377 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.377 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.377 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2024-04-19T01:03:59.377 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.377 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.377 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.377 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.155:6821", 2024-04-19T01:03:59.377 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 741132602 2024-04-19T01:03:59.377 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.377 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.377 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.378 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.155:6822", 2024-04-19T01:03:59.378 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 741132602 2024-04-19T01:03:59.378 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.378 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.378 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.378 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2024-04-19T01:03:59.378 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-04-19T01:03:59.378 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.378 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-04-19T01:03:59.378 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.155:6818", 2024-04-19T01:03:59.378 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 741132602 2024-04-19T01:03:59.378 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.378 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.378 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-04-19T01:03:59.378 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.155:6819", 2024-04-19T01:03:59.379 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 741132602 2024-04-19T01:03:59.379 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.379 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.379 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.379 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.155:6810/741132602", 2024-04-19T01:03:59.379 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.155:6815/741132602", 2024-04-19T01:03:59.379 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.155:6822/741132602", 2024-04-19T01:03:59.379 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.155:6819/741132602", 2024-04-19T01:03:59.379 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2024-04-19T01:03:59.379 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2024-04-19T01:03:59.379 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2024-04-19T01:03:59.379 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:03:59.379 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.379 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-04-19T01:03:59.380 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_xinfo": [ 2024-04-19T01:03:59.380 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.380 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 0, 2024-04-19T01:03:59.380 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-04-19T01:03:59.380 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-04-19T01:03:59.380 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-04-19T01:03:59.380 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540701547738038271, 2024-04-19T01:03:59.380 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-04-19T01:03:59.380 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-04-19T01:03:46.813806+0000", 2024-04-19T01:03:59.380 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-04-19T01:03:59.380 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.380 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.380 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 1, 2024-04-19T01:03:59.380 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-04-19T01:03:59.380 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-04-19T01:03:59.381 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-04-19T01:03:59.381 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540701547738038271, 2024-04-19T01:03:59.381 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-04-19T01:03:59.381 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-04-19T01:03:46.677618+0000", 2024-04-19T01:03:59.381 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-04-19T01:03:59.381 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.381 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.381 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 2, 2024-04-19T01:03:59.381 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-04-19T01:03:59.381 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-04-19T01:03:59.381 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-04-19T01:03:59.381 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540701547738038271, 2024-04-19T01:03:59.381 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-04-19T01:03:59.381 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-04-19T01:03:46.676628+0000", 2024-04-19T01:03:59.381 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-04-19T01:03:59.382 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.382 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.382 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 3, 2024-04-19T01:03:59.382 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-04-19T01:03:59.382 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-04-19T01:03:59.382 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-04-19T01:03:59.382 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540701547738038271, 2024-04-19T01:03:59.382 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-04-19T01:03:59.382 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-04-19T01:03:46.659122+0000", 2024-04-19T01:03:59.382 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-04-19T01:03:59.382 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.382 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.382 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 4, 2024-04-19T01:03:59.383 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-04-19T01:03:59.383 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-04-19T01:03:59.383 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-04-19T01:03:59.383 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540701547738038271, 2024-04-19T01:03:59.383 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-04-19T01:03:59.383 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-04-19T01:03:46.648122+0000", 2024-04-19T01:03:59.383 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-04-19T01:03:59.383 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.383 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.383 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 5, 2024-04-19T01:03:59.383 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-04-19T01:03:59.383 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-04-19T01:03:59.383 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-04-19T01:03:59.383 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540701547738038271, 2024-04-19T01:03:59.383 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-04-19T01:03:59.384 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-04-19T01:03:46.859759+0000", 2024-04-19T01:03:59.384 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-04-19T01:03:59.384 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.384 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.384 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 6, 2024-04-19T01:03:59.384 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-04-19T01:03:59.384 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-04-19T01:03:59.384 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-04-19T01:03:59.384 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540701547738038271, 2024-04-19T01:03:59.384 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-04-19T01:03:59.384 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-04-19T01:03:46.759643+0000", 2024-04-19T01:03:59.384 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-04-19T01:03:59.384 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.384 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.384 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 7, 2024-04-19T01:03:59.385 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-04-19T01:03:59.385 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-04-19T01:03:59.385 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-04-19T01:03:59.385 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540701547738038271, 2024-04-19T01:03:59.385 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-04-19T01:03:59.385 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-04-19T01:03:46.897847+0000", 2024-04-19T01:03:59.385 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-04-19T01:03:59.385 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.385 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.385 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 8, 2024-04-19T01:03:59.385 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-04-19T01:03:59.385 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-04-19T01:03:59.385 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-04-19T01:03:59.385 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540701547738038271, 2024-04-19T01:03:59.386 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-04-19T01:03:59.386 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-04-19T01:03:46.779763+0000", 2024-04-19T01:03:59.386 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-04-19T01:03:59.386 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.386 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.386 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 9, 2024-04-19T01:03:59.386 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-04-19T01:03:59.386 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-04-19T01:03:59.386 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-04-19T01:03:59.386 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540701547738038271, 2024-04-19T01:03:59.386 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-04-19T01:03:59.386 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-04-19T01:03:46.813460+0000", 2024-04-19T01:03:59.386 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-04-19T01:03:59.386 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.387 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.387 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 10, 2024-04-19T01:03:59.387 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-04-19T01:03:59.387 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-04-19T01:03:59.387 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-04-19T01:03:59.387 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540701547738038271, 2024-04-19T01:03:59.387 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-04-19T01:03:59.387 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-04-19T01:03:46.479489+0000", 2024-04-19T01:03:59.387 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-04-19T01:03:59.387 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.387 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.387 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 11, 2024-04-19T01:03:59.387 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-04-19T01:03:59.387 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-04-19T01:03:59.387 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-04-19T01:03:59.388 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540701547738038271, 2024-04-19T01:03:59.388 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-04-19T01:03:59.388 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-04-19T01:03:46.853984+0000", 2024-04-19T01:03:59.388 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-04-19T01:03:59.388 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.388 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.388 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 12, 2024-04-19T01:03:59.388 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-04-19T01:03:59.388 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-04-19T01:03:59.388 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-04-19T01:03:59.388 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540701547738038271, 2024-04-19T01:03:59.388 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-04-19T01:03:59.388 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-04-19T01:03:46.736100+0000", 2024-04-19T01:03:59.388 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-04-19T01:03:59.389 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.389 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.389 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 13, 2024-04-19T01:03:59.389 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-04-19T01:03:59.389 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-04-19T01:03:59.389 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-04-19T01:03:59.389 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540701547738038271, 2024-04-19T01:03:59.389 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-04-19T01:03:59.389 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-04-19T01:03:46.838084+0000", 2024-04-19T01:03:59.389 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-04-19T01:03:59.389 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.389 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.389 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 14, 2024-04-19T01:03:59.389 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-04-19T01:03:59.389 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-04-19T01:03:59.390 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-04-19T01:03:59.390 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540701547738038271, 2024-04-19T01:03:59.390 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-04-19T01:03:59.390 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-04-19T01:03:46.629904+0000", 2024-04-19T01:03:59.390 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-04-19T01:03:59.390 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.390 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:03:59.390 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 15, 2024-04-19T01:03:59.390 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2024-04-19T01:03:59.390 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2024-04-19T01:03:59.390 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2024-04-19T01:03:59.390 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540701547738038271, 2024-04-19T01:03:59.390 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2024-04-19T01:03:59.390 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2024-04-19T01:03:46.712166+0000", 2024-04-19T01:03:59.391 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2024-04-19T01:03:59.391 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.391 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-04-19T01:03:59.391 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_upmap": [], 2024-04-19T01:03:59.391 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_upmap_items": [], 2024-04-19T01:03:59.391 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_upmap_primaries": [], 2024-04-19T01:03:59.391 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_temp": [], 2024-04-19T01:03:59.391 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_temp": [], 2024-04-19T01:03:59.391 INFO:teuthology.orchestra.run.smithi033.stdout: "blocklist": {}, 2024-04-19T01:03:59.391 INFO:teuthology.orchestra.run.smithi033.stdout: "range_blocklist": {}, 2024-04-19T01:03:59.391 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profiles": { 2024-04-19T01:03:59.391 INFO:teuthology.orchestra.run.smithi033.stdout: "default": { 2024-04-19T01:03:59.391 INFO:teuthology.orchestra.run.smithi033.stdout: "crush-failure-domain": "osd", 2024-04-19T01:03:59.391 INFO:teuthology.orchestra.run.smithi033.stdout: "k": "2", 2024-04-19T01:03:59.391 INFO:teuthology.orchestra.run.smithi033.stdout: "m": "1", 2024-04-19T01:03:59.392 INFO:teuthology.orchestra.run.smithi033.stdout: "plugin": "jerasure", 2024-04-19T01:03:59.392 INFO:teuthology.orchestra.run.smithi033.stdout: "technique": "reed_sol_van" 2024-04-19T01:03:59.392 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.392 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:03:59.392 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps_queue": [], 2024-04-19T01:03:59.392 INFO:teuthology.orchestra.run.smithi033.stdout: "new_removed_snaps": [], 2024-04-19T01:03:59.392 INFO:teuthology.orchestra.run.smithi033.stdout: "new_purged_snaps": [], 2024-04-19T01:03:59.392 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_node_flags": {}, 2024-04-19T01:03:59.392 INFO:teuthology.orchestra.run.smithi033.stdout: "device_class_flags": {}, 2024-04-19T01:03:59.392 INFO:teuthology.orchestra.run.smithi033.stdout: "stretch_mode": { 2024-04-19T01:03:59.392 INFO:teuthology.orchestra.run.smithi033.stdout: "stretch_mode_enabled": false, 2024-04-19T01:03:59.392 INFO:teuthology.orchestra.run.smithi033.stdout: "stretch_bucket_count": 0, 2024-04-19T01:03:59.392 INFO:teuthology.orchestra.run.smithi033.stdout: "degraded_stretch_mode": 0, 2024-04-19T01:03:59.392 INFO:teuthology.orchestra.run.smithi033.stdout: "recovering_stretch_mode": 0, 2024-04-19T01:03:59.392 INFO:teuthology.orchestra.run.smithi033.stdout: "stretch_mode_bucket": 0 2024-04-19T01:03:59.393 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:03:59.393 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:03:59.394 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-04-19T01:03:59.394 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T01:03:59.396 INFO:tasks.thrashosds.thrasher:j is {'epoch': 28, 'fsid': 'f11bd2d5-4456-4ffa-83c0-722dddd22ad3', 'created': '2024-04-19T01:03:36.766089+0000', 'modified': '2024-04-19T01:03:54.645990+0000', 'last_up_change': '2024-04-19T01:03:50.591452+0000', 'last_in_change': '2024-04-19T01:03:41.227644+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 16, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-19T01:03:49.571609+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 15.380000114440918, 'score_stable': 15.380000114440918, 'optimal_score': 0.12999999523162842, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-04-19T01:03:51.913782+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.019999980926514, 'score_stable': 6.019999980926514, 'optimal_score': 0.8100000023841858, 'raw_score_acting': 4.880000114440918, 'raw_score_stable': 4.880000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '296899fd-2b2b-4071-8b1f-0f92c3d0bbc0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 24, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6824', 'nonce': 1633616806}, {'type': 'v1', 'addr': '172.21.15.33:6825', 'nonce': 1633616806}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6826', 'nonce': 1633616806}, {'type': 'v1', 'addr': '172.21.15.33:6827', 'nonce': 1633616806}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6830', 'nonce': 1633616806}, {'type': 'v1', 'addr': '172.21.15.33:6831', 'nonce': 1633616806}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6828', 'nonce': 1633616806}, {'type': 'v1', 'addr': '172.21.15.33:6829', 'nonce': 1633616806}]}, 'public_addr': '172.21.15.33:6825/1633616806', 'cluster_addr': '172.21.15.33:6827/1633616806', 'heartbeat_back_addr': '172.21.15.33:6831/1633616806', 'heartbeat_front_addr': '172.21.15.33:6829/1633616806', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '021f7190-0ec3-4553-a44f-15095765cccb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6805', 'nonce': 4270148213}, {'type': 'v1', 'addr': '172.21.15.110:6808', 'nonce': 4270148213}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6811', 'nonce': 4270148213}, {'type': 'v1', 'addr': '172.21.15.110:6814', 'nonce': 4270148213}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6822', 'nonce': 4270148213}, {'type': 'v1', 'addr': '172.21.15.110:6823', 'nonce': 4270148213}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6817', 'nonce': 4270148213}, {'type': 'v1', 'addr': '172.21.15.110:6820', 'nonce': 4270148213}]}, 'public_addr': '172.21.15.110:6808/4270148213', 'cluster_addr': '172.21.15.110:6814/4270148213', 'heartbeat_back_addr': '172.21.15.110:6823/4270148213', 'heartbeat_front_addr': '172.21.15.110:6820/4270148213', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'c283d0bc-5e34-4763-9019-328ee9b6c07a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 24, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6808', 'nonce': 13248188}, {'type': 'v1', 'addr': '172.21.15.118:6809', 'nonce': 13248188}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6810', 'nonce': 13248188}, {'type': 'v1', 'addr': '172.21.15.118:6811', 'nonce': 13248188}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6814', 'nonce': 13248188}, {'type': 'v1', 'addr': '172.21.15.118:6815', 'nonce': 13248188}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6812', 'nonce': 13248188}, {'type': 'v1', 'addr': '172.21.15.118:6813', 'nonce': 13248188}]}, 'public_addr': '172.21.15.118:6809/13248188', 'cluster_addr': '172.21.15.118:6811/13248188', 'heartbeat_back_addr': '172.21.15.118:6815/13248188', 'heartbeat_front_addr': '172.21.15.118:6813/13248188', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'd45888f2-eecd-480b-80f5-c0731a06ff3d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6802', 'nonce': 2658385904}, {'type': 'v1', 'addr': '172.21.15.155:6803', 'nonce': 2658385904}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6804', 'nonce': 2658385904}, {'type': 'v1', 'addr': '172.21.15.155:6805', 'nonce': 2658385904}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6813', 'nonce': 2658385904}, {'type': 'v1', 'addr': '172.21.15.155:6816', 'nonce': 2658385904}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6806', 'nonce': 2658385904}, {'type': 'v1', 'addr': '172.21.15.155:6809', 'nonce': 2658385904}]}, 'public_addr': '172.21.15.155:6803/2658385904', 'cluster_addr': '172.21.15.155:6805/2658385904', 'heartbeat_back_addr': '172.21.15.155:6816/2658385904', 'heartbeat_front_addr': '172.21.15.155:6809/2658385904', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'd060d939-8c94-4a07-9fd5-42af0044a16f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6800', 'nonce': 2389566155}, {'type': 'v1', 'addr': '172.21.15.33:6801', 'nonce': 2389566155}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6802', 'nonce': 2389566155}, {'type': 'v1', 'addr': '172.21.15.33:6803', 'nonce': 2389566155}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6806', 'nonce': 2389566155}, {'type': 'v1', 'addr': '172.21.15.33:6807', 'nonce': 2389566155}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6804', 'nonce': 2389566155}, {'type': 'v1', 'addr': '172.21.15.33:6805', 'nonce': 2389566155}]}, 'public_addr': '172.21.15.33:6801/2389566155', 'cluster_addr': '172.21.15.33:6803/2389566155', 'heartbeat_back_addr': '172.21.15.33:6807/2389566155', 'heartbeat_front_addr': '172.21.15.33:6805/2389566155', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '9700d2f7-ed56-4143-ba9d-6cebe524e877', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6824', 'nonce': 2821892844}, {'type': 'v1', 'addr': '172.21.15.110:6825', 'nonce': 2821892844}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6826', 'nonce': 2821892844}, {'type': 'v1', 'addr': '172.21.15.110:6827', 'nonce': 2821892844}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6830', 'nonce': 2821892844}, {'type': 'v1', 'addr': '172.21.15.110:6831', 'nonce': 2821892844}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6828', 'nonce': 2821892844}, {'type': 'v1', 'addr': '172.21.15.110:6829', 'nonce': 2821892844}]}, 'public_addr': '172.21.15.110:6825/2821892844', 'cluster_addr': '172.21.15.110:6827/2821892844', 'heartbeat_back_addr': '172.21.15.110:6831/2821892844', 'heartbeat_front_addr': '172.21.15.110:6829/2821892844', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '49324695-d83b-41ad-92e1-5348e8e9149f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 26, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6816', 'nonce': 273776428}, {'type': 'v1', 'addr': '172.21.15.118:6818', 'nonce': 273776428}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6820', 'nonce': 273776428}, {'type': 'v1', 'addr': '172.21.15.118:6822', 'nonce': 273776428}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6828', 'nonce': 273776428}, {'type': 'v1', 'addr': '172.21.15.118:6830', 'nonce': 273776428}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6824', 'nonce': 273776428}, {'type': 'v1', 'addr': '172.21.15.118:6826', 'nonce': 273776428}]}, 'public_addr': '172.21.15.118:6818/273776428', 'cluster_addr': '172.21.15.118:6822/273776428', 'heartbeat_back_addr': '172.21.15.118:6830/273776428', 'heartbeat_front_addr': '172.21.15.118:6826/273776428', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'afef160d-0911-4406-ad49-cd6d61ee8a2f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 24, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6826', 'nonce': 1006872173}, {'type': 'v1', 'addr': '172.21.15.155:6827', 'nonce': 1006872173}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6828', 'nonce': 1006872173}, {'type': 'v1', 'addr': '172.21.15.155:6829', 'nonce': 1006872173}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6832', 'nonce': 1006872173}, {'type': 'v1', 'addr': '172.21.15.155:6833', 'nonce': 1006872173}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6830', 'nonce': 1006872173}, {'type': 'v1', 'addr': '172.21.15.155:6831', 'nonce': 1006872173}]}, 'public_addr': '172.21.15.155:6827/1006872173', 'cluster_addr': '172.21.15.155:6829/1006872173', 'heartbeat_back_addr': '172.21.15.155:6833/1006872173', 'heartbeat_front_addr': '172.21.15.155:6831/1006872173', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '298396bf-31a2-4832-91a4-55fcb050130f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 26, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6816', 'nonce': 744651323}, {'type': 'v1', 'addr': '172.21.15.33:6817', 'nonce': 744651323}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6818', 'nonce': 744651323}, {'type': 'v1', 'addr': '172.21.15.33:6819', 'nonce': 744651323}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6822', 'nonce': 744651323}, {'type': 'v1', 'addr': '172.21.15.33:6823', 'nonce': 744651323}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6820', 'nonce': 744651323}, {'type': 'v1', 'addr': '172.21.15.33:6821', 'nonce': 744651323}]}, 'public_addr': '172.21.15.33:6817/744651323', 'cluster_addr': '172.21.15.33:6819/744651323', 'heartbeat_back_addr': '172.21.15.33:6823/744651323', 'heartbeat_front_addr': '172.21.15.33:6821/744651323', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'c151ae6b-279d-4174-aceb-4671fbdac3b3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 24, 'up_thru': 26, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6800', 'nonce': 872519654}, {'type': 'v1', 'addr': '172.21.15.110:6802', 'nonce': 872519654}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6804', 'nonce': 872519654}, {'type': 'v1', 'addr': '172.21.15.110:6807', 'nonce': 872519654}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6816', 'nonce': 872519654}, {'type': 'v1', 'addr': '172.21.15.110:6819', 'nonce': 872519654}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6810', 'nonce': 872519654}, {'type': 'v1', 'addr': '172.21.15.110:6813', 'nonce': 872519654}]}, 'public_addr': '172.21.15.110:6802/872519654', 'cluster_addr': '172.21.15.110:6807/872519654', 'heartbeat_back_addr': '172.21.15.110:6819/872519654', 'heartbeat_front_addr': '172.21.15.110:6813/872519654', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '0b292dfb-2029-4e1e-b206-ea7e0f6969ab', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6817', 'nonce': 4099141917}, {'type': 'v1', 'addr': '172.21.15.118:6819', 'nonce': 4099141917}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6821', 'nonce': 4099141917}, {'type': 'v1', 'addr': '172.21.15.118:6823', 'nonce': 4099141917}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6829', 'nonce': 4099141917}, {'type': 'v1', 'addr': '172.21.15.118:6831', 'nonce': 4099141917}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6825', 'nonce': 4099141917}, {'type': 'v1', 'addr': '172.21.15.118:6827', 'nonce': 4099141917}]}, 'public_addr': '172.21.15.118:6819/4099141917', 'cluster_addr': '172.21.15.118:6823/4099141917', 'heartbeat_back_addr': '172.21.15.118:6831/4099141917', 'heartbeat_front_addr': '172.21.15.118:6827/4099141917', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '10c61619-ad11-40f0-b013-9ef52bcabf59', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 26, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6808', 'nonce': 3981951559}, {'type': 'v1', 'addr': '172.21.15.155:6811', 'nonce': 3981951559}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6814', 'nonce': 3981951559}, {'type': 'v1', 'addr': '172.21.15.155:6817', 'nonce': 3981951559}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6824', 'nonce': 3981951559}, {'type': 'v1', 'addr': '172.21.15.155:6825', 'nonce': 3981951559}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6820', 'nonce': 3981951559}, {'type': 'v1', 'addr': '172.21.15.155:6823', 'nonce': 3981951559}]}, 'public_addr': '172.21.15.155:6811/3981951559', 'cluster_addr': '172.21.15.155:6817/3981951559', 'heartbeat_back_addr': '172.21.15.155:6825/3981951559', 'heartbeat_front_addr': '172.21.15.155:6823/3981951559', 'state': ['exists', 'up']}, {'osd': 12, 'uuid': '3f0d26e5-f3e2-47b6-91a8-0daa30d0544f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6808', 'nonce': 4133738119}, {'type': 'v1', 'addr': '172.21.15.33:6809', 'nonce': 4133738119}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6810', 'nonce': 4133738119}, {'type': 'v1', 'addr': '172.21.15.33:6811', 'nonce': 4133738119}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6814', 'nonce': 4133738119}, {'type': 'v1', 'addr': '172.21.15.33:6815', 'nonce': 4133738119}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6812', 'nonce': 4133738119}, {'type': 'v1', 'addr': '172.21.15.33:6813', 'nonce': 4133738119}]}, 'public_addr': '172.21.15.33:6809/4133738119', 'cluster_addr': '172.21.15.33:6811/4133738119', 'heartbeat_back_addr': '172.21.15.33:6815/4133738119', 'heartbeat_front_addr': '172.21.15.33:6813/4133738119', 'state': ['exists', 'up']}, {'osd': 13, 'uuid': 'c8d6213f-af5e-4646-9241-6f91589b0e1b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 24, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6801', 'nonce': 1583232449}, {'type': 'v1', 'addr': '172.21.15.110:6803', 'nonce': 1583232449}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6806', 'nonce': 1583232449}, {'type': 'v1', 'addr': '172.21.15.110:6809', 'nonce': 1583232449}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6818', 'nonce': 1583232449}, {'type': 'v1', 'addr': '172.21.15.110:6821', 'nonce': 1583232449}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6812', 'nonce': 1583232449}, {'type': 'v1', 'addr': '172.21.15.110:6815', 'nonce': 1583232449}]}, 'public_addr': '172.21.15.110:6803/1583232449', 'cluster_addr': '172.21.15.110:6809/1583232449', 'heartbeat_back_addr': '172.21.15.110:6821/1583232449', 'heartbeat_front_addr': '172.21.15.110:6815/1583232449', 'state': ['exists', 'up']}, {'osd': 14, 'uuid': '7b3d56df-62bf-4877-b984-6f17fe1fab33', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 24, 'up_thru': 26, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6800', 'nonce': 506416463}, {'type': 'v1', 'addr': '172.21.15.118:6801', 'nonce': 506416463}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6802', 'nonce': 506416463}, {'type': 'v1', 'addr': '172.21.15.118:6803', 'nonce': 506416463}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6806', 'nonce': 506416463}, {'type': 'v1', 'addr': '172.21.15.118:6807', 'nonce': 506416463}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6804', 'nonce': 506416463}, {'type': 'v1', 'addr': '172.21.15.118:6805', 'nonce': 506416463}]}, 'public_addr': '172.21.15.118:6801/506416463', 'cluster_addr': '172.21.15.118:6803/506416463', 'heartbeat_back_addr': '172.21.15.118:6807/506416463', 'heartbeat_front_addr': '172.21.15.118:6805/506416463', 'state': ['exists', 'up']}, {'osd': 15, 'uuid': '6d526edb-f1d8-415a-ad55-81e810ff4944', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 26, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6807', 'nonce': 741132602}, {'type': 'v1', 'addr': '172.21.15.155:6810', 'nonce': 741132602}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6812', 'nonce': 741132602}, {'type': 'v1', 'addr': '172.21.15.155:6815', 'nonce': 741132602}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6821', 'nonce': 741132602}, {'type': 'v1', 'addr': '172.21.15.155:6822', 'nonce': 741132602}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6818', 'nonce': 741132602}, {'type': 'v1', 'addr': '172.21.15.155:6819', 'nonce': 741132602}]}, 'public_addr': '172.21.15.155:6810/741132602', 'cluster_addr': '172.21.15.155:6815/741132602', 'heartbeat_back_addr': '172.21.15.155:6822/741132602', 'heartbeat_front_addr': '172.21.15.155:6819/741132602', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-19T01:03:46.813806+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-19T01:03:46.677618+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-19T01:03:46.676628+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-19T01:03:46.659122+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-19T01:03:46.648122+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-19T01:03:46.859759+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-19T01:03:46.759643+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-19T01:03:46.897847+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-19T01:03:46.779763+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-19T01:03:46.813460+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-19T01:03:46.479489+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-19T01:03:46.853984+0000', 'dead_epoch': 0}, {'osd': 12, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-19T01:03:46.736100+0000', 'dead_epoch': 0}, {'osd': 13, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-19T01:03:46.838084+0000', 'dead_epoch': 0}, {'osd': 14, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-19T01:03:46.629904+0000', 'dead_epoch': 0}, {'osd': 15, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-19T01:03:46.712166+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-19T01:03:59.396 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T01:03:59.397 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-04-19T01:03:59.397 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T01:03:59.397 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:03:59.397 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:03:59.398 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-04-19T01:03:59.398 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T01:03:59.401 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-19T01:03:59.401 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2024-04-19T01:03:59.401 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T01:03:59.402 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:03:59.393+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:03:59.402 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2024-04-19T01:03:59.402 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T01:03:59.403 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2024-04-19T01:03:59.404 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T01:03:59.407 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2024-04-19T01:03:59.407 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T01:03:59.425 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2024-04-19T01:03:59.426 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-19T01:03:59.446 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2024-04-19T01:03:59.447 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-19T01:03:59.469 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2024-04-19T01:03:59.469 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-19T01:03:59.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:03:59.468+0000 7f5edb59e640 0 -- 172.21.15.33:0/297981443 >> [v2:172.21.15.155:6807/741132602,v1:172.21.15.155:6810/741132602] conn(0x7f5eb4033670 msgr2=0x7f5eb4035a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-04-19T01:03:59.495 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:03:59.497+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:03:59.519 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:03:59.519 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:03:59.533 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-19T01:03:59.596 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:03:59.597+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:03:59.693 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2024-04-19T01:03:59.693 INFO:teuthology.orchestra.run.smithi033.stderr:load: shec 2024-04-19T01:03:59.696 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:03:59.696 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:03:59.697 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2024-04-19T01:03:59.697 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-19T01:03:59.698 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:03:59.696+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:03:59.704 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:03:59.705 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-04-19T01:03:59.731 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-19T01:03:59.732 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-04-19T01:03:59.696824+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":13,"kb":1509949440,"kb_used":453240,"kb_used_data":4240,"kb_used_omap":24,"kb_used_meta":448743,"kb_avail":1509496200,"statfs":{"total":1546188226560,"available":1545724108800,"internally_reserved":0,"allocated":4341760,"data_stored":2885764,"data_compressed":160728,"data_compressed_allocated":385024,"data_compressed_original":1662976,"omap_allocated":25520,"internal_metadata":459512912},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.041542"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-19T01:03:54.658079+0000","last_change":"2024-04-19T01:03:53.939441+0000","last_active":"2024-04-19T01:03:54.658079+0000","last_peered":"2024-04-19T01:03:54.658079+0000","last_clean":"2024-04-19T01:03:54.658079+0000","last_became_active":"2024-04-19T01:03:53.938774+0000","last_became_peered":"2024-04-19T01:03:53.938774+0000","last_unstale":"2024-04-19T01:03:54.658079+0000","last_undegraded":"2024-04-19T01:03:54.658079+0000","last_fullsized":"2024-04-19T01:03:54.658079+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:14.081709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-19T01:03:55.308912+0000","last_change":"2024-04-19T01:03:53.736290+0000","last_active":"2024-04-19T01:03:55.308912+0000","last_peered":"2024-04-19T01:03:55.308912+0000","last_clean":"2024-04-19T01:03:55.308912+0000","last_became_active":"2024-04-19T01:03:53.735549+0000","last_became_peered":"2024-04-19T01:03:53.735549+0000","last_unstale":"2024-04-19T01:03:55.308912+0000","last_undegraded":"2024-04-19T01:03:55.308912+0000","last_fullsized":"2024-04-19T01:03:55.308912+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:07.640918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13],"acting":[14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-19T01:03:55.734940+0000","last_change":"2024-04-19T01:03:53.772709+0000","last_active":"2024-04-19T01:03:55.734940+0000","last_peered":"2024-04-19T01:03:55.734940+0000","last_clean":"2024-04-19T01:03:55.734940+0000","last_became_active":"2024-04-19T01:03:53.771887+0000","last_became_peered":"2024-04-19T01:03:53.771887+0000","last_unstale":"2024-04-19T01:03:55.734940+0000","last_undegraded":"2024-04-19T01:03:55.734940+0000","last_fullsized":"2024-04-19T01:03:55.734940+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:17.479766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-19T01:03:54.659881+0000","last_change":"2024-04-19T01:03:53.716606+0000","last_active":"2024-04-19T01:03:54.659881+0000","last_peered":"2024-04-19T01:03:54.659881+0000","last_clean":"2024-04-19T01:03:54.659881+0000","last_became_active":"2024-04-19T01:03:53.716040+0000","last_became_peered":"2024-04-19T01:03:53.716040+0000","last_unstale":"2024-04-19T01:03:54.659881+0000","last_undegraded":"2024-04-19T01:03:54.659881+0000","last_fullsized":"2024-04-19T01:03:54.659881+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:01.975515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-19T01:03:54.658077+0000","last_change":"2024-04-19T01:03:54.012138+0000","last_active":"2024-04-19T01:03:54.658077+0000","last_peered":"2024-04-19T01:03:54.658077+0000","last_clean":"2024-04-19T01:03:54.658077+0000","last_became_active":"2024-04-19T01:03:54.011460+0000","last_became_peered":"2024-04-19T01:03:54.011460+0000","last_unstale":"2024-04-19T01:03:54.658077+0000","last_undegraded":"2024-04-19T01:03:54.658077+0000","last_fullsized":"2024-04-19T01:03:54.658077+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:04:58.336612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-19T01:03:57.525932+0000","last_change":"2024-04-19T01:03:53.650434+0000","last_active":"2024-04-19T01:03:57.525932+0000","last_peered":"2024-04-19T01:03:57.525932+0000","last_clean":"2024-04-19T01:03:57.525932+0000","last_became_active":"2024-04-19T01:03:53.649606+0000","last_became_peered":"2024-04-19T01:03:53.649606+0000","last_unstale":"2024-04-19T01:03:57.525932+0000","last_undegraded":"2024-04-19T01:03:57.525932+0000","last_fullsized":"2024-04-19T01:03:57.525932+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:17.424729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-19T01:03:54.657910+0000","last_change":"2024-04-19T01:03:53.760075+0000","last_active":"2024-04-19T01:03:54.657910+0000","last_peered":"2024-04-19T01:03:54.657910+0000","last_clean":"2024-04-19T01:03:54.657910+0000","last_became_active":"2024-04-19T01:03:53.759326+0000","last_became_peered":"2024-04-19T01:03:53.759326+0000","last_unstale":"2024-04-19T01:03:54.657910+0000","last_undegraded":"2024-04-19T01:03:54.657910+0000","last_fullsized":"2024-04-19T01:03:54.657910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:02.718012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-19T01:03:56.166318+0000","last_change":"2024-04-19T01:03:53.766587+0000","last_active":"2024-04-19T01:03:56.166318+0000","last_peered":"2024-04-19T01:03:56.166318+0000","last_clean":"2024-04-19T01:03:56.166318+0000","last_became_active":"2024-04-19T01:03:53.765967+0000","last_became_peered":"2024-04-19T01:03:53.765967+0000","last_unstale":"2024-04-19T01:03:56.166318+0000","last_undegraded":"2024-04-19T01:03:56.166318+0000","last_fullsized":"2024-04-19T01:03:56.166318+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:18.833542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"25'157","reported_seq":214,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-19T01:03:56.166162+0000","last_change":"2024-04-19T01:03:50.776344+0000","last_active":"2024-04-19T01:03:56.166162+0000","last_peered":"2024-04-19T01:03:56.166162+0000","last_clean":"2024-04-19T01:03:56.166162+0000","last_became_active":"2024-04-19T01:03:50.774078+0000","last_became_peered":"2024-04-19T01:03:50.774078+0000","last_unstale":"2024-04-19T01:03:56.166162+0000","last_undegraded":"2024-04-19T01:03:56.166162+0000","last_fullsized":"2024-04-19T01:03:56.166162+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:49.580851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:49.580851+0000","last_clean_scrub_stamp":"2024-04-19T01:03:49.580851+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:07.495976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":13},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":15546,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":14,"up_from":24,"seq":103079215109,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31440,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340400,"statfs":{"total":96636764160,"available":96604569600,"internally_reserved":0,"allocated":266240,"data_stored":115427,"data_compressed":9308,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,7,8,9,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":13,"up_from":24,"seq":103079215110,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31452,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340388,"statfs":{"total":96636764160,"available":96604557312,"internally_reserved":0,"allocated":266240,"data_stored":115427,"data_compressed":9308,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,8,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":24,"seq":103079215111,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31440,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340400,"statfs":{"total":96636764160,"available":96604569600,"internally_reserved":0,"allocated":266240,"data_stored":115427,"data_compressed":9308,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313222,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":115427,"data_compressed":9308,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[3,4,5,6,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":115427,"data_compressed":9308,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,7,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":115427,"data_compressed":9308,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,5,6,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":115427,"data_compressed":9308,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[2,4,5,6,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":115427,"data_compressed":9308,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27332,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344508,"statfs":{"total":96636764160,"available":96608776192,"internally_reserved":0,"allocated":319488,"data_stored":644355,"data_compressed":17081,"data_compressed_allocated":53248,"data_compressed_original":552960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,6,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280518,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":115427,"data_compressed":9308,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[3,4,5,6,7,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27332,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344508,"statfs":{"total":96636764160,"available":96608776192,"internally_reserved":0,"allocated":319488,"data_stored":644355,"data_compressed":17081,"data_compressed_allocated":53248,"data_compressed_original":552960,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,3,4,5,6,8,9,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":115427,"data_compressed":9308,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,8,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":115427,"data_compressed":9308,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,4,5,6,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":24,"seq":103079215109,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31428,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340412,"statfs":{"total":96636764160,"available":96604581888,"internally_reserved":0,"allocated":253952,"data_stored":105965,"data_compressed":7435,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215110,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":105965,"data_compressed":7435,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,10,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":24,"seq":103079215109,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":115427,"data_compressed":9308,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T01:03:59.733 INFO:tasks.thrashosds.thrasher:Setting 2.1 to [13, 6, 0, 5] 2024-04-19T01:03:59.733 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.1', '13', '6', '0', '5'] 2024-04-19T01:03:59.734 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 2.1 13 6 0 5 2024-04-19T01:03:59.737 INFO:tasks.rados:starting run 0 out of 1 2024-04-19T01:03:59.738 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-04-19T01:03:59.738 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 16 erasure shecprofile 2024-04-19T01:03:59.797 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:03:59.800+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:03:59.887 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:03:59.887 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:03:59.899 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:03:59.896+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:03:59.904 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-19T01:03:59.999 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:04:00.001+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:04:00.065 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:00.066 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:00.078 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-19T01:04:00.101 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:04:00.101+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:04:00.188 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:00.189 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:00.201 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-19T01:04:00.202 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:04:00.201+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:04:00.302 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:04:00.300+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:04:00.322 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:00.323 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:00.336 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-04-19T01:04:00.424 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:04:00.424+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:04:00.504 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:00.505 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:00.505 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:04:00.504+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:04:00.520 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-04-19T01:04:00.605 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:04:00.605+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:04:00.696 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:00.696 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:00.706 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:04:00.708+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:04:00.709 INFO:teuthology.orchestra.run.smithi033.stderr:set 2.1 pg_upmap_items mapping to [13->6,0->5] 2024-04-19T01:04:00.711 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-04-19T01:04:00.807 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:04:00.809+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:04:00.820 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:00.820 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:00.832 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-19T01:04:00.908 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:04:00.909+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:04:00.963 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:00.963 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:00.977 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-19T01:04:01.022 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:04:01.024+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:04:01.098 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:01.098 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:01.112 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-19T01:04:01.113 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:04:01.108+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:04:01.212 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:04:01.213+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:04:01.270 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:01.270 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:01.284 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-19T01:04:01.330 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:04:01.332+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:04:01.410 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:01.410 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:01.414 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:04:01.412+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:04:01.424 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-19T01:04:01.515 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:04:01.517+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:04:01.590 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:01.590 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:01.604 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-19T01:04:01.617 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:04:01.616+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:04:01.719 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:04:01.721+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:04:01.720 INFO:teuthology.orchestra.run.smithi033.stderr:pool 'unique_pool_0' created 2024-04-19T01:04:01.722 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-19T01:04:01.723 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-19T01:04:01.724 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:01.724 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:01.741 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-19T01:04:01.753 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-04-19T01:04:01.819 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:01.821+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:01.867 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:01.868 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:01.885 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-04-19T01:04:01.886 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T01:04:01.886 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-19T01:04:01.917 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-04-19T01:04:01.917 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T01:04:01.921 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:04:01.921+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:04:01.938 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-04-19T01:04:01.938 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T01:04:01.972 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-04-19T01:04:01.972 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T01:04:01.992 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-04-19T01:04:01.992 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T01:04:02.009 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:02.010 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:02.011 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-04-19T01:04:02.011 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T01:04:02.021 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:04:02.020+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:04:02.024 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-19T01:04:02.032 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-04-19T01:04:02.033 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T01:04:02.053 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-04-19T01:04:02.053 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T01:04:02.070 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2024-04-19T01:04:02.070 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T01:04:02.089 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2024-04-19T01:04:02.090 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T01:04:02.108 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2024-04-19T01:04:02.108 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T01:04:02.122 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:04:02.124+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:04:02.129 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2024-04-19T01:04:02.129 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T01:04:02.145 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2024-04-19T01:04:02.145 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-19T01:04:02.150 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:02.150 INFO:teuthology.orchestra.run.smithi110.stdout: "size": 20, 2024-04-19T01:04:02.150 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 600, 2024-04-19T01:04:02.150 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [] 2024-04-19T01:04:02.150 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:02.163 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-04-19T01:04:02.165 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2024-04-19T01:04:02.165 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-19T01:04:02.185 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2024-04-19T01:04:02.185 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-19T01:04:02.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:04:02.188+0000 7f00837fe640 0 -- 172.21.15.33:0/1377012906 >> [v2:172.21.15.155:6807/741132602,v1:172.21.15.155:6810/741132602] conn(0x7f0084226680 msgr2=0x7f0084228a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-19T01:04:02.207 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2024-04-19T01:04:02.208 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-19T01:04:02.223 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:04:02.220+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:04:02.324 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:04:02.324+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:04:02.328 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:02.329 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [], 2024-04-19T01:04:02.329 INFO:teuthology.orchestra.run.smithi110.stdout: "num_ops": 0 2024-04-19T01:04:02.329 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:02.342 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-04-19T01:04:02.425 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:02.428+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:02.466 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:02.466 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [], 2024-04-19T01:04:02.466 INFO:teuthology.orchestra.run.smithi110.stdout: "complaint_time": 30, 2024-04-19T01:04:02.466 INFO:teuthology.orchestra.run.smithi110.stdout: "num_blocked_ops": 0 2024-04-19T01:04:02.466 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:02.481 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-04-19T01:04:02.527 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:04:02.529+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:04:02.628 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:02.628+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:02.651 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:02.651 INFO:teuthology.orchestra.run.smithi110.stdout: "size": 20, 2024-04-19T01:04:02.651 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 600, 2024-04-19T01:04:02.651 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [] 2024-04-19T01:04:02.651 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:02.665 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-19T01:04:02.728 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:02.729+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:02.729 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2024-04-19T01:04:02.729 INFO:teuthology.orchestra.run.smithi033.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-04-19T01:04:02.803 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:04:02.804 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-04-19T01:04:02.804 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-04-19T01:04:02.804 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:04:02.818 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-19T01:04:02.829 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:04:02.828+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:04:02.931 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:04:02.929+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:04:02.990 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:04:02.990 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-04-19T01:04:02.990 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-04-19T01:04:02.990 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-04-19T01:04:02.990 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:04:03.003 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-19T01:04:03.031 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:04:03.032+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:04:03.122 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:04:03.122 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-04-19T01:04:03.122 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-04-19T01:04:03.122 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [] 2024-04-19T01:04:03.122 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:04:03.132 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:03.133+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:03.136 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-19T01:04:03.232 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:04:03.233+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:04:03.296 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:04:03.297 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-04-19T01:04:03.297 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-04-19T01:04:03.297 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:04:03.311 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-19T01:04:03.333 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:04:03.332+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:04:03.434 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:04:03.432+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:04:03.472 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:04:03.472 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-04-19T01:04:03.472 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-04-19T01:04:03.472 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-04-19T01:04:03.472 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:04:03.487 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-19T01:04:03.535 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:04:03.537+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:04:03.610 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:04:03.611 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-04-19T01:04:03.611 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-04-19T01:04:03.611 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [] 2024-04-19T01:04:03.611 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:04:03.627 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-19T01:04:03.671 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:04:03.673+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:04:03.738 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:04:03.739+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:04:03.767 DEBUG:teuthology.orchestra.run.smithi155:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 400 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0 2024-04-19T01:04:03.788 INFO:tasks.rados.rados.0.smithi155.stdout:adding op weight read -> 100 2024-04-19T01:04:03.789 INFO:tasks.rados.rados.0.smithi155.stdout:adding op weight delete -> 50 2024-04-19T01:04:03.789 INFO:tasks.rados.rados.0.smithi155.stdout:adding op weight snap_create -> 50 2024-04-19T01:04:03.789 INFO:tasks.rados.rados.0.smithi155.stdout:adding op weight snap_remove -> 50 2024-04-19T01:04:03.789 INFO:tasks.rados.rados.0.smithi155.stdout:adding op weight rollback -> 50 2024-04-19T01:04:03.789 INFO:tasks.rados.rados.0.smithi155.stdout:adding op weight setattr -> 25 2024-04-19T01:04:03.789 INFO:tasks.rados.rados.0.smithi155.stdout:adding op weight rmattr -> 25 2024-04-19T01:04:03.789 INFO:tasks.rados.rados.0.smithi155.stdout:adding op weight copy_from -> 50 2024-04-19T01:04:03.789 INFO:tasks.rados.rados.0.smithi155.stdout:adding op weight append -> 100 2024-04-19T01:04:03.789 INFO:tasks.rados.rados.0.smithi155.stdout:ceph version 19.0.0-3154-g8b4ab94f (8b4ab94f9c30da63487fa2f241ce12f483378860) squid (dev) 2024-04-19T01:04:03.789 INFO:tasks.rados.rados.0.smithi155.stdout:Configuration: 2024-04-19T01:04:03.789 INFO:tasks.rados.rados.0.smithi155.stdout: Number of operations: 400 2024-04-19T01:04:03.789 INFO:tasks.rados.rados.0.smithi155.stdout: Number of objects: 50 2024-04-19T01:04:03.790 INFO:tasks.rados.rados.0.smithi155.stdout: Max in flight operations: 16 2024-04-19T01:04:03.790 INFO:tasks.rados.rados.0.smithi155.stdout: Object size (in bytes): 4000000 2024-04-19T01:04:03.790 INFO:tasks.rados.rados.0.smithi155.stdout: Write stride min: 400000 2024-04-19T01:04:03.790 INFO:tasks.rados.rados.0.smithi155.stdout: Write stride max: 800000 2024-04-19T01:04:03.798 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:04:03.799 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-04-19T01:04:03.799 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-04-19T01:04:03.799 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:04:03.814 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-19T01:04:03.838 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:04:03.839+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:04:03.935 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:04:03.935 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-04-19T01:04:03.935 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-04-19T01:04:03.935 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-04-19T01:04:03.935 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:04:03.939 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:04:03.937+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:04:03.949 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-19T01:04:04.040 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:04:04.041+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:04:04.065 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:04:04.065 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-04-19T01:04:04.065 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-04-19T01:04:04.065 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [] 2024-04-19T01:04:04.065 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:04:04.079 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-04-19T01:04:04.166 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:04:04.169+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:04:04.225 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-19T01:04:04.237 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:04:04.237 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-04-19T01:04:04.238 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-04-19T01:04:04.238 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:04:04.241 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:04.241+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:04.253 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-04-19T01:04:04.342 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:04:04.341+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:04:04.389 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-04-19T01:04:04.389 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T01:04:04.412 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-04-19T01:04:04.412 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T01:04:04.424 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:04:04.424 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-04-19T01:04:04.424 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-04-19T01:04:04.424 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-04-19T01:04:04.424 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:04:04.434 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-04-19T01:04:04.435 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T01:04:04.439 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-04-19T01:04:04.443 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:04:04.440+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:04:04.459 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-04-19T01:04:04.460 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T01:04:04.481 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-04-19T01:04:04.482 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T01:04:04.502 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-04-19T01:04:04.503 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T01:04:04.526 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-04-19T01:04:04.526 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T01:04:04.545 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:04:04.545+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:04:04.551 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-04-19T01:04:04.551 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T01:04:04.556 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:04:04.557 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-04-19T01:04:04.557 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-04-19T01:04:04.557 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [] 2024-04-19T01:04:04.557 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:04:04.571 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-19T01:04:04.572 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2024-04-19T01:04:04.573 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T01:04:04.597 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2024-04-19T01:04:04.597 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T01:04:04.619 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2024-04-19T01:04:04.619 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T01:04:04.642 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2024-04-19T01:04:04.642 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T01:04:04.645 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:04:04.645+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:04:04.664 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2024-04-19T01:04:04.664 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-19T01:04:04.689 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2024-04-19T01:04:04.689 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-19T01:04:04.705 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:04.705 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:04.713 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2024-04-19T01:04:04.714 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-19T01:04:04.719 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-19T01:04:04.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:04:04.732+0000 7f35e91f3640 0 -- 172.21.15.33:0/3146091135 >> [v2:172.21.15.155:6807/741132602,v1:172.21.15.155:6810/741132602] conn(0x7f35e42245f0 msgr2=0x7f35e42269e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-19T01:04:04.739 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2024-04-19T01:04:04.747 INFO:tasks.rados.rados.0.smithi155.stdout:1: write initial oid 1 2024-04-19T01:04:04.747 INFO:tasks.rados.rados.0.smithi155.stdout:1: seq_num 0 ranges {0=32768} 2024-04-19T01:04:04.747 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:04:04.745+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:04:04.748 INFO:tasks.rados.rados.0.smithi155.stdout:1: writing smithi15528762-1 from 0 to 32768 tid 1 2024-04-19T01:04:04.748 INFO:tasks.rados.rados.0.smithi155.stdout:2: write initial oid 2 2024-04-19T01:04:04.748 INFO:tasks.rados.rados.0.smithi155.stdout:2: seq_num 1 ranges {0=32768} 2024-04-19T01:04:04.749 INFO:tasks.rados.rados.0.smithi155.stdout:2: writing smithi15528762-2 from 0 to 32768 tid 1 2024-04-19T01:04:04.749 INFO:tasks.rados.rados.0.smithi155.stdout:3: write initial oid 3 2024-04-19T01:04:04.749 INFO:tasks.rados.rados.0.smithi155.stdout:3: seq_num 2 ranges {0=49152} 2024-04-19T01:04:04.749 INFO:tasks.rados.rados.0.smithi155.stdout:3: writing smithi15528762-3 from 0 to 49152 tid 1 2024-04-19T01:04:04.749 INFO:tasks.rados.rados.0.smithi155.stdout:4: write initial oid 4 2024-04-19T01:04:04.749 INFO:tasks.rados.rados.0.smithi155.stdout:4: seq_num 3 ranges {0=65536} 2024-04-19T01:04:04.750 INFO:tasks.rados.rados.0.smithi155.stdout:4: writing smithi15528762-4 from 0 to 65536 tid 1 2024-04-19T01:04:04.751 INFO:tasks.rados.rados.0.smithi155.stdout:5: write initial oid 5 2024-04-19T01:04:04.751 INFO:tasks.rados.rados.0.smithi155.stdout:5: seq_num 4 ranges {0=81920} 2024-04-19T01:04:04.751 INFO:tasks.rados.rados.0.smithi155.stdout:5: writing smithi15528762-5 from 0 to 81920 tid 1 2024-04-19T01:04:04.751 INFO:tasks.rados.rados.0.smithi155.stdout:6: write initial oid 6 2024-04-19T01:04:04.751 INFO:tasks.rados.rados.0.smithi155.stdout:6: seq_num 5 ranges {0=98304} 2024-04-19T01:04:04.751 INFO:tasks.rados.rados.0.smithi155.stdout:6: writing smithi15528762-6 from 0 to 98304 tid 1 2024-04-19T01:04:04.751 INFO:tasks.rados.rados.0.smithi155.stdout:7: write initial oid 7 2024-04-19T01:04:04.751 INFO:tasks.rados.rados.0.smithi155.stdout:7: seq_num 6 ranges {0=114688} 2024-04-19T01:04:04.752 INFO:tasks.rados.rados.0.smithi155.stdout:7: writing smithi15528762-7 from 0 to 114688 tid 1 2024-04-19T01:04:04.752 INFO:tasks.rados.rados.0.smithi155.stdout:8: write initial oid 8 2024-04-19T01:04:04.752 INFO:tasks.rados.rados.0.smithi155.stdout:8: seq_num 7 ranges {0=131072} 2024-04-19T01:04:04.752 INFO:tasks.rados.rados.0.smithi155.stdout:8: writing smithi15528762-8 from 0 to 131072 tid 1 2024-04-19T01:04:04.752 INFO:tasks.rados.rados.0.smithi155.stdout:9: write initial oid 9 2024-04-19T01:04:04.752 INFO:tasks.rados.rados.0.smithi155.stdout:9: seq_num 8 ranges {0=147456} 2024-04-19T01:04:04.754 INFO:tasks.rados.rados.0.smithi155.stdout:9: writing smithi15528762-9 from 0 to 147456 tid 1 2024-04-19T01:04:04.754 INFO:tasks.rados.rados.0.smithi155.stdout:10: write initial oid 10 2024-04-19T01:04:04.754 INFO:tasks.rados.rados.0.smithi155.stdout:10: seq_num 9 ranges {0=163840} 2024-04-19T01:04:04.756 INFO:tasks.rados.rados.0.smithi155.stdout:10: writing smithi15528762-10 from 0 to 163840 tid 1 2024-04-19T01:04:04.756 INFO:tasks.rados.rados.0.smithi155.stdout:11: write initial oid 11 2024-04-19T01:04:04.756 INFO:tasks.rados.rados.0.smithi155.stdout:11: seq_num 10 ranges {0=180224} 2024-04-19T01:04:04.758 INFO:tasks.rados.rados.0.smithi155.stdout:11: writing smithi15528762-11 from 0 to 180224 tid 1 2024-04-19T01:04:04.758 INFO:tasks.rados.rados.0.smithi155.stdout:12: write initial oid 12 2024-04-19T01:04:04.758 INFO:tasks.rados.rados.0.smithi155.stdout:12: seq_num 11 ranges {0=196608} 2024-04-19T01:04:04.760 INFO:tasks.rados.rados.0.smithi155.stdout:12: writing smithi15528762-12 from 0 to 196608 tid 1 2024-04-19T01:04:04.760 INFO:tasks.rados.rados.0.smithi155.stdout:13: write initial oid 13 2024-04-19T01:04:04.760 INFO:tasks.rados.rados.0.smithi155.stdout:13: seq_num 12 ranges {0=212992} 2024-04-19T01:04:04.762 INFO:tasks.rados.rados.0.smithi155.stdout:13: writing smithi15528762-13 from 0 to 212992 tid 1 2024-04-19T01:04:04.762 INFO:tasks.rados.rados.0.smithi155.stdout:14: write initial oid 14 2024-04-19T01:04:04.763 INFO:tasks.rados.rados.0.smithi155.stdout:14: seq_num 13 ranges {0=229376} 2024-04-19T01:04:04.764 INFO:tasks.rados.rados.0.smithi155.stdout:14: writing smithi15528762-14 from 0 to 229376 tid 1 2024-04-19T01:04:04.764 INFO:tasks.rados.rados.0.smithi155.stdout:15: write initial oid 15 2024-04-19T01:04:04.764 INFO:tasks.rados.rados.0.smithi155.stdout:15: seq_num 14 ranges {0=245760} 2024-04-19T01:04:04.766 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-19T01:04:04.768 INFO:tasks.rados.rados.0.smithi155.stdout:15: writing smithi15528762-15 from 0 to 245760 tid 1 2024-04-19T01:04:04.768 INFO:tasks.rados.rados.0.smithi155.stdout:16: write initial oid 16 2024-04-19T01:04:04.768 INFO:tasks.rados.rados.0.smithi155.stdout:16: seq_num 15 ranges {0=262144} 2024-04-19T01:04:04.770 INFO:tasks.rados.rados.0.smithi155.stdout:16: writing smithi15528762-16 from 0 to 262144 tid 1 2024-04-19T01:04:04.770 INFO:tasks.rados.rados.0.smithi155.stdout: waiting on 16 2024-04-19T01:04:04.774 INFO:tasks.rados.rados.0.smithi155.stdout:5: finishing write tid 1 to smithi15528762-5 2024-04-19T01:04:04.774 INFO:tasks.rados.rados.0.smithi155.stdout:1: finishing write tid 1 to smithi15528762-1 2024-04-19T01:04:04.778 INFO:tasks.rados.rados.0.smithi155.stdout:1: finishing write tid 2 to smithi15528762-1 2024-04-19T01:04:04.780 INFO:tasks.rados.rados.0.smithi155.stdout:3: finishing write tid 1 to smithi15528762-3 2024-04-19T01:04:04.780 INFO:tasks.rados.rados.0.smithi155.stdout:7: finishing write tid 1 to smithi15528762-7 2024-04-19T01:04:04.780 INFO:tasks.rados.rados.0.smithi155.stdout:6: finishing write tid 1 to smithi15528762-6 2024-04-19T01:04:04.781 INFO:tasks.rados.rados.0.smithi155.stdout:4: finishing write tid 1 to smithi15528762-4 2024-04-19T01:04:04.781 INFO:tasks.rados.rados.0.smithi155.stdout:1: finishing write tid 3 to smithi15528762-1 2024-04-19T01:04:04.781 INFO:tasks.rados.rados.0.smithi155.stdout:1: oid 1 updating version 0 to 1 2024-04-19T01:04:04.781 INFO:tasks.rados.rados.0.smithi155.stdout:1: oid 1 updating version 1 to 2 2024-04-19T01:04:04.781 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 1 v 2 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-19T01:04:04.781 INFO:tasks.rados.rados.0.smithi155.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2024-04-19T01:04:04.781 INFO:tasks.rados.rados.0.smithi155.stdout:1: done (15 left) 2024-04-19T01:04:04.782 INFO:tasks.rados.rados.0.smithi155.stdout:17: write initial oid 17 2024-04-19T01:04:04.782 INFO:tasks.rados.rados.0.smithi155.stdout:17: seq_num 16 ranges {0=278528} 2024-04-19T01:04:04.786 INFO:tasks.rados.rados.0.smithi155.stdout:17: writing smithi15528762-17 from 0 to 278528 tid 1 2024-04-19T01:04:04.787 INFO:tasks.rados.rados.0.smithi155.stdout: waiting on 16 2024-04-19T01:04:04.787 INFO:tasks.rados.rados.0.smithi155.stdout:7: finishing write tid 2 to smithi15528762-7 2024-04-19T01:04:04.787 INFO:tasks.rados.rados.0.smithi155.stdout:6: finishing write tid 2 to smithi15528762-6 2024-04-19T01:04:04.787 INFO:tasks.rados.rados.0.smithi155.stdout:5: finishing write tid 2 to smithi15528762-5 2024-04-19T01:04:04.787 INFO:tasks.rados.rados.0.smithi155.stdout:10: finishing write tid 1 to smithi15528762-10 2024-04-19T01:04:04.787 INFO:tasks.rados.rados.0.smithi155.stdout:10: finishing write tid 2 to smithi15528762-10 2024-04-19T01:04:04.787 INFO:tasks.rados.rados.0.smithi155.stdout:3: finishing write tid 2 to smithi15528762-3 2024-04-19T01:04:04.788 INFO:tasks.rados.rados.0.smithi155.stdout:2: finishing write tid 1 to smithi15528762-2 2024-04-19T01:04:04.790 INFO:tasks.rados.rados.0.smithi155.stdout:10: finishing write tid 3 to smithi15528762-10 2024-04-19T01:04:04.790 INFO:tasks.rados.rados.0.smithi155.stdout:10: oid 10 updating version 0 to 1 2024-04-19T01:04:04.790 INFO:tasks.rados.rados.0.smithi155.stdout:10: oid 10 updating version 1 to 2 2024-04-19T01:04:04.790 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 10 v 2 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-04-19T01:04:04.790 INFO:tasks.rados.rados.0.smithi155.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2024-04-19T01:04:04.790 INFO:tasks.rados.rados.0.smithi155.stdout:10: done (15 left) 2024-04-19T01:04:04.790 INFO:tasks.rados.rados.0.smithi155.stdout:18: write initial oid 18 2024-04-19T01:04:04.790 INFO:tasks.rados.rados.0.smithi155.stdout:18: seq_num 17 ranges {0=294912} 2024-04-19T01:04:04.793 INFO:tasks.rados.rados.0.smithi155.stdout:18: writing smithi15528762-18 from 0 to 294912 tid 1 2024-04-19T01:04:04.793 INFO:tasks.rados.rados.0.smithi155.stdout: waiting on 16 2024-04-19T01:04:04.793 INFO:tasks.rados.rados.0.smithi155.stdout:5: finishing write tid 3 to smithi15528762-5 2024-04-19T01:04:04.793 INFO:tasks.rados.rados.0.smithi155.stdout:5: oid 5 updating version 0 to 1 2024-04-19T01:04:04.793 INFO:tasks.rados.rados.0.smithi155.stdout:5: oid 5 updating version 1 to 2 2024-04-19T01:04:04.793 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 5 v 2 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-04-19T01:04:04.793 INFO:tasks.rados.rados.0.smithi155.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2024-04-19T01:04:04.793 INFO:tasks.rados.rados.0.smithi155.stdout:6: finishing write tid 3 to smithi15528762-6 2024-04-19T01:04:04.793 INFO:tasks.rados.rados.0.smithi155.stdout:6: oid 6 updating version 0 to 1 2024-04-19T01:04:04.793 INFO:tasks.rados.rados.0.smithi155.stdout:6: oid 6 updating version 1 to 2 2024-04-19T01:04:04.793 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 6 v 2 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-04-19T01:04:04.794 INFO:tasks.rados.rados.0.smithi155.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2024-04-19T01:04:04.794 INFO:tasks.rados.rados.0.smithi155.stdout:5: done (15 left) 2024-04-19T01:04:04.794 INFO:tasks.rados.rados.0.smithi155.stdout:6: done (14 left) 2024-04-19T01:04:04.794 INFO:tasks.rados.rados.0.smithi155.stdout:19: write initial oid 19 2024-04-19T01:04:04.794 INFO:tasks.rados.rados.0.smithi155.stdout:19: seq_num 18 ranges {0=311296} 2024-04-19T01:04:04.795 INFO:tasks.rados.rados.0.smithi155.stdout:19: writing smithi15528762-19 from 0 to 311296 tid 1 2024-04-19T01:04:04.795 INFO:tasks.rados.rados.0.smithi155.stdout:11: finishing write tid 1 to smithi15528762-11 2024-04-19T01:04:04.796 INFO:tasks.rados.rados.0.smithi155.stdout:12: finishing write tid 1 to smithi15528762-12 2024-04-19T01:04:04.796 INFO:tasks.rados.rados.0.smithi155.stdout:11: finishing write tid 2 to smithi15528762-11 2024-04-19T01:04:04.796 INFO:tasks.rados.rados.0.smithi155.stdout:12: finishing write tid 2 to smithi15528762-12 2024-04-19T01:04:04.796 INFO:tasks.rados.rados.0.smithi155.stdout:9: finishing write tid 1 to smithi15528762-9 2024-04-19T01:04:04.796 INFO:tasks.rados.rados.0.smithi155.stdout:2: finishing write tid 2 to smithi15528762-2 2024-04-19T01:04:04.796 INFO:tasks.rados.rados.0.smithi155.stdout:13: finishing write tid 1 to smithi15528762-13 2024-04-19T01:04:04.796 INFO:tasks.rados.rados.0.smithi155.stdout:13: finishing write tid 2 to smithi15528762-13 2024-04-19T01:04:04.796 INFO:tasks.rados.rados.0.smithi155.stdout:8: finishing write tid 1 to smithi15528762-8 2024-04-19T01:04:04.796 INFO:tasks.rados.rados.0.smithi155.stdout:9: finishing write tid 2 to smithi15528762-9 2024-04-19T01:04:04.796 INFO:tasks.rados.rados.0.smithi155.stdout:20: write initial oid 20 2024-04-19T01:04:04.796 INFO:tasks.rados.rados.0.smithi155.stdout:20: seq_num 19 ranges {0=327680} 2024-04-19T01:04:04.799 INFO:tasks.rados.rados.0.smithi155.stdout:20: writing smithi15528762-20 from 0 to 327680 tid 1 2024-04-19T01:04:04.799 INFO:tasks.rados.rados.0.smithi155.stdout:8: finishing write tid 2 to smithi15528762-8 2024-04-19T01:04:04.799 INFO:tasks.rados.rados.0.smithi155.stdout:12: finishing write tid 3 to smithi15528762-12 2024-04-19T01:04:04.799 INFO:tasks.rados.rados.0.smithi155.stdout:12: oid 12 updating version 0 to 2 2024-04-19T01:04:04.799 INFO:tasks.rados.rados.0.smithi155.stdout:12: oid 12 version 2 is already newer than 1 2024-04-19T01:04:04.799 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 12 v 2 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-04-19T01:04:04.799 INFO:tasks.rados.rados.0.smithi155.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2024-04-19T01:04:04.799 INFO:tasks.rados.rados.0.smithi155.stdout:2: finishing write tid 3 to smithi15528762-2 2024-04-19T01:04:04.800 INFO:tasks.rados.rados.0.smithi155.stdout:2: oid 2 updating version 0 to 3 2024-04-19T01:04:04.800 INFO:tasks.rados.rados.0.smithi155.stdout:2: oid 2 updating version 3 to 4 2024-04-19T01:04:04.800 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 2 v 4 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-19T01:04:04.800 INFO:tasks.rados.rados.0.smithi155.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2024-04-19T01:04:04.800 INFO:tasks.rados.rados.0.smithi155.stdout:11: finishing write tid 3 to smithi15528762-11 2024-04-19T01:04:04.800 INFO:tasks.rados.rados.0.smithi155.stdout:11: oid 11 updating version 0 to 3 2024-04-19T01:04:04.800 INFO:tasks.rados.rados.0.smithi155.stdout:11: oid 11 updating version 3 to 4 2024-04-19T01:04:04.800 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 11 v 4 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-04-19T01:04:04.800 INFO:tasks.rados.rados.0.smithi155.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2024-04-19T01:04:04.800 INFO:tasks.rados.rados.0.smithi155.stdout:2: done (15 left) 2024-04-19T01:04:04.800 INFO:tasks.rados.rados.0.smithi155.stdout:11: done (14 left) 2024-04-19T01:04:04.800 INFO:tasks.rados.rados.0.smithi155.stdout:12: done (13 left) 2024-04-19T01:04:04.800 INFO:tasks.rados.rados.0.smithi155.stdout:21: write initial oid 21 2024-04-19T01:04:04.800 INFO:tasks.rados.rados.0.smithi155.stdout:21: seq_num 20 ranges {0=344064} 2024-04-19T01:04:04.802 INFO:tasks.rados.rados.0.smithi155.stdout:21: writing smithi15528762-21 from 0 to 344064 tid 1 2024-04-19T01:04:04.802 INFO:tasks.rados.rados.0.smithi155.stdout:22: write initial oid 22 2024-04-19T01:04:04.802 INFO:tasks.rados.rados.0.smithi155.stdout:22: seq_num 21 ranges {0=360448} 2024-04-19T01:04:04.805 INFO:tasks.rados.rados.0.smithi155.stdout:22: writing smithi15528762-22 from 0 to 360448 tid 1 2024-04-19T01:04:04.806 INFO:tasks.rados.rados.0.smithi155.stdout:23: write initial oid 23 2024-04-19T01:04:04.806 INFO:tasks.rados.rados.0.smithi155.stdout:23: seq_num 22 ranges {0=376832} 2024-04-19T01:04:04.810 INFO:tasks.rados.rados.0.smithi155.stdout:23: writing smithi15528762-23 from 0 to 376832 tid 1 2024-04-19T01:04:04.810 INFO:tasks.rados.rados.0.smithi155.stdout:13: finishing write tid 3 to smithi15528762-13 2024-04-19T01:04:04.810 INFO:tasks.rados.rados.0.smithi155.stdout:13: oid 13 updating version 0 to 3 2024-04-19T01:04:04.810 INFO:tasks.rados.rados.0.smithi155.stdout:13: oid 13 updating version 3 to 4 2024-04-19T01:04:04.810 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 13 v 4 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-04-19T01:04:04.811 INFO:tasks.rados.rados.0.smithi155.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2024-04-19T01:04:04.811 INFO:tasks.rados.rados.0.smithi155.stdout:8: finishing write tid 3 to smithi15528762-8 2024-04-19T01:04:04.811 INFO:tasks.rados.rados.0.smithi155.stdout:8: oid 8 updating version 0 to 1 2024-04-19T01:04:04.811 INFO:tasks.rados.rados.0.smithi155.stdout:8: oid 8 updating version 1 to 2 2024-04-19T01:04:04.811 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 8 v 2 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-04-19T01:04:04.811 INFO:tasks.rados.rados.0.smithi155.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2024-04-19T01:04:04.811 INFO:tasks.rados.rados.0.smithi155.stdout:14: finishing write tid 1 to smithi15528762-14 2024-04-19T01:04:04.811 INFO:tasks.rados.rados.0.smithi155.stdout:14: finishing write tid 2 to smithi15528762-14 2024-04-19T01:04:04.811 INFO:tasks.rados.rados.0.smithi155.stdout:7: finishing write tid 3 to smithi15528762-7 2024-04-19T01:04:04.811 INFO:tasks.rados.rados.0.smithi155.stdout:7: oid 7 updating version 0 to 2 2024-04-19T01:04:04.811 INFO:tasks.rados.rados.0.smithi155.stdout:7: oid 7 version 2 is already newer than 1 2024-04-19T01:04:04.811 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 7 v 2 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-04-19T01:04:04.811 INFO:tasks.rados.rados.0.smithi155.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2024-04-19T01:04:04.811 INFO:tasks.rados.rados.0.smithi155.stdout:3: finishing write tid 3 to smithi15528762-3 2024-04-19T01:04:04.812 INFO:tasks.rados.rados.0.smithi155.stdout:3: oid 3 updating version 0 to 1 2024-04-19T01:04:04.812 INFO:tasks.rados.rados.0.smithi155.stdout:3: oid 3 updating version 1 to 2 2024-04-19T01:04:04.812 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 3 v 2 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-04-19T01:04:04.812 INFO:tasks.rados.rados.0.smithi155.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2024-04-19T01:04:04.812 INFO:tasks.rados.rados.0.smithi155.stdout:4: finishing write tid 2 to smithi15528762-4 2024-04-19T01:04:04.812 INFO:tasks.rados.rados.0.smithi155.stdout:3: done (15 left) 2024-04-19T01:04:04.812 INFO:tasks.rados.rados.0.smithi155.stdout:7: done (14 left) 2024-04-19T01:04:04.812 INFO:tasks.rados.rados.0.smithi155.stdout:8: done (13 left) 2024-04-19T01:04:04.812 INFO:tasks.rados.rados.0.smithi155.stdout:13: done (12 left) 2024-04-19T01:04:04.812 INFO:tasks.rados.rados.0.smithi155.stdout:24: write initial oid 24 2024-04-19T01:04:04.812 INFO:tasks.rados.rados.0.smithi155.stdout:24: seq_num 23 ranges {0=393216} 2024-04-19T01:04:04.814 INFO:tasks.rados.rados.0.smithi155.stdout:24: writing smithi15528762-24 from 0 to 393216 tid 1 2024-04-19T01:04:04.814 INFO:tasks.rados.rados.0.smithi155.stdout:25: write initial oid 25 2024-04-19T01:04:04.814 INFO:tasks.rados.rados.0.smithi155.stdout:25: seq_num 24 ranges {0=409600} 2024-04-19T01:04:04.818 INFO:tasks.rados.rados.0.smithi155.stdout:25: writing smithi15528762-25 from 0 to 409600 tid 1 2024-04-19T01:04:04.818 INFO:tasks.rados.rados.0.smithi155.stdout:26: write initial oid 26 2024-04-19T01:04:04.818 INFO:tasks.rados.rados.0.smithi155.stdout:26: seq_num 25 ranges {0=425984} 2024-04-19T01:04:04.821 INFO:tasks.rados.rados.0.smithi155.stdout:26: writing smithi15528762-26 from 0 to 425984 tid 1 2024-04-19T01:04:04.821 INFO:tasks.rados.rados.0.smithi155.stdout:27: write initial oid 27 2024-04-19T01:04:04.821 INFO:tasks.rados.rados.0.smithi155.stdout:27: seq_num 26 ranges {0=442368} 2024-04-19T01:04:04.826 INFO:tasks.rados.rados.0.smithi155.stdout:27: writing smithi15528762-27 from 0 to 442368 tid 1 2024-04-19T01:04:04.826 INFO:tasks.rados.rados.0.smithi155.stdout: waiting on 16 2024-04-19T01:04:04.826 INFO:tasks.rados.rados.0.smithi155.stdout:14: finishing write tid 3 to smithi15528762-14 2024-04-19T01:04:04.826 INFO:tasks.rados.rados.0.smithi155.stdout:14: oid 14 updating version 0 to 4 2024-04-19T01:04:04.826 INFO:tasks.rados.rados.0.smithi155.stdout:14: oid 14 version 4 is already newer than 3 2024-04-19T01:04:04.826 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 14 v 4 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-04-19T01:04:04.826 INFO:tasks.rados.rados.0.smithi155.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2024-04-19T01:04:04.826 INFO:tasks.rados.rados.0.smithi155.stdout:15: finishing write tid 1 to smithi15528762-15 2024-04-19T01:04:04.826 INFO:tasks.rados.rados.0.smithi155.stdout:15: finishing write tid 2 to smithi15528762-15 2024-04-19T01:04:04.826 INFO:tasks.rados.rados.0.smithi155.stdout:4: finishing write tid 3 to smithi15528762-4 2024-04-19T01:04:04.827 INFO:tasks.rados.rados.0.smithi155.stdout:4: oid 4 updating version 0 to 1 2024-04-19T01:04:04.827 INFO:tasks.rados.rados.0.smithi155.stdout:4: oid 4 updating version 1 to 2 2024-04-19T01:04:04.827 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 4 v 2 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-19T01:04:04.827 INFO:tasks.rados.rados.0.smithi155.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2024-04-19T01:04:04.827 INFO:tasks.rados.rados.0.smithi155.stdout:9: finishing write tid 3 to smithi15528762-9 2024-04-19T01:04:04.827 INFO:tasks.rados.rados.0.smithi155.stdout:9: oid 9 updating version 0 to 1 2024-04-19T01:04:04.827 INFO:tasks.rados.rados.0.smithi155.stdout:9: oid 9 updating version 1 to 2 2024-04-19T01:04:04.827 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 9 v 2 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-04-19T01:04:04.827 INFO:tasks.rados.rados.0.smithi155.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2024-04-19T01:04:04.827 INFO:tasks.rados.rados.0.smithi155.stdout:19: finishing write tid 1 to smithi15528762-19 2024-04-19T01:04:04.827 INFO:tasks.rados.rados.0.smithi155.stdout:4: done (15 left) 2024-04-19T01:04:04.827 INFO:tasks.rados.rados.0.smithi155.stdout:9: done (14 left) 2024-04-19T01:04:04.827 INFO:tasks.rados.rados.0.smithi155.stdout:14: done (13 left) 2024-04-19T01:04:04.827 INFO:tasks.rados.rados.0.smithi155.stdout:28: write initial oid 28 2024-04-19T01:04:04.827 INFO:tasks.rados.rados.0.smithi155.stdout:28: seq_num 27 ranges {0=458752} 2024-04-19T01:04:04.829 INFO:tasks.rados.rados.0.smithi155.stdout:28: writing smithi15528762-28 from 0 to 458752 tid 1 2024-04-19T01:04:04.829 INFO:tasks.rados.rados.0.smithi155.stdout:29: write initial oid 29 2024-04-19T01:04:04.829 INFO:tasks.rados.rados.0.smithi155.stdout:29: seq_num 28 ranges {0=475136} 2024-04-19T01:04:04.834 INFO:tasks.rados.rados.0.smithi155.stdout:29: writing smithi15528762-29 from 0 to 475136 tid 1 2024-04-19T01:04:04.835 INFO:tasks.rados.rados.0.smithi155.stdout:19: finishing write tid 2 to smithi15528762-19 2024-04-19T01:04:04.835 INFO:tasks.rados.rados.0.smithi155.stdout:16: finishing write tid 1 to smithi15528762-16 2024-04-19T01:04:04.835 INFO:tasks.rados.rados.0.smithi155.stdout:23: finishing write tid 1 to smithi15528762-23 2024-04-19T01:04:04.835 INFO:tasks.rados.rados.0.smithi155.stdout:23: finishing write tid 2 to smithi15528762-23 2024-04-19T01:04:04.835 INFO:tasks.rados.rados.0.smithi155.stdout:19: finishing write tid 3 to smithi15528762-19 2024-04-19T01:04:04.835 INFO:tasks.rados.rados.0.smithi155.stdout:19: oid 19 updating version 0 to 1 2024-04-19T01:04:04.835 INFO:tasks.rados.rados.0.smithi155.stdout:19: oid 19 updating version 1 to 2 2024-04-19T01:04:04.835 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 19 v 2 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-04-19T01:04:04.835 INFO:tasks.rados.rados.0.smithi155.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2024-04-19T01:04:04.835 INFO:tasks.rados.rados.0.smithi155.stdout:19: done (14 left) 2024-04-19T01:04:04.836 INFO:tasks.rados.rados.0.smithi155.stdout:30: write initial oid 30 2024-04-19T01:04:04.836 INFO:tasks.rados.rados.0.smithi155.stdout:30: seq_num 29 ranges {0=491520} 2024-04-19T01:04:04.839 INFO:tasks.rados.rados.0.smithi155.stdout:30: writing smithi15528762-30 from 0 to 491520 tid 1 2024-04-19T01:04:04.839 INFO:tasks.rados.rados.0.smithi155.stdout:31: write initial oid 31 2024-04-19T01:04:04.839 INFO:tasks.rados.rados.0.smithi155.stdout:31: seq_num 30 ranges {0=507904} 2024-04-19T01:04:04.844 INFO:tasks.rados.rados.0.smithi155.stdout:31: writing smithi15528762-31 from 0 to 507904 tid 1 2024-04-19T01:04:04.846 INFO:tasks.rados.rados.0.smithi155.stdout: waiting on 16 2024-04-19T01:04:04.846 INFO:tasks.rados.rados.0.smithi155.stdout:17: finishing write tid 1 to smithi15528762-17 2024-04-19T01:04:04.846 INFO:tasks.rados.rados.0.smithi155.stdout:17: finishing write tid 2 to smithi15528762-17 2024-04-19T01:04:04.846 INFO:tasks.rados.rados.0.smithi155.stdout:24: finishing write tid 1 to smithi15528762-24 2024-04-19T01:04:04.846 INFO:tasks.rados.rados.0.smithi155.stdout:23: finishing write tid 3 to smithi15528762-23 2024-04-19T01:04:04.846 INFO:tasks.rados.rados.0.smithi155.stdout:23: oid 23 updating version 0 to 1 2024-04-19T01:04:04.846 INFO:tasks.rados.rados.0.smithi155.stdout:23: oid 23 updating version 1 to 2 2024-04-19T01:04:04.846 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 23 v 2 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-04-19T01:04:04.846 INFO:tasks.rados.rados.0.smithi155.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2024-04-19T01:04:04.846 INFO:tasks.rados.rados.0.smithi155.stdout:18: finishing write tid 1 to smithi15528762-18 2024-04-19T01:04:04.846 INFO:tasks.rados.rados.0.smithi155.stdout:18: finishing write tid 2 to smithi15528762-18 2024-04-19T01:04:04.846 INFO:tasks.rados.rados.0.smithi155.stdout:22: finishing write tid 1 to smithi15528762-22 2024-04-19T01:04:04.846 INFO:tasks.rados.rados.0.smithi155.stdout:20: finishing write tid 1 to smithi15528762-20 2024-04-19T01:04:04.847 INFO:tasks.rados.rados.0.smithi155.stdout:16: finishing write tid 2 to smithi15528762-16 2024-04-19T01:04:04.847 INFO:tasks.rados.rados.0.smithi155.stdout:20: finishing write tid 2 to smithi15528762-20 2024-04-19T01:04:04.847 INFO:tasks.rados.rados.0.smithi155.stdout:21: finishing write tid 1 to smithi15528762-21 2024-04-19T01:04:04.847 INFO:tasks.rados.rados.0.smithi155.stdout:17: finishing write tid 3 to smithi15528762-17 2024-04-19T01:04:04.847 INFO:tasks.rados.rados.0.smithi155.stdout:17: oid 17 updating version 0 to 2 2024-04-19T01:04:04.847 INFO:tasks.rados.rados.0.smithi155.stdout:17: oid 17 version 2 is already newer than 1 2024-04-19T01:04:04.847 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 17 v 2 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-04-19T01:04:04.847 INFO:tasks.rados.rados.0.smithi155.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2024-04-19T01:04:04.847 INFO:tasks.rados.rados.0.smithi155.stdout:18: finishing write tid 3 to smithi15528762-18 2024-04-19T01:04:04.847 INFO:tasks.rados.rados.0.smithi155.stdout:18: oid 18 updating version 0 to 4 2024-04-19T01:04:04.847 INFO:tasks.rados.rados.0.smithi155.stdout:18: oid 18 version 4 is already newer than 3 2024-04-19T01:04:04.847 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 18 v 4 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-04-19T01:04:04.847 INFO:tasks.rados.rados.0.smithi155.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2024-04-19T01:04:04.847 INFO:tasks.rados.rados.0.smithi155.stdout:17: done (15 left) 2024-04-19T01:04:04.848 INFO:tasks.rados.rados.0.smithi155.stdout:18: done (14 left) 2024-04-19T01:04:04.848 INFO:tasks.rados.rados.0.smithi155.stdout:23: done (13 left) 2024-04-19T01:04:04.848 INFO:tasks.rados.rados.0.smithi155.stdout:32: write initial oid 32 2024-04-19T01:04:04.848 INFO:tasks.rados.rados.0.smithi155.stdout:32: seq_num 31 ranges {0=524288} 2024-04-19T01:04:04.849 INFO:tasks.rados.rados.0.smithi155.stdout:32: writing smithi15528762-32 from 0 to 524288 tid 1 2024-04-19T01:04:04.849 INFO:tasks.rados.rados.0.smithi155.stdout:33: write initial oid 33 2024-04-19T01:04:04.849 INFO:tasks.rados.rados.0.smithi155.stdout:33: seq_num 32 ranges {0=540672} 2024-04-19T01:04:04.852 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:04:04.852+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:04:04.853 INFO:tasks.rados.rados.0.smithi155.stdout:33: writing smithi15528762-33 from 0 to 540672 tid 1 2024-04-19T01:04:04.853 INFO:tasks.rados.rados.0.smithi155.stdout:34: write initial oid 34 2024-04-19T01:04:04.853 INFO:tasks.rados.rados.0.smithi155.stdout:34: seq_num 33 ranges {0=557056} 2024-04-19T01:04:04.859 INFO:tasks.rados.rados.0.smithi155.stdout:34: writing smithi15528762-34 from 0 to 557056 tid 1 2024-04-19T01:04:04.859 INFO:tasks.rados.rados.0.smithi155.stdout:25: finishing write tid 1 to smithi15528762-25 2024-04-19T01:04:04.859 INFO:tasks.rados.rados.0.smithi155.stdout:15: finishing write tid 3 to smithi15528762-15 2024-04-19T01:04:04.860 INFO:tasks.rados.rados.0.smithi155.stdout:15: oid 15 updating version 0 to 3 2024-04-19T01:04:04.860 INFO:tasks.rados.rados.0.smithi155.stdout:15: oid 15 updating version 3 to 4 2024-04-19T01:04:04.860 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 15 v 4 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-04-19T01:04:04.860 INFO:tasks.rados.rados.0.smithi155.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2024-04-19T01:04:04.860 INFO:tasks.rados.rados.0.smithi155.stdout:25: finishing write tid 2 to smithi15528762-25 2024-04-19T01:04:04.860 INFO:tasks.rados.rados.0.smithi155.stdout:21: finishing write tid 2 to smithi15528762-21 2024-04-19T01:04:04.860 INFO:tasks.rados.rados.0.smithi155.stdout:22: finishing write tid 2 to smithi15528762-22 2024-04-19T01:04:04.860 INFO:tasks.rados.rados.0.smithi155.stdout:25: finishing write tid 3 to smithi15528762-25 2024-04-19T01:04:04.860 INFO:tasks.rados.rados.0.smithi155.stdout:25: oid 25 updating version 0 to 4 2024-04-19T01:04:04.860 INFO:tasks.rados.rados.0.smithi155.stdout:25: oid 25 version 4 is already newer than 3 2024-04-19T01:04:04.860 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 25 v 4 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-04-19T01:04:04.860 INFO:tasks.rados.rados.0.smithi155.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2024-04-19T01:04:04.860 INFO:tasks.rados.rados.0.smithi155.stdout:26: finishing write tid 1 to smithi15528762-26 2024-04-19T01:04:04.860 INFO:tasks.rados.rados.0.smithi155.stdout:24: finishing write tid 2 to smithi15528762-24 2024-04-19T01:04:04.861 INFO:tasks.rados.rados.0.smithi155.stdout:27: finishing write tid 1 to smithi15528762-27 2024-04-19T01:04:04.861 INFO:tasks.rados.rados.0.smithi155.stdout:27: finishing write tid 2 to smithi15528762-27 2024-04-19T01:04:04.861 INFO:tasks.rados.rados.0.smithi155.stdout:22: finishing write tid 3 to smithi15528762-22 2024-04-19T01:04:04.861 INFO:tasks.rados.rados.0.smithi155.stdout:22: oid 22 updating version 0 to 6 2024-04-19T01:04:04.861 INFO:tasks.rados.rados.0.smithi155.stdout:22: oid 22 version 6 is already newer than 5 2024-04-19T01:04:04.861 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 22 v 6 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-04-19T01:04:04.861 INFO:tasks.rados.rados.0.smithi155.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2024-04-19T01:04:04.861 INFO:tasks.rados.rados.0.smithi155.stdout:26: finishing write tid 2 to smithi15528762-26 2024-04-19T01:04:04.861 INFO:tasks.rados.rados.0.smithi155.stdout:16: finishing write tid 3 to smithi15528762-16 2024-04-19T01:04:04.861 INFO:tasks.rados.rados.0.smithi155.stdout:16: oid 16 updating version 0 to 5 2024-04-19T01:04:04.861 INFO:tasks.rados.rados.0.smithi155.stdout:16: oid 16 updating version 5 to 6 2024-04-19T01:04:04.861 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 16 v 6 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-04-19T01:04:04.861 INFO:tasks.rados.rados.0.smithi155.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2024-04-19T01:04:04.861 INFO:tasks.rados.rados.0.smithi155.stdout:28: finishing write tid 1 to smithi15528762-28 2024-04-19T01:04:04.862 INFO:tasks.rados.rados.0.smithi155.stdout:28: finishing write tid 2 to smithi15528762-28 2024-04-19T01:04:04.862 INFO:tasks.rados.rados.0.smithi155.stdout:29: finishing write tid 1 to smithi15528762-29 2024-04-19T01:04:04.862 INFO:tasks.rados.rados.0.smithi155.stdout:29: finishing write tid 2 to smithi15528762-29 2024-04-19T01:04:04.862 INFO:tasks.rados.rados.0.smithi155.stdout:27: finishing write tid 3 to smithi15528762-27 2024-04-19T01:04:04.862 INFO:tasks.rados.rados.0.smithi155.stdout:27: oid 27 updating version 0 to 3 2024-04-19T01:04:04.862 INFO:tasks.rados.rados.0.smithi155.stdout:27: oid 27 updating version 3 to 4 2024-04-19T01:04:04.862 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 27 v 4 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-04-19T01:04:04.862 INFO:tasks.rados.rados.0.smithi155.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2024-04-19T01:04:04.862 INFO:tasks.rados.rados.0.smithi155.stdout:15: done (15 left) 2024-04-19T01:04:04.862 INFO:tasks.rados.rados.0.smithi155.stdout:16: done (14 left) 2024-04-19T01:04:04.862 INFO:tasks.rados.rados.0.smithi155.stdout:22: done (13 left) 2024-04-19T01:04:04.862 INFO:tasks.rados.rados.0.smithi155.stdout:25: done (12 left) 2024-04-19T01:04:04.862 INFO:tasks.rados.rados.0.smithi155.stdout:27: done (11 left) 2024-04-19T01:04:04.862 INFO:tasks.rados.rados.0.smithi155.stdout:35: write initial oid 35 2024-04-19T01:04:04.863 INFO:tasks.rados.rados.0.smithi155.stdout:35: seq_num 34 ranges {0=573440} 2024-04-19T01:04:04.863 INFO:tasks.rados.rados.0.smithi155.stdout:35: writing smithi15528762-35 from 0 to 573440 tid 1 2024-04-19T01:04:04.863 INFO:tasks.rados.rados.0.smithi155.stdout:30: finishing write tid 1 to smithi15528762-30 2024-04-19T01:04:04.863 INFO:tasks.rados.rados.0.smithi155.stdout:31: finishing write tid 1 to smithi15528762-31 2024-04-19T01:04:04.863 INFO:tasks.rados.rados.0.smithi155.stdout:31: finishing write tid 2 to smithi15528762-31 2024-04-19T01:04:04.864 INFO:tasks.rados.rados.0.smithi155.stdout:36: write initial oid 36 2024-04-19T01:04:04.864 INFO:tasks.rados.rados.0.smithi155.stdout:36: seq_num 35 ranges {0=589824} 2024-04-19T01:04:04.865 INFO:tasks.rados.rados.0.smithi155.stdout:36: writing smithi15528762-36 from 0 to 589824 tid 1 2024-04-19T01:04:04.865 INFO:tasks.rados.rados.0.smithi155.stdout:37: write initial oid 37 2024-04-19T01:04:04.865 INFO:tasks.rados.rados.0.smithi155.stdout:37: seq_num 36 ranges {0=606208} 2024-04-19T01:04:04.870 INFO:tasks.rados.rados.0.smithi155.stdout:37: writing smithi15528762-37 from 0 to 606208 tid 1 2024-04-19T01:04:04.870 INFO:tasks.rados.rados.0.smithi155.stdout:38: write initial oid 38 2024-04-19T01:04:04.870 INFO:tasks.rados.rados.0.smithi155.stdout:38: seq_num 37 ranges {0=622592} 2024-04-19T01:04:04.874 INFO:tasks.rados.rados.0.smithi155.stdout:38: writing smithi15528762-38 from 0 to 622592 tid 1 2024-04-19T01:04:04.874 INFO:tasks.rados.rados.0.smithi155.stdout:39: write initial oid 39 2024-04-19T01:04:04.874 INFO:tasks.rados.rados.0.smithi155.stdout:39: seq_num 38 ranges {0=638976} 2024-04-19T01:04:04.878 INFO:tasks.rados.rados.0.smithi155.stdout:39: writing smithi15528762-39 from 0 to 638976 tid 1 2024-04-19T01:04:04.878 INFO:tasks.rados.rados.0.smithi155.stdout: waiting on 16 2024-04-19T01:04:04.879 INFO:tasks.rados.rados.0.smithi155.stdout:24: finishing write tid 3 to smithi15528762-24 2024-04-19T01:04:04.879 INFO:tasks.rados.rados.0.smithi155.stdout:24: oid 24 updating version 0 to 4 2024-04-19T01:04:04.879 INFO:tasks.rados.rados.0.smithi155.stdout:24: oid 24 version 4 is already newer than 3 2024-04-19T01:04:04.879 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 24 v 4 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-04-19T01:04:04.879 INFO:tasks.rados.rados.0.smithi155.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2024-04-19T01:04:04.879 INFO:tasks.rados.rados.0.smithi155.stdout:24: done (15 left) 2024-04-19T01:04:04.879 INFO:tasks.rados.rados.0.smithi155.stdout:40: write initial oid 40 2024-04-19T01:04:04.879 INFO:tasks.rados.rados.0.smithi155.stdout:40: seq_num 39 ranges {0=671744} 2024-04-19T01:04:04.883 INFO:tasks.rados.rados.0.smithi155.stdout:40: writing smithi15528762-40 from 0 to 671744 tid 1 2024-04-19T01:04:04.883 INFO:tasks.rados.rados.0.smithi155.stdout: waiting on 16 2024-04-19T01:04:04.883 INFO:tasks.rados.rados.0.smithi155.stdout:20: finishing write tid 3 to smithi15528762-20 2024-04-19T01:04:04.883 INFO:tasks.rados.rados.0.smithi155.stdout:20: oid 20 updating version 0 to 1 2024-04-19T01:04:04.883 INFO:tasks.rados.rados.0.smithi155.stdout:20: oid 20 updating version 1 to 2 2024-04-19T01:04:04.883 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 20 v 2 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-04-19T01:04:04.883 INFO:tasks.rados.rados.0.smithi155.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2024-04-19T01:04:04.883 INFO:tasks.rados.rados.0.smithi155.stdout:29: finishing write tid 3 to smithi15528762-29 2024-04-19T01:04:04.883 INFO:tasks.rados.rados.0.smithi155.stdout:29: oid 29 updating version 0 to 5 2024-04-19T01:04:04.883 INFO:tasks.rados.rados.0.smithi155.stdout:29: oid 29 updating version 5 to 6 2024-04-19T01:04:04.883 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 29 v 6 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-19T01:04:04.883 INFO:tasks.rados.rados.0.smithi155.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2024-04-19T01:04:04.883 INFO:tasks.rados.rados.0.smithi155.stdout:31: finishing write tid 3 to smithi15528762-31 2024-04-19T01:04:04.884 INFO:tasks.rados.rados.0.smithi155.stdout:31: oid 31 updating version 0 to 3 2024-04-19T01:04:04.884 INFO:tasks.rados.rados.0.smithi155.stdout:31: oid 31 updating version 3 to 4 2024-04-19T01:04:04.884 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 31 v 4 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-04-19T01:04:04.884 INFO:tasks.rados.rados.0.smithi155.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2024-04-19T01:04:04.884 INFO:tasks.rados.rados.0.smithi155.stdout:28: finishing write tid 3 to smithi15528762-28 2024-04-19T01:04:04.884 INFO:tasks.rados.rados.0.smithi155.stdout:28: oid 28 updating version 0 to 3 2024-04-19T01:04:04.884 INFO:tasks.rados.rados.0.smithi155.stdout:28: oid 28 updating version 3 to 4 2024-04-19T01:04:04.884 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 28 v 4 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-04-19T01:04:04.884 INFO:tasks.rados.rados.0.smithi155.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2024-04-19T01:04:04.884 INFO:tasks.rados.rados.0.smithi155.stdout:20: done (15 left) 2024-04-19T01:04:04.884 INFO:tasks.rados.rados.0.smithi155.stdout:28: done (14 left) 2024-04-19T01:04:04.884 INFO:tasks.rados.rados.0.smithi155.stdout:29: done (13 left) 2024-04-19T01:04:04.884 INFO:tasks.rados.rados.0.smithi155.stdout:31: done (12 left) 2024-04-19T01:04:04.884 INFO:tasks.rados.rados.0.smithi155.stdout:41: write initial oid 41 2024-04-19T01:04:04.885 INFO:tasks.rados.rados.0.smithi155.stdout:41: seq_num 40 ranges {0=688128} 2024-04-19T01:04:04.887 INFO:tasks.rados.rados.0.smithi155.stdout:41: writing smithi15528762-41 from 0 to 688128 tid 1 2024-04-19T01:04:04.887 INFO:tasks.rados.rados.0.smithi155.stdout:42: write initial oid 42 2024-04-19T01:04:04.887 INFO:tasks.rados.rados.0.smithi155.stdout:42: seq_num 41 ranges {0=704512} 2024-04-19T01:04:04.891 INFO:tasks.rados.rados.0.smithi155.stdout:42: writing smithi15528762-42 from 0 to 704512 tid 1 2024-04-19T01:04:04.891 INFO:tasks.rados.rados.0.smithi155.stdout:43: write initial oid 43 2024-04-19T01:04:04.891 INFO:tasks.rados.rados.0.smithi155.stdout:43: seq_num 42 ranges {0=720896} 2024-04-19T01:04:04.896 INFO:tasks.rados.rados.0.smithi155.stdout:43: writing smithi15528762-43 from 0 to 720896 tid 1 2024-04-19T01:04:04.896 INFO:tasks.rados.rados.0.smithi155.stdout:44: write initial oid 44 2024-04-19T01:04:04.896 INFO:tasks.rados.rados.0.smithi155.stdout:44: seq_num 43 ranges {0=737280} 2024-04-19T01:04:04.903 INFO:tasks.rados.rados.0.smithi155.stdout:44: writing smithi15528762-44 from 0 to 737280 tid 1 2024-04-19T01:04:04.903 INFO:tasks.rados.rados.0.smithi155.stdout: waiting on 16 2024-04-19T01:04:04.903 INFO:tasks.rados.rados.0.smithi155.stdout:21: finishing write tid 3 to smithi15528762-21 2024-04-19T01:04:04.903 INFO:tasks.rados.rados.0.smithi155.stdout:21: oid 21 updating version 0 to 8 2024-04-19T01:04:04.903 INFO:tasks.rados.rados.0.smithi155.stdout:21: oid 21 version 8 is already newer than 7 2024-04-19T01:04:04.903 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 21 v 8 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-04-19T01:04:04.903 INFO:tasks.rados.rados.0.smithi155.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2024-04-19T01:04:04.903 INFO:tasks.rados.rados.0.smithi155.stdout:30: finishing write tid 2 to smithi15528762-30 2024-04-19T01:04:04.903 INFO:tasks.rados.rados.0.smithi155.stdout:33: finishing write tid 1 to smithi15528762-33 2024-04-19T01:04:04.903 INFO:tasks.rados.rados.0.smithi155.stdout:26: finishing write tid 3 to smithi15528762-26 2024-04-19T01:04:04.904 INFO:tasks.rados.rados.0.smithi155.stdout:26: oid 26 updating version 0 to 4 2024-04-19T01:04:04.904 INFO:tasks.rados.rados.0.smithi155.stdout:26: oid 26 version 4 is already newer than 3 2024-04-19T01:04:04.904 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 26 v 4 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-04-19T01:04:04.904 INFO:tasks.rados.rados.0.smithi155.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2024-04-19T01:04:04.904 INFO:tasks.rados.rados.0.smithi155.stdout:33: finishing write tid 2 to smithi15528762-33 2024-04-19T01:04:04.904 INFO:tasks.rados.rados.0.smithi155.stdout:30: finishing write tid 3 to smithi15528762-30 2024-04-19T01:04:04.904 INFO:tasks.rados.rados.0.smithi155.stdout:30: oid 30 updating version 0 to 6 2024-04-19T01:04:04.904 INFO:tasks.rados.rados.0.smithi155.stdout:30: oid 30 version 6 is already newer than 5 2024-04-19T01:04:04.904 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 30 v 6 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-04-19T01:04:04.904 INFO:tasks.rados.rados.0.smithi155.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2024-04-19T01:04:04.904 INFO:tasks.rados.rados.0.smithi155.stdout:33: finishing write tid 3 to smithi15528762-33 2024-04-19T01:04:04.904 INFO:tasks.rados.rados.0.smithi155.stdout:33: oid 33 updating version 0 to 3 2024-04-19T01:04:04.904 INFO:tasks.rados.rados.0.smithi155.stdout:33: oid 33 updating version 3 to 4 2024-04-19T01:04:04.904 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 33 v 4 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-04-19T01:04:04.904 INFO:tasks.rados.rados.0.smithi155.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2024-04-19T01:04:04.905 INFO:tasks.rados.rados.0.smithi155.stdout:32: finishing write tid 1 to smithi15528762-32 2024-04-19T01:04:04.905 INFO:tasks.rados.rados.0.smithi155.stdout:32: finishing write tid 2 to smithi15528762-32 2024-04-19T01:04:04.905 INFO:tasks.rados.rados.0.smithi155.stdout:35: finishing write tid 1 to smithi15528762-35 2024-04-19T01:04:04.905 INFO:tasks.rados.rados.0.smithi155.stdout:35: finishing write tid 2 to smithi15528762-35 2024-04-19T01:04:04.905 INFO:tasks.rados.rados.0.smithi155.stdout:32: finishing write tid 3 to smithi15528762-32 2024-04-19T01:04:04.905 INFO:tasks.rados.rados.0.smithi155.stdout:32: oid 32 updating version 0 to 5 2024-04-19T01:04:04.905 INFO:tasks.rados.rados.0.smithi155.stdout:32: oid 32 updating version 5 to 6 2024-04-19T01:04:04.905 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 32 v 6 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-04-19T01:04:04.905 INFO:tasks.rados.rados.0.smithi155.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2024-04-19T01:04:04.905 INFO:tasks.rados.rados.0.smithi155.stdout:35: finishing write tid 3 to smithi15528762-35 2024-04-19T01:04:04.905 INFO:tasks.rados.rados.0.smithi155.stdout:35: oid 35 updating version 0 to 1 2024-04-19T01:04:04.905 INFO:tasks.rados.rados.0.smithi155.stdout:35: oid 35 updating version 1 to 2 2024-04-19T01:04:04.905 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 35 v 2 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-04-19T01:04:04.905 INFO:tasks.rados.rados.0.smithi155.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2024-04-19T01:04:04.906 INFO:tasks.rados.rados.0.smithi155.stdout:36: finishing write tid 1 to smithi15528762-36 2024-04-19T01:04:04.906 INFO:tasks.rados.rados.0.smithi155.stdout:34: finishing write tid 1 to smithi15528762-34 2024-04-19T01:04:04.906 INFO:tasks.rados.rados.0.smithi155.stdout:34: finishing write tid 2 to smithi15528762-34 2024-04-19T01:04:04.906 INFO:tasks.rados.rados.0.smithi155.stdout:36: finishing write tid 2 to smithi15528762-36 2024-04-19T01:04:04.906 INFO:tasks.rados.rados.0.smithi155.stdout:34: finishing write tid 3 to smithi15528762-34 2024-04-19T01:04:04.906 INFO:tasks.rados.rados.0.smithi155.stdout:34: oid 34 updating version 0 to 8 2024-04-19T01:04:04.906 INFO:tasks.rados.rados.0.smithi155.stdout:34: oid 34 version 8 is already newer than 7 2024-04-19T01:04:04.906 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 34 v 8 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-04-19T01:04:04.906 INFO:tasks.rados.rados.0.smithi155.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2024-04-19T01:04:04.906 INFO:tasks.rados.rados.0.smithi155.stdout:36: finishing write tid 3 to smithi15528762-36 2024-04-19T01:04:04.906 INFO:tasks.rados.rados.0.smithi155.stdout:36: oid 36 updating version 0 to 6 2024-04-19T01:04:04.906 INFO:tasks.rados.rados.0.smithi155.stdout:36: oid 36 version 6 is already newer than 5 2024-04-19T01:04:04.906 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 36 v 6 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-04-19T01:04:04.906 INFO:tasks.rados.rados.0.smithi155.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2024-04-19T01:04:04.906 INFO:tasks.rados.rados.0.smithi155.stdout:40: finishing write tid 1 to smithi15528762-40 2024-04-19T01:04:04.907 INFO:tasks.rados.rados.0.smithi155.stdout:37: finishing write tid 1 to smithi15528762-37 2024-04-19T01:04:04.907 INFO:tasks.rados.rados.0.smithi155.stdout:37: finishing write tid 2 to smithi15528762-37 2024-04-19T01:04:04.907 INFO:tasks.rados.rados.0.smithi155.stdout:38: finishing write tid 1 to smithi15528762-38 2024-04-19T01:04:04.907 INFO:tasks.rados.rados.0.smithi155.stdout:38: finishing write tid 2 to smithi15528762-38 2024-04-19T01:04:04.907 INFO:tasks.rados.rados.0.smithi155.stdout:40: finishing write tid 2 to smithi15528762-40 2024-04-19T01:04:04.907 INFO:tasks.rados.rados.0.smithi155.stdout:40: finishing write tid 3 to smithi15528762-40 2024-04-19T01:04:04.907 INFO:tasks.rados.rados.0.smithi155.stdout:40: oid 40 updating version 0 to 5 2024-04-19T01:04:04.907 INFO:tasks.rados.rados.0.smithi155.stdout:40: oid 40 updating version 5 to 6 2024-04-19T01:04:04.907 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 40 v 6 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-19T01:04:04.907 INFO:tasks.rados.rados.0.smithi155.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2024-04-19T01:04:04.907 INFO:tasks.rados.rados.0.smithi155.stdout:21: done (15 left) 2024-04-19T01:04:04.907 INFO:tasks.rados.rados.0.smithi155.stdout:26: done (14 left) 2024-04-19T01:04:04.907 INFO:tasks.rados.rados.0.smithi155.stdout:30: done (13 left) 2024-04-19T01:04:04.907 INFO:tasks.rados.rados.0.smithi155.stdout:32: done (12 left) 2024-04-19T01:04:04.908 INFO:tasks.rados.rados.0.smithi155.stdout:33: done (11 left) 2024-04-19T01:04:04.908 INFO:tasks.rados.rados.0.smithi155.stdout:34: done (10 left) 2024-04-19T01:04:04.908 INFO:tasks.rados.rados.0.smithi155.stdout:35: done (9 left) 2024-04-19T01:04:04.908 INFO:tasks.rados.rados.0.smithi155.stdout:36: done (8 left) 2024-04-19T01:04:04.908 INFO:tasks.rados.rados.0.smithi155.stdout:40: done (7 left) 2024-04-19T01:04:04.908 INFO:tasks.rados.rados.0.smithi155.stdout:45: write initial oid 45 2024-04-19T01:04:04.908 INFO:tasks.rados.rados.0.smithi155.stdout:45: seq_num 44 ranges {0=753664} 2024-04-19T01:04:04.908 INFO:tasks.rados.rados.0.smithi155.stdout:45: writing smithi15528762-45 from 0 to 753664 tid 1 2024-04-19T01:04:04.909 INFO:tasks.rados.rados.0.smithi155.stdout:46: write initial oid 46 2024-04-19T01:04:04.909 INFO:tasks.rados.rados.0.smithi155.stdout:46: seq_num 45 ranges {0=770048} 2024-04-19T01:04:04.909 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:04.910 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:04.912 INFO:tasks.rados.rados.0.smithi155.stdout:46: writing smithi15528762-46 from 0 to 770048 tid 1 2024-04-19T01:04:04.912 INFO:tasks.rados.rados.0.smithi155.stdout:47: write initial oid 47 2024-04-19T01:04:04.912 INFO:tasks.rados.rados.0.smithi155.stdout:47: seq_num 46 ranges {0=786432} 2024-04-19T01:04:04.918 INFO:tasks.rados.rados.0.smithi155.stdout:47: writing smithi15528762-47 from 0 to 786432 tid 1 2024-04-19T01:04:04.918 INFO:tasks.rados.rados.0.smithi155.stdout:48: write initial oid 48 2024-04-19T01:04:04.918 INFO:tasks.rados.rados.0.smithi155.stdout:48: seq_num 47 ranges {0=425984,425984=376832} 2024-04-19T01:04:04.921 INFO:tasks.rados.rados.0.smithi155.stdout:48: writing smithi15528762-48 from 0 to 425984 tid 1 2024-04-19T01:04:04.924 INFO:tasks.rados.rados.0.smithi155.stdout:48: writing smithi15528762-48 from 425984 to 802816 tid 2 2024-04-19T01:04:04.924 INFO:tasks.rados.rados.0.smithi155.stdout:49: write initial oid 49 2024-04-19T01:04:04.924 INFO:tasks.rados.rados.0.smithi155.stdout:49: seq_num 48 ranges {0=442368,442368=376832} 2024-04-19T01:04:04.927 INFO:tasks.rados.rados.0.smithi155.stdout:49: writing smithi15528762-49 from 0 to 442368 tid 1 2024-04-19T01:04:04.929 INFO:tasks.rados.rados.0.smithi155.stdout:49: writing smithi15528762-49 from 442368 to 819200 tid 2 2024-04-19T01:04:04.929 INFO:tasks.rados.rados.0.smithi155.stdout:50: write initial oid 50 2024-04-19T01:04:04.929 INFO:tasks.rados.rados.0.smithi155.stdout:50: seq_num 49 ranges {0=458752,458752=376832} 2024-04-19T01:04:04.933 INFO:tasks.rados.rados.0.smithi155.stdout:50: writing smithi15528762-50 from 0 to 458752 tid 1 2024-04-19T01:04:04.936 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-19T01:04:04.937 INFO:tasks.rados.rados.0.smithi155.stdout:50: writing smithi15528762-50 from 458752 to 835584 tid 2 2024-04-19T01:04:04.938 INFO:tasks.rados.rados.0.smithi155.stdout:51: delete oid 21 current snap is 0 2024-04-19T01:04:04.938 INFO:tasks.rados.rados.0.smithi155.stdout:37: finishing write tid 3 to smithi15528762-37 2024-04-19T01:04:04.938 INFO:tasks.rados.rados.0.smithi155.stdout:37: oid 37 updating version 0 to 7 2024-04-19T01:04:04.938 INFO:tasks.rados.rados.0.smithi155.stdout:37: oid 37 updating version 7 to 8 2024-04-19T01:04:04.938 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 37 v 8 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-04-19T01:04:04.938 INFO:tasks.rados.rados.0.smithi155.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2024-04-19T01:04:04.938 INFO:tasks.rados.rados.0.smithi155.stdout:39: finishing write tid 1 to smithi15528762-39 2024-04-19T01:04:04.938 INFO:tasks.rados.rados.0.smithi155.stdout:39: finishing write tid 2 to smithi15528762-39 2024-04-19T01:04:04.938 INFO:tasks.rados.rados.0.smithi155.stdout:38: finishing write tid 3 to smithi15528762-38 2024-04-19T01:04:04.938 INFO:tasks.rados.rados.0.smithi155.stdout:38: oid 38 updating version 0 to 10 2024-04-19T01:04:04.938 INFO:tasks.rados.rados.0.smithi155.stdout:38: oid 38 version 10 is already newer than 9 2024-04-19T01:04:04.938 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 38 v 10 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-04-19T01:04:04.939 INFO:tasks.rados.rados.0.smithi155.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2024-04-19T01:04:04.939 INFO:tasks.rados.rados.0.smithi155.stdout:41: finishing write tid 1 to smithi15528762-41 2024-04-19T01:04:04.939 INFO:tasks.rados.rados.0.smithi155.stdout:41: finishing write tid 2 to smithi15528762-41 2024-04-19T01:04:04.939 INFO:tasks.rados.rados.0.smithi155.stdout:39: finishing write tid 3 to smithi15528762-39 2024-04-19T01:04:04.939 INFO:tasks.rados.rados.0.smithi155.stdout:39: oid 39 updating version 0 to 11 2024-04-19T01:04:04.939 INFO:tasks.rados.rados.0.smithi155.stdout:39: oid 39 updating version 11 to 12 2024-04-19T01:04:04.939 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 39 v 12 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-04-19T01:04:04.939 INFO:tasks.rados.rados.0.smithi155.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2024-04-19T01:04:04.939 INFO:tasks.rados.rados.0.smithi155.stdout:43: finishing write tid 1 to smithi15528762-43 2024-04-19T01:04:04.939 INFO:tasks.rados.rados.0.smithi155.stdout:41: finishing write tid 3 to smithi15528762-41 2024-04-19T01:04:04.939 INFO:tasks.rados.rados.0.smithi155.stdout:41: oid 41 updating version 0 to 3 2024-04-19T01:04:04.940 INFO:tasks.rados.rados.0.smithi155.stdout:41: oid 41 updating version 3 to 4 2024-04-19T01:04:04.940 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 41 v 4 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-04-19T01:04:04.940 INFO:tasks.rados.rados.0.smithi155.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2024-04-19T01:04:04.940 INFO:tasks.rados.rados.0.smithi155.stdout:43: finishing write tid 2 to smithi15528762-43 2024-04-19T01:04:04.940 INFO:tasks.rados.rados.0.smithi155.stdout:45: finishing write tid 1 to smithi15528762-45 2024-04-19T01:04:04.940 INFO:tasks.rados.rados.0.smithi155.stdout:45: finishing write tid 2 to smithi15528762-45 2024-04-19T01:04:04.940 INFO:tasks.rados.rados.0.smithi155.stdout:44: finishing write tid 1 to smithi15528762-44 2024-04-19T01:04:04.940 INFO:tasks.rados.rados.0.smithi155.stdout:43: finishing write tid 3 to smithi15528762-43 2024-04-19T01:04:04.940 INFO:tasks.rados.rados.0.smithi155.stdout:43: oid 43 updating version 0 to 10 2024-04-19T01:04:04.940 INFO:tasks.rados.rados.0.smithi155.stdout:43: oid 43 version 10 is already newer than 9 2024-04-19T01:04:04.940 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 43 v 10 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-04-19T01:04:04.940 INFO:tasks.rados.rados.0.smithi155.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2024-04-19T01:04:04.941 INFO:tasks.rados.rados.0.smithi155.stdout:44: finishing write tid 2 to smithi15528762-44 2024-04-19T01:04:04.941 INFO:tasks.rados.rados.0.smithi155.stdout:44: finishing write tid 3 to smithi15528762-44 2024-04-19T01:04:04.941 INFO:tasks.rados.rados.0.smithi155.stdout:44: oid 44 updating version 0 to 11 2024-04-19T01:04:04.941 INFO:tasks.rados.rados.0.smithi155.stdout:44: oid 44 updating version 11 to 12 2024-04-19T01:04:04.941 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 44 v 12 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-04-19T01:04:04.941 INFO:tasks.rados.rados.0.smithi155.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2024-04-19T01:04:04.941 INFO:tasks.rados.rados.0.smithi155.stdout:45: finishing write tid 3 to smithi15528762-45 2024-04-19T01:04:04.941 INFO:tasks.rados.rados.0.smithi155.stdout:45: oid 45 updating version 0 to 5 2024-04-19T01:04:04.941 INFO:tasks.rados.rados.0.smithi155.stdout:45: oid 45 updating version 5 to 6 2024-04-19T01:04:04.941 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 45 v 6 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-04-19T01:04:04.941 INFO:tasks.rados.rados.0.smithi155.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2024-04-19T01:04:04.941 INFO:tasks.rados.rados.0.smithi155.stdout:42: finishing write tid 1 to smithi15528762-42 2024-04-19T01:04:04.941 INFO:tasks.rados.rados.0.smithi155.stdout:42: finishing write tid 2 to smithi15528762-42 2024-04-19T01:04:04.941 INFO:tasks.rados.rados.0.smithi155.stdout:48: finishing write tid 1 to smithi15528762-48 2024-04-19T01:04:04.943 INFO:tasks.rados.rados.0.smithi155.stdout:46: finishing write tid 1 to smithi15528762-46 2024-04-19T01:04:04.943 INFO:tasks.rados.rados.0.smithi155.stdout:42: finishing write tid 3 to smithi15528762-42 2024-04-19T01:04:04.943 INFO:tasks.rados.rados.0.smithi155.stdout:42: oid 42 updating version 0 to 6 2024-04-19T01:04:04.943 INFO:tasks.rados.rados.0.smithi155.stdout:42: oid 42 version 6 is already newer than 5 2024-04-19T01:04:04.943 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 42 v 6 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-19T01:04:04.943 INFO:tasks.rados.rados.0.smithi155.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2024-04-19T01:04:04.943 INFO:tasks.rados.rados.0.smithi155.stdout:46: finishing write tid 2 to smithi15528762-46 2024-04-19T01:04:04.945 INFO:tasks.rados.rados.0.smithi155.stdout:37: done (13 left) 2024-04-19T01:04:04.945 INFO:tasks.rados.rados.0.smithi155.stdout:38: done (12 left) 2024-04-19T01:04:04.945 INFO:tasks.rados.rados.0.smithi155.stdout:39: done (11 left) 2024-04-19T01:04:04.945 INFO:tasks.rados.rados.0.smithi155.stdout:41: done (10 left) 2024-04-19T01:04:04.945 INFO:tasks.rados.rados.0.smithi155.stdout:42: done (9 left) 2024-04-19T01:04:04.945 INFO:tasks.rados.rados.0.smithi155.stdout:43: done (8 left) 2024-04-19T01:04:04.946 INFO:tasks.rados.rados.0.smithi155.stdout:44: done (7 left) 2024-04-19T01:04:04.946 INFO:tasks.rados.rados.0.smithi155.stdout:45: done (6 left) 2024-04-19T01:04:04.946 INFO:tasks.rados.rados.0.smithi155.stdout:51: done (5 left) 2024-04-19T01:04:04.946 INFO:tasks.rados.rados.0.smithi155.stdout:52: read oid 13 snap -1 2024-04-19T01:04:04.946 INFO:tasks.rados.rados.0.smithi155.stdout:52: expect (ObjNum 12 snap 0 seq_num 12) 2024-04-19T01:04:04.946 INFO:tasks.rados.rados.0.smithi155.stdout:53: snap_create 2024-04-19T01:04:04.946 INFO:tasks.rados.rados.0.smithi155.stdout:46: finishing write tid 3 to smithi15528762-46 2024-04-19T01:04:04.946 INFO:tasks.rados.rados.0.smithi155.stdout:46: oid 46 updating version 0 to 5 2024-04-19T01:04:04.946 INFO:tasks.rados.rados.0.smithi155.stdout:46: oid 46 updating version 5 to 6 2024-04-19T01:04:04.946 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 46 v 6 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-04-19T01:04:04.946 INFO:tasks.rados.rados.0.smithi155.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2024-04-19T01:04:04.946 INFO:tasks.rados.rados.0.smithi155.stdout:48: finishing write tid 2 to smithi15528762-48 2024-04-19T01:04:04.946 INFO:tasks.rados.rados.0.smithi155.stdout:48: finishing write tid 3 to smithi15528762-48 2024-04-19T01:04:04.947 INFO:tasks.rados.rados.0.smithi155.stdout:50: finishing write tid 1 to smithi15528762-50 2024-04-19T01:04:04.948 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:04:04.944+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:04:04.949 INFO:tasks.rados.rados.0.smithi155.stdout:50: finishing write tid 2 to smithi15528762-50 2024-04-19T01:04:04.950 INFO:tasks.rados.rados.0.smithi155.stdout:48: finishing write tid 4 to smithi15528762-48 2024-04-19T01:04:04.950 INFO:tasks.rados.rados.0.smithi155.stdout:48: oid 48 updating version 0 to 6 2024-04-19T01:04:04.951 INFO:tasks.rados.rados.0.smithi155.stdout:48: oid 48 updating version 6 to 7 2024-04-19T01:04:04.951 INFO:tasks.rados.rados.0.smithi155.stdout:48: oid 48 version 7 is already newer than 5 2024-04-19T01:04:04.951 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 48 v 7 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-04-19T01:04:04.951 INFO:tasks.rados.rados.0.smithi155.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2024-04-19T01:04:04.951 INFO:tasks.rados.rados.0.smithi155.stdout:50: finishing write tid 3 to smithi15528762-50 2024-04-19T01:04:04.951 INFO:tasks.rados.rados.0.smithi155.stdout:49: finishing write tid 1 to smithi15528762-49 2024-04-19T01:04:04.951 INFO:tasks.rados.rados.0.smithi155.stdout:49: finishing write tid 2 to smithi15528762-49 2024-04-19T01:04:04.951 INFO:tasks.rados.rados.0.smithi155.stdout:49: finishing write tid 3 to smithi15528762-49 2024-04-19T01:04:04.952 INFO:tasks.rados.rados.0.smithi155.stdout:52: expect (ObjNum 12 snap 0 seq_num 12) 2024-04-19T01:04:04.956 INFO:tasks.rados.rados.0.smithi155.stdout:47: finishing write tid 1 to smithi15528762-47 2024-04-19T01:04:04.956 INFO:tasks.rados.rados.0.smithi155.stdout:47: finishing write tid 2 to smithi15528762-47 2024-04-19T01:04:04.956 INFO:tasks.rados.rados.0.smithi155.stdout:50: finishing write tid 4 to smithi15528762-50 2024-04-19T01:04:04.956 INFO:tasks.rados.rados.0.smithi155.stdout:50: oid 50 updating version 0 to 9 2024-04-19T01:04:04.956 INFO:tasks.rados.rados.0.smithi155.stdout:50: oid 50 version 9 is already newer than 7 2024-04-19T01:04:04.956 INFO:tasks.rados.rados.0.smithi155.stdout:50: oid 50 version 9 is already newer than 8 2024-04-19T01:04:04.956 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 50 v 9 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-04-19T01:04:04.956 INFO:tasks.rados.rados.0.smithi155.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2024-04-19T01:04:04.956 INFO:tasks.rados.rados.0.smithi155.stdout:49: finishing write tid 4 to smithi15528762-49 2024-04-19T01:04:04.956 INFO:tasks.rados.rados.0.smithi155.stdout:49: oid 49 updating version 0 to 14 2024-04-19T01:04:04.957 INFO:tasks.rados.rados.0.smithi155.stdout:49: oid 49 updating version 14 to 15 2024-04-19T01:04:04.957 INFO:tasks.rados.rados.0.smithi155.stdout:49: oid 49 version 15 is already newer than 13 2024-04-19T01:04:04.957 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 49 v 15 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-19T01:04:04.957 INFO:tasks.rados.rados.0.smithi155.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2024-04-19T01:04:04.957 INFO:tasks.rados.rados.0.smithi155.stdout:47: finishing write tid 3 to smithi15528762-47 2024-04-19T01:04:04.957 INFO:tasks.rados.rados.0.smithi155.stdout:47: oid 47 updating version 0 to 7 2024-04-19T01:04:04.957 INFO:tasks.rados.rados.0.smithi155.stdout:47: oid 47 updating version 7 to 8 2024-04-19T01:04:04.957 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 47 v 8 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-04-19T01:04:04.957 INFO:tasks.rados.rados.0.smithi155.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2024-04-19T01:04:05.049 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:04:05.049+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:04:05.055 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:05.055 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:05.070 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-19T01:04:05.150 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:04:05.151+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:04:05.233 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:05.233 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:05.247 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-19T01:04:05.250 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:05.253+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:05.351 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:04:05.353+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:04:05.368 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:05.368 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:05.382 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-19T01:04:05.452 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:04:05.453+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:04:05.547 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:05.548 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:05.552 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:04:05.552+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:04:05.562 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-19T01:04:05.653 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:04:05.655+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:04:05.685 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:05.686 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:05.700 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-19T01:04:05.731 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-04-19T01:04:05.731 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-19T01:04:05.754 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2024-04-19T01:04:05.754 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:04:05.753+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:04:05.759 INFO:tasks.rados.rados.0.smithi155.stdout:46: done (6 left) 2024-04-19T01:04:05.760 INFO:tasks.rados.rados.0.smithi155.stdout:47: done (5 left) 2024-04-19T01:04:05.760 INFO:tasks.rados.rados.0.smithi155.stdout:48: done (4 left) 2024-04-19T01:04:05.760 INFO:tasks.rados.rados.0.smithi155.stdout:49: done (3 left) 2024-04-19T01:04:05.760 INFO:tasks.rados.rados.0.smithi155.stdout:50: done (2 left) 2024-04-19T01:04:05.760 INFO:tasks.rados.rados.0.smithi155.stdout:52: done (1 left) 2024-04-19T01:04:05.760 INFO:tasks.rados.rados.0.smithi155.stdout:53: done (0 left) 2024-04-19T01:04:05.760 INFO:tasks.rados.rados.0.smithi155.stdout:54: rollback oid 36 current snap is 1 2024-04-19T01:04:05.760 INFO:tasks.rados.rados.0.smithi155.stdout:rollback oid 36 to 0 2024-04-19T01:04:05.760 INFO:tasks.rados.rados.0.smithi155.stdout:55: delete oid 37 current snap is 1 2024-04-19T01:04:05.775 INFO:tasks.rados.rados.0.smithi155.stdout:54: finishing rollback tid 0 to smithi15528762-36 2024-04-19T01:04:05.775 INFO:tasks.rados.rados.0.smithi155.stdout:55: done (1 left) 2024-04-19T01:04:05.775 INFO:tasks.rados.rados.0.smithi155.stdout:56: copy_from oid 21 from oid 12 current snap is 1 2024-04-19T01:04:05.775 INFO:tasks.rados.rados.0.smithi155.stdout:57: snap_create 2024-04-19T01:04:05.776 INFO:tasks.rados.rados.0.smithi155.stdout:54: finishing rollback tid 1 to smithi15528762-36 2024-04-19T01:04:05.776 INFO:tasks.rados.rados.0.smithi155.stdout:54: finishing rollback tid 2 to smithi15528762-36 2024-04-19T01:04:05.776 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 36 v 13 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-04-19T01:04:05.789 INFO:tasks.rados.rados.0.smithi155.stdout:56: finishing copy_from racing read to smithi15528762-21 2024-04-19T01:04:05.792 INFO:tasks.rados.rados.0.smithi155.stdout:56: finishing copy_from to smithi15528762-21 2024-04-19T01:04:05.792 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 21 v 2 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-04-19T01:04:05.824 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:05.825 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:05.839 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-19T01:04:05.855 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:04:05.855+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:04:05.956 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:04:05.956+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:04:06.002 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:06.003 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:06.014 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-04-19T01:04:06.057 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:04:06.056+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:04:06.134 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:06.135 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:06.149 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-04-19T01:04:06.196 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:04:06.197+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:04:06.259 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:04:06.257+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:04:06.266 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:06.266 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:06.280 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-04-19T01:04:06.367 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:04:06.369+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:04:06.448 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:06.448 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:06.461 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:04:06.460+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:04:06.463 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-19T01:04:06.562 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:04:06.561+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:04:06.598 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:06.598 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:06.613 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-19T01:04:06.662 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:04:06.660+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:04:06.733 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:06.734 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:06.749 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-19T01:04:06.764 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:04:06.765+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:04:06.765 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.0 to 1 (10000) 2024-04-19T01:04:06.768 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-19T01:04:06.770 INFO:tasks.rados.rados.0.smithi155.stdout:54: done (2 left) 2024-04-19T01:04:06.770 INFO:tasks.rados.rados.0.smithi155.stdout:56: done (1 left) 2024-04-19T01:04:06.770 INFO:tasks.rados.rados.0.smithi155.stdout:57: done (0 left) 2024-04-19T01:04:06.770 INFO:tasks.rados.rados.0.smithi155.stdout:58: snap_create 2024-04-19T01:04:06.788 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-19T01:04:06.872 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:04:06.872+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:04:06.887 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:06.887 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:06.904 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-19T01:04:06.966 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:06.967+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:06.982 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-04-19T01:04:06.982 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T01:04:07.002 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-04-19T01:04:07.002 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T01:04:07.024 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-04-19T01:04:07.024 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T01:04:07.046 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-04-19T01:04:07.046 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T01:04:07.063 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-04-19T01:04:07.063 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T01:04:07.066 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:07.067 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:04:07.067 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2024-04-19T01:04:07.067 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:07.067 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:04:07.068+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:04:07.081 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-19T01:04:07.084 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-04-19T01:04:07.084 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T01:04:07.106 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-04-19T01:04:07.107 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T01:04:07.127 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-04-19T01:04:07.128 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T01:04:07.145 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2024-04-19T01:04:07.145 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T01:04:07.166 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2024-04-19T01:04:07.166 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T01:04:07.167 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:04:07.168+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:04:07.188 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2024-04-19T01:04:07.188 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T01:04:07.200 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:07.201 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:04:07.201 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2024-04-19T01:04:07.201 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2024-04-19T01:04:07.201 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:07.208 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2024-04-19T01:04:07.209 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T01:04:07.215 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-19T01:04:07.227 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2024-04-19T01:04:07.228 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-19T01:04:07.250 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2024-04-19T01:04:07.251 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-19T01:04:07.268 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:04:07.268+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:04:07.269 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2024-04-19T01:04:07.269 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-19T01:04:07.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:04:07.268+0000 7f23eed9d640 0 -- 172.21.15.33:0/2814217770 >> [v2:172.21.15.155:6807/741132602,v1:172.21.15.155:6810/741132602] conn(0x7f23f0226630 msgr2=0x7f23f0228a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-19T01:04:07.345 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:07.345 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2024-04-19T01:04:07.345 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2024-04-19T01:04:07.346 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [] 2024-04-19T01:04:07.346 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:07.360 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-19T01:04:07.370 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:04:07.369+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:04:07.471 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:04:07.472+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:04:07.496 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2024-04-19T01:04:07.496 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-19T01:04:07.538 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:07.538 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:04:07.538 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2024-04-19T01:04:07.538 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:07.552 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-19T01:04:07.572 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:04:07.572+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:04:07.674 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:07.671+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:07.718 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:07.719 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:04:07.719 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2024-04-19T01:04:07.719 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2024-04-19T01:04:07.719 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:07.733 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-19T01:04:07.774 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:04:07.773+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:04:07.779 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.1 to 1 (10000) 2024-04-19T01:04:07.786 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-19T01:04:07.787 INFO:tasks.rados.rados.0.smithi155.stdout:58: done (0 left) 2024-04-19T01:04:07.787 INFO:tasks.rados.rados.0.smithi155.stdout:59: snap_create 2024-04-19T01:04:07.803 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-19T01:04:07.864 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:07.864 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2024-04-19T01:04:07.864 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2024-04-19T01:04:07.864 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [] 2024-04-19T01:04:07.865 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:07.875 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:07.876+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:07.880 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-04-19T01:04:07.976 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:04:07.972+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:04:08.037 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:08.037 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:04:08.037 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2024-04-19T01:04:08.037 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:08.051 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-04-19T01:04:08.092 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:04:08.092+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:04:08.178 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:04:08.180+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:04:08.203 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:08.203 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:04:08.203 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2024-04-19T01:04:08.203 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2024-04-19T01:04:08.203 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:08.218 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-04-19T01:04:08.279 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:04:08.276+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:04:08.339 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:08.340 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2024-04-19T01:04:08.340 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2024-04-19T01:04:08.340 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [] 2024-04-19T01:04:08.340 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:08.354 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-19T01:04:08.402 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:08.403+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:08.481 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:04:08.479+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:04:08.486 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:08.487 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [], 2024-04-19T01:04:08.487 INFO:teuthology.orchestra.run.smithi110.stdout: "num_ops": 0 2024-04-19T01:04:08.487 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:08.500 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-19T01:04:08.586 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:04:08.587+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:04:08.665 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:08.666 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [], 2024-04-19T01:04:08.666 INFO:teuthology.orchestra.run.smithi110.stdout: "complaint_time": 30, 2024-04-19T01:04:08.666 INFO:teuthology.orchestra.run.smithi110.stdout: "num_blocked_ops": 0 2024-04-19T01:04:08.666 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:08.679 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-19T01:04:08.682 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:04:08.684+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:04:08.783 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:04:08.784+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:04:08.788 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2024-04-19T01:04:08.788 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.2 to 1 (10000) 2024-04-19T01:04:08.795 INFO:tasks.rados.rados.0.smithi155.stdout:59: done (0 left) 2024-04-19T01:04:08.795 INFO:tasks.rados.rados.0.smithi155.stdout:60: setattr oid 44 current snap is 4 2024-04-19T01:04:08.798 INFO:tasks.rados.rados.0.smithi155.stdout:61: rollback oid 5 current snap is 4 2024-04-19T01:04:08.798 INFO:tasks.rados.rados.0.smithi155.stdout:rollback oid 5 to 1 2024-04-19T01:04:08.798 INFO:tasks.rados.rados.0.smithi155.stdout:62: setattr oid 4 current snap is 4 2024-04-19T01:04:08.801 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 31 current snap is 4 2024-04-19T01:04:08.801 INFO:tasks.rados.rados.0.smithi155.stdout:63: seq_num 52 ranges {507904=507904,1015808=376832} 2024-04-19T01:04:08.813 INFO:tasks.rados.rados.0.smithi155.stdout:63: writing smithi15528762-31 from 507904 to 1015808 tid 1 2024-04-19T01:04:08.818 INFO:tasks.rados.rados.0.smithi155.stdout:63: writing smithi15528762-31 from 1015808 to 1392640 tid 2 2024-04-19T01:04:08.818 INFO:tasks.rados.rados.0.smithi155.stdout:64: snap_create 2024-04-19T01:04:08.818 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 44 v 14 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-04-19T01:04:08.818 INFO:tasks.rados.rados.0.smithi155.stdout:61: finishing rollback tid 0 to smithi15528762-5 2024-04-19T01:04:08.819 INFO:tasks.rados.rados.0.smithi155.stdout:61: finishing rollback tid 1 to smithi15528762-5 2024-04-19T01:04:08.819 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 4 v 15 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-19T01:04:08.819 INFO:tasks.rados.rados.0.smithi155.stdout:61: finishing rollback tid 2 to smithi15528762-5 2024-04-19T01:04:08.819 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 5 v 10 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-04-19T01:04:08.821 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-19T01:04:08.839 INFO:tasks.rados.rados.0.smithi155.stdout:63: finishing write tid 1 to smithi15528762-31 2024-04-19T01:04:08.839 INFO:tasks.rados.rados.0.smithi155.stdout:63: finishing write tid 2 to smithi15528762-31 2024-04-19T01:04:08.841 INFO:tasks.rados.rados.0.smithi155.stdout:63: finishing write tid 3 to smithi15528762-31 2024-04-19T01:04:08.843 INFO:tasks.rados.rados.0.smithi155.stdout:63: finishing write tid 4 to smithi15528762-31 2024-04-19T01:04:08.843 INFO:tasks.rados.rados.0.smithi155.stdout:63: oid 31 updating version 0 to 11 2024-04-19T01:04:08.843 INFO:tasks.rados.rados.0.smithi155.stdout:63: oid 31 version 11 is already newer than 10 2024-04-19T01:04:08.843 INFO:tasks.rados.rados.0.smithi155.stdout:63: oid 31 version 11 is already newer than 9 2024-04-19T01:04:08.843 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 31 v 11 (ObjNum 52 snap 4 seq_num 52) dirty exists 2024-04-19T01:04:08.843 INFO:tasks.rados.rados.0.smithi155.stdout:63: left oid 31 (ObjNum 52 snap 4 seq_num 52) 2024-04-19T01:04:08.856 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:08.857 INFO:teuthology.orchestra.run.smithi110.stdout: "size": 20, 2024-04-19T01:04:08.857 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 600, 2024-04-19T01:04:08.857 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [] 2024-04-19T01:04:08.857 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:08.870 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-19T01:04:08.884 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:04:08.885+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:04:08.985 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:04:08.985+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:04:08.993 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:08.993 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [], 2024-04-19T01:04:08.993 INFO:teuthology.orchestra.run.smithi110.stdout: "num_ops": 0 2024-04-19T01:04:08.994 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:09.008 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-19T01:04:09.086 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:04:09.084+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:04:09.126 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:09.126 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [], 2024-04-19T01:04:09.126 INFO:teuthology.orchestra.run.smithi110.stdout: "complaint_time": 30, 2024-04-19T01:04:09.126 INFO:teuthology.orchestra.run.smithi110.stdout: "num_blocked_ops": 0 2024-04-19T01:04:09.126 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:09.140 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-19T01:04:09.186 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:04:09.187+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:04:09.268 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:09.268 INFO:teuthology.orchestra.run.smithi110.stdout: "size": 20, 2024-04-19T01:04:09.268 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 600, 2024-04-19T01:04:09.268 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [ 2024-04-19T01:04:09.268 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.269 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_repop(client.4260.0:20 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'10, mlcod=24'8)", 2024-04-19T01:04:09.269 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:03:50.815683+0000", 2024-04-19T01:04:09.269 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 18.452002821000001, 2024-04-19T01:04:09.269 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.003824222, 2024-04-19T01:04:09.269 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:09.269 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:09.269 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:09.269 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:09.269 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.269 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:09.269 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.815683+0000", 2024-04-19T01:04:09.269 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.269 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.269 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.269 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:09.270 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.815683+0000", 2024-04-19T01:04:09.270 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.270 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.270 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.270 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:09.270 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.815685+0000", 2024-04-19T01:04:09.270 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.889e-06 2024-04-19T01:04:09.270 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.270 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.270 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:09.270 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.815902+0000", 2024-04-19T01:04:09.270 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00021718099999999999 2024-04-19T01:04:09.270 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.271 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.271 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:09.271 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.815930+0000", 2024-04-19T01:04:09.271 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.8581999999999998e-05 2024-04-19T01:04:09.271 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.271 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.271 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:09.271 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.815945+0000", 2024-04-19T01:04:09.271 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.4935e-05 2024-04-19T01:04:09.271 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.271 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.271 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:09.271 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.816070+0000", 2024-04-19T01:04:09.271 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00012439600000000001 2024-04-19T01:04:09.271 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.272 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.272 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:09.272 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.816129+0000", 2024-04-19T01:04:09.272 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.9284999999999999e-05 2024-04-19T01:04:09.272 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.272 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.272 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:09.272 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.819417+0000", 2024-04-19T01:04:09.272 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.003288231 2024-04-19T01:04:09.272 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.272 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.272 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:09.272 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.819507+0000", 2024-04-19T01:04:09.272 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.9722999999999995e-05 2024-04-19T01:04:09.272 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.273 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:09.273 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.273 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.273 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.273 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_repop(client.4260.0:38 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'24, mlcod=24'22)", 2024-04-19T01:04:09.273 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:03:50.848838+0000", 2024-04-19T01:04:09.273 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 18.418847802999998, 2024-04-19T01:04:09.273 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0037599840000000001, 2024-04-19T01:04:09.273 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:09.273 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:09.273 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:09.273 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:09.273 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.273 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:09.274 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.848838+0000", 2024-04-19T01:04:09.274 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.274 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.274 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.274 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:09.274 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.848838+0000", 2024-04-19T01:04:09.274 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.274 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.274 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.275 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:09.275 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.848839+0000", 2024-04-19T01:04:09.275 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.3999999999999999e-06 2024-04-19T01:04:09.275 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.275 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.275 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:09.275 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.848967+0000", 2024-04-19T01:04:09.275 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00012833500000000001 2024-04-19T01:04:09.275 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.275 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.275 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:09.275 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.848990+0000", 2024-04-19T01:04:09.275 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2858e-05 2024-04-19T01:04:09.275 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.275 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.276 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:09.276 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.849003+0000", 2024-04-19T01:04:09.276 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.2799000000000001e-05 2024-04-19T01:04:09.276 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.276 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.276 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:09.276 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.849136+0000", 2024-04-19T01:04:09.276 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00013309199999999999 2024-04-19T01:04:09.276 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.276 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.276 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:09.276 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.849195+0000", 2024-04-19T01:04:09.276 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.8477999999999998e-05 2024-04-19T01:04:09.276 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.277 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.277 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:09.277 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.852523+0000", 2024-04-19T01:04:09.277 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0033278420000000001 2024-04-19T01:04:09.277 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.277 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.277 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:09.277 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.852598+0000", 2024-04-19T01:04:09.277 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.5179999999999995e-05 2024-04-19T01:04:09.277 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.277 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:09.277 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.277 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.277 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.277 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_repop(client.4260.0:57 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'35, mlcod=24'33)", 2024-04-19T01:04:09.278 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:03:50.885502+0000", 2024-04-19T01:04:09.278 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 18.382183995999998, 2024-04-19T01:04:09.278 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0036481769999999998, 2024-04-19T01:04:09.278 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:09.278 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:09.278 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:09.278 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:09.278 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.278 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:09.278 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.885502+0000", 2024-04-19T01:04:09.278 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.278 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.278 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.278 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:09.279 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.885502+0000", 2024-04-19T01:04:09.279 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.279 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.279 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.279 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:09.279 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.885504+0000", 2024-04-19T01:04:09.279 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0200000000000001e-06 2024-04-19T01:04:09.279 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.279 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.279 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:09.279 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.885618+0000", 2024-04-19T01:04:09.279 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000113973 2024-04-19T01:04:09.279 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.279 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.279 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:09.280 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.885646+0000", 2024-04-19T01:04:09.280 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.8724000000000001e-05 2024-04-19T01:04:09.280 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.280 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.280 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:09.280 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.885660+0000", 2024-04-19T01:04:09.280 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.3269e-05 2024-04-19T01:04:09.280 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.280 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.280 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:09.280 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.885785+0000", 2024-04-19T01:04:09.280 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00012533600000000001 2024-04-19T01:04:09.280 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.280 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.280 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:09.281 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.885841+0000", 2024-04-19T01:04:09.281 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.5705000000000003e-05 2024-04-19T01:04:09.281 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.281 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.281 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:09.281 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.889029+0000", 2024-04-19T01:04:09.281 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0031879669999999999 2024-04-19T01:04:09.281 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.281 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.281 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:09.281 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.889150+0000", 2024-04-19T01:04:09.281 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00012118299999999999 2024-04-19T01:04:09.281 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.281 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:09.281 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.282 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.282 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.282 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_repop(client.4260.0:71 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'45, mlcod=24'43)", 2024-04-19T01:04:09.282 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:03:50.913382+0000", 2024-04-19T01:04:09.282 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 18.354303204000001, 2024-04-19T01:04:09.282 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0035805559999999999, 2024-04-19T01:04:09.282 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:09.282 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:09.282 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:09.282 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:09.282 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.282 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:09.283 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.913382+0000", 2024-04-19T01:04:09.283 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.283 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.283 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.283 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:09.283 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.913382+0000", 2024-04-19T01:04:09.283 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.283 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.283 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.283 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:09.283 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.913384+0000", 2024-04-19T01:04:09.283 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0059999999999999e-06 2024-04-19T01:04:09.283 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.283 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.284 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:09.284 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.913446+0000", 2024-04-19T01:04:09.284 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.1808000000000002e-05 2024-04-19T01:04:09.284 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.284 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.284 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:09.284 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.913475+0000", 2024-04-19T01:04:09.284 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.8464999999999999e-05 2024-04-19T01:04:09.284 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.284 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.284 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:09.284 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.913493+0000", 2024-04-19T01:04:09.284 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.8465e-05 2024-04-19T01:04:09.285 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.285 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.285 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:09.285 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.913619+0000", 2024-04-19T01:04:09.285 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000125745 2024-04-19T01:04:09.285 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.285 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.285 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:09.285 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.913655+0000", 2024-04-19T01:04:09.285 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.5994000000000001e-05 2024-04-19T01:04:09.285 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.285 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.285 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:09.285 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.916849+0000", 2024-04-19T01:04:09.285 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0031939799999999999 2024-04-19T01:04:09.286 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.286 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.286 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:09.286 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.916963+0000", 2024-04-19T01:04:09.286 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00011409299999999999 2024-04-19T01:04:09.286 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.286 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:09.286 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.286 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.286 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.286 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_repop(client.4260.0:72 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'46, mlcod=24'43)", 2024-04-19T01:04:09.286 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:03:50.915181+0000", 2024-04-19T01:04:09.286 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 18.352504755999998, 2024-04-19T01:04:09.286 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0034728379999999998, 2024-04-19T01:04:09.287 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:09.287 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:09.287 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:09.287 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:09.287 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.287 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:09.287 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.915181+0000", 2024-04-19T01:04:09.287 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.287 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.287 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.287 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:09.287 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.915181+0000", 2024-04-19T01:04:09.287 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.287 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.287 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.288 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:09.288 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.915182+0000", 2024-04-19T01:04:09.288 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.59e-06 2024-04-19T01:04:09.288 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.288 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.288 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:09.288 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.915244+0000", 2024-04-19T01:04:09.288 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.19e-05 2024-04-19T01:04:09.288 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.288 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.288 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:09.288 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.915266+0000", 2024-04-19T01:04:09.288 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1739e-05 2024-04-19T01:04:09.288 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.288 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.289 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:09.289 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.915278+0000", 2024-04-19T01:04:09.289 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.2354999999999999e-05 2024-04-19T01:04:09.289 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.289 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.289 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:09.289 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.915416+0000", 2024-04-19T01:04:09.289 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00013732499999999999 2024-04-19T01:04:09.289 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.289 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.289 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:09.289 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.915470+0000", 2024-04-19T01:04:09.289 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.3826000000000003e-05 2024-04-19T01:04:09.289 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.289 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.290 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:09.290 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.918590+0000", 2024-04-19T01:04:09.290 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.003120149 2024-04-19T01:04:09.290 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.290 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.290 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:09.290 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.918654+0000", 2024-04-19T01:04:09.290 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.3954000000000005e-05 2024-04-19T01:04:09.290 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.290 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:09.290 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.290 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.291 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.291 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_repop(client.4260.0:89 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'59, mlcod=24'57)", 2024-04-19T01:04:09.291 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:03:50.944075+0000", 2024-04-19T01:04:09.291 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 18.323610969000001, 2024-04-19T01:04:09.291 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0038283380000000001, 2024-04-19T01:04:09.291 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:09.291 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:09.291 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:09.291 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:09.291 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.291 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:09.291 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.944075+0000", 2024-04-19T01:04:09.291 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.291 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.292 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.292 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:09.292 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.944075+0000", 2024-04-19T01:04:09.292 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.292 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.292 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.292 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:09.292 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.944077+0000", 2024-04-19T01:04:09.292 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9929999999999998e-06 2024-04-19T01:04:09.292 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.292 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.292 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:09.292 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.944200+0000", 2024-04-19T01:04:09.292 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00012294300000000001 2024-04-19T01:04:09.292 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.293 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.293 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:09.293 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.944229+0000", 2024-04-19T01:04:09.293 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.9213999999999998e-05 2024-04-19T01:04:09.293 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.293 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.293 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:09.293 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.944242+0000", 2024-04-19T01:04:09.293 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.3345e-05 2024-04-19T01:04:09.293 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.293 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.293 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:09.293 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.944366+0000", 2024-04-19T01:04:09.293 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00012437 2024-04-19T01:04:09.293 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.294 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.294 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:09.294 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.944420+0000", 2024-04-19T01:04:09.294 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.3814999999999999e-05 2024-04-19T01:04:09.294 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.294 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.294 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:09.294 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.947820+0000", 2024-04-19T01:04:09.294 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0034002149999999998 2024-04-19T01:04:09.294 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.294 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.294 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:09.294 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:50.947903+0000", 2024-04-19T01:04:09.294 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.2442999999999997e-05 2024-04-19T01:04:09.295 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.295 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:09.295 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.295 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.295 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.295 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_repop(client.4260.0:167 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'121, mlcod=24'118)", 2024-04-19T01:04:09.295 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:03:51.295296+0000", 2024-04-19T01:04:09.295 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 17.972389824, 2024-04-19T01:04:09.295 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.003773041, 2024-04-19T01:04:09.295 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:09.295 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:09.295 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:09.296 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:09.296 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.296 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:09.296 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.295296+0000", 2024-04-19T01:04:09.296 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.296 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.296 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.296 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:09.296 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.295296+0000", 2024-04-19T01:04:09.296 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.296 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.296 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.296 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:09.296 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.295302+0000", 2024-04-19T01:04:09.297 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.2160000000000003e-06 2024-04-19T01:04:09.297 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.297 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.297 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:09.297 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.295356+0000", 2024-04-19T01:04:09.297 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.4422000000000002e-05 2024-04-19T01:04:09.297 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.297 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.297 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:09.297 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.295381+0000", 2024-04-19T01:04:09.297 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4774999999999999e-05 2024-04-19T01:04:09.297 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.297 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.297 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:09.298 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.295393+0000", 2024-04-19T01:04:09.298 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.2038999999999999e-05 2024-04-19T01:04:09.298 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.298 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.298 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:09.298 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.295494+0000", 2024-04-19T01:04:09.298 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000100645 2024-04-19T01:04:09.298 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.298 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.298 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:09.298 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.295527+0000", 2024-04-19T01:04:09.298 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.3389999999999997e-05 2024-04-19T01:04:09.298 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.298 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.299 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:09.299 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.298988+0000", 2024-04-19T01:04:09.299 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0034611970000000001 2024-04-19T01:04:09.299 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.299 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.299 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:09.299 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.299069+0000", 2024-04-19T01:04:09.299 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.0357000000000003e-05 2024-04-19T01:04:09.299 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.299 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:09.299 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.299 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.299 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.299 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_repop(client.4260.0:168 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'122, mlcod=24'118)", 2024-04-19T01:04:09.299 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:03:51.297073+0000", 2024-04-19T01:04:09.300 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 17.970612127999999, 2024-04-19T01:04:09.300 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0039083089999999996, 2024-04-19T01:04:09.300 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:09.300 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:09.300 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:09.300 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:09.300 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.300 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:09.300 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.297073+0000", 2024-04-19T01:04:09.300 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.300 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.300 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.300 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:09.300 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.297073+0000", 2024-04-19T01:04:09.301 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.301 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.301 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.301 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:09.301 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.297075+0000", 2024-04-19T01:04:09.301 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.1340000000000001e-06 2024-04-19T01:04:09.301 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.301 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.301 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:09.301 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.297150+0000", 2024-04-19T01:04:09.301 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.5313000000000002e-05 2024-04-19T01:04:09.301 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.301 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.301 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:09.301 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.297183+0000", 2024-04-19T01:04:09.302 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.3043999999999997e-05 2024-04-19T01:04:09.302 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.302 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.302 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:09.302 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.297193+0000", 2024-04-19T01:04:09.302 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.8630000000000003e-06 2024-04-19T01:04:09.302 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.302 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.302 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:09.302 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.297330+0000", 2024-04-19T01:04:09.302 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000137214 2024-04-19T01:04:09.302 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.302 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.303 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:09.303 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.297360+0000", 2024-04-19T01:04:09.303 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.9604999999999999e-05 2024-04-19T01:04:09.303 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.303 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.306 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:09.306 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.300878+0000", 2024-04-19T01:04:09.306 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0035188480000000002 2024-04-19T01:04:09.306 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.306 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.306 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:09.306 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.300982+0000", 2024-04-19T01:04:09.306 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000103288 2024-04-19T01:04:09.306 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.306 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:09.307 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.307 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.307 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.307 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_repop(client.4260.0:182 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'132, mlcod=24'125)", 2024-04-19T01:04:09.307 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:03:51.378028+0000", 2024-04-19T01:04:09.307 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 17.889657527000001, 2024-04-19T01:04:09.307 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0051310590000000003, 2024-04-19T01:04:09.307 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:09.307 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:09.307 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:09.307 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:09.307 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.307 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:09.307 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.378028+0000", 2024-04-19T01:04:09.308 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.308 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.308 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.308 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:09.308 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.378028+0000", 2024-04-19T01:04:09.308 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.308 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.308 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.308 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:09.308 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.378030+0000", 2024-04-19T01:04:09.308 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.04e-06 2024-04-19T01:04:09.308 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.308 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.308 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:09.308 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.378089+0000", 2024-04-19T01:04:09.309 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.8485e-05 2024-04-19T01:04:09.309 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.309 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.309 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:09.309 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.378114+0000", 2024-04-19T01:04:09.309 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5388000000000002e-05 2024-04-19T01:04:09.309 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.309 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.309 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:09.309 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.378127+0000", 2024-04-19T01:04:09.309 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.3003000000000001e-05 2024-04-19T01:04:09.309 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.309 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.309 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:09.309 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.378756+0000", 2024-04-19T01:04:09.310 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00062911900000000003 2024-04-19T01:04:09.310 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.310 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.310 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:09.310 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.378792+0000", 2024-04-19T01:04:09.310 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.5654000000000001e-05 2024-04-19T01:04:09.310 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.310 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.310 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:09.310 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.383081+0000", 2024-04-19T01:04:09.310 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0042894099999999996 2024-04-19T01:04:09.310 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.310 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.310 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:09.311 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.383159+0000", 2024-04-19T01:04:09.311 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.7960000000000006e-05 2024-04-19T01:04:09.311 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.311 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:09.311 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.311 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.311 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.311 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_repop(client.4260.0:183 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'133, mlcod=24'126)", 2024-04-19T01:04:09.311 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:03:51.378731+0000", 2024-04-19T01:04:09.311 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 17.888954106, 2024-04-19T01:04:09.311 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0044892930000000001, 2024-04-19T01:04:09.311 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:09.311 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:09.311 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:09.311 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:09.312 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.312 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:09.312 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.378731+0000", 2024-04-19T01:04:09.312 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.312 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.312 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.312 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:09.312 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.378731+0000", 2024-04-19T01:04:09.312 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.312 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.312 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.312 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:09.312 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.378733+0000", 2024-04-19T01:04:09.312 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.3060000000000001e-06 2024-04-19T01:04:09.312 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.313 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.313 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:09.313 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.378743+0000", 2024-04-19T01:04:09.313 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.0635999999999999e-05 2024-04-19T01:04:09.313 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.313 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.313 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:09.313 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.378766+0000", 2024-04-19T01:04:09.313 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2952e-05 2024-04-19T01:04:09.313 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.313 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.313 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:09.313 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.378779+0000", 2024-04-19T01:04:09.314 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.2972e-05 2024-04-19T01:04:09.314 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.314 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.314 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:09.314 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.379525+0000", 2024-04-19T01:04:09.314 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00074595300000000005 2024-04-19T01:04:09.314 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.314 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.314 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:09.314 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.379555+0000", 2024-04-19T01:04:09.314 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.0125e-05 2024-04-19T01:04:09.314 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.314 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.314 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:09.315 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.383164+0000", 2024-04-19T01:04:09.315 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0036090300000000001 2024-04-19T01:04:09.315 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.315 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.315 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:09.315 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.383221+0000", 2024-04-19T01:04:09.315 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.6319000000000001e-05 2024-04-19T01:04:09.315 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.315 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:09.315 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.315 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.315 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.315 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_repop(client.4260.0:184 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'134, mlcod=24'126)", 2024-04-19T01:04:09.315 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:03:51.379162+0000", 2024-04-19T01:04:09.316 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 17.888523212999999, 2024-04-19T01:04:09.316 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0041721830000000003, 2024-04-19T01:04:09.316 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:09.316 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:09.316 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:09.316 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:09.316 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.316 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:09.316 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.379162+0000", 2024-04-19T01:04:09.316 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.316 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.316 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.316 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:09.316 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.379162+0000", 2024-04-19T01:04:09.316 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.317 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.317 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.317 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:09.317 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.379164+0000", 2024-04-19T01:04:09.317 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.404e-06 2024-04-19T01:04:09.317 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.317 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.317 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:09.317 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.379174+0000", 2024-04-19T01:04:09.317 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.0319000000000001e-05 2024-04-19T01:04:09.317 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.317 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.317 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:09.317 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.379194+0000", 2024-04-19T01:04:09.318 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0278000000000001e-05 2024-04-19T01:04:09.318 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.318 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.318 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:09.318 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.379209+0000", 2024-04-19T01:04:09.318 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.4392e-05 2024-04-19T01:04:09.318 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.318 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.318 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:09.318 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.380261+0000", 2024-04-19T01:04:09.318 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0010519539999999999 2024-04-19T01:04:09.318 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.318 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.318 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:09.319 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.380294+0000", 2024-04-19T01:04:09.319 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.3430999999999999e-05 2024-04-19T01:04:09.319 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.319 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.319 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:09.319 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.383225+0000", 2024-04-19T01:04:09.319 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0029308810000000002 2024-04-19T01:04:09.319 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.319 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.319 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:09.319 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.383335+0000", 2024-04-19T01:04:09.319 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000109524 2024-04-19T01:04:09.319 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.319 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:09.320 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.320 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.320 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.320 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_repop(client.4260.0:185 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'135, mlcod=24'128)", 2024-04-19T01:04:09.320 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:03:51.379997+0000", 2024-04-19T01:04:09.320 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 17.887688164, 2024-04-19T01:04:09.320 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0053250729999999996, 2024-04-19T01:04:09.320 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:09.320 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:09.320 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:09.320 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:09.320 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.320 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:09.320 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.379997+0000", 2024-04-19T01:04:09.320 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.321 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.321 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.321 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:09.321 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.379997+0000", 2024-04-19T01:04:09.321 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.321 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.321 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.321 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:09.321 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.379999+0000", 2024-04-19T01:04:09.321 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9769999999999999e-06 2024-04-19T01:04:09.321 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.321 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.321 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:09.321 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.380052+0000", 2024-04-19T01:04:09.322 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.3086e-05 2024-04-19T01:04:09.322 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.322 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.322 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:09.322 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.380078+0000", 2024-04-19T01:04:09.322 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5494e-05 2024-04-19T01:04:09.322 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.322 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.322 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:09.322 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.380091+0000", 2024-04-19T01:04:09.322 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.2849e-05 2024-04-19T01:04:09.322 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.322 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.323 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:09.323 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.380745+0000", 2024-04-19T01:04:09.323 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00065471099999999999 2024-04-19T01:04:09.323 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.323 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.323 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:09.323 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.380774+0000", 2024-04-19T01:04:09.323 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.8824e-05 2024-04-19T01:04:09.323 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.323 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.323 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:09.323 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.385251+0000", 2024-04-19T01:04:09.323 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0044768280000000004 2024-04-19T01:04:09.324 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.324 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.324 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:09.324 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.385322+0000", 2024-04-19T01:04:09.324 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.1303999999999994e-05 2024-04-19T01:04:09.324 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.324 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:09.324 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.324 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.324 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.324 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_repop(client.4260.0:186 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'136, mlcod=24'128)", 2024-04-19T01:04:09.324 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:03:51.381061+0000", 2024-04-19T01:04:09.324 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 17.886624950000002, 2024-04-19T01:04:09.324 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00432089, 2024-04-19T01:04:09.325 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:09.325 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:09.325 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:09.325 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:09.325 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.325 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:09.325 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.381061+0000", 2024-04-19T01:04:09.325 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.325 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.325 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.325 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:09.325 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.381061+0000", 2024-04-19T01:04:09.325 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.325 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.326 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.326 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:09.326 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.381062+0000", 2024-04-19T01:04:09.326 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.023e-06 2024-04-19T01:04:09.326 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.326 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.326 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:09.326 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.381073+0000", 2024-04-19T01:04:09.326 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.1602e-05 2024-04-19T01:04:09.326 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.326 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.326 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:09.326 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.381089+0000", 2024-04-19T01:04:09.326 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5509e-05 2024-04-19T01:04:09.327 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.327 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.327 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:09.327 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.381129+0000", 2024-04-19T01:04:09.327 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.057e-05 2024-04-19T01:04:09.327 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.327 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.327 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:09.327 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.382610+0000", 2024-04-19T01:04:09.327 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0014805949999999999 2024-04-19T01:04:09.327 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.327 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.327 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:09.327 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.382645+0000", 2024-04-19T01:04:09.328 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.5209999999999997e-05 2024-04-19T01:04:09.328 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.328 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.328 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:09.328 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.385328+0000", 2024-04-19T01:04:09.328 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0026828590000000001 2024-04-19T01:04:09.328 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.328 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.328 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:09.328 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.385381+0000", 2024-04-19T01:04:09.328 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.3522e-05 2024-04-19T01:04:09.328 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.328 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:09.328 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.329 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.329 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.329 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_repop(client.4260.0:187 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'137, mlcod=24'130)", 2024-04-19T01:04:09.329 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:03:51.381838+0000", 2024-04-19T01:04:09.329 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 17.885847973000001, 2024-04-19T01:04:09.329 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0036235159999999998, 2024-04-19T01:04:09.329 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:09.329 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:09.329 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:09.329 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:09.329 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.329 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:09.329 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.381838+0000", 2024-04-19T01:04:09.329 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.330 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.330 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.330 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:09.330 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.381838+0000", 2024-04-19T01:04:09.330 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.330 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.330 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.330 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:09.330 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.381839+0000", 2024-04-19T01:04:09.330 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.8929999999999999e-06 2024-04-19T01:04:09.330 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.330 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.330 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:09.331 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.381852+0000", 2024-04-19T01:04:09.331 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.2228999999999999e-05 2024-04-19T01:04:09.331 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.331 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.331 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:09.331 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.381879+0000", 2024-04-19T01:04:09.331 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.7548000000000001e-05 2024-04-19T01:04:09.331 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.331 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.331 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:09.331 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.381892+0000", 2024-04-19T01:04:09.331 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.2901999999999999e-05 2024-04-19T01:04:09.331 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.331 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.332 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:09.332 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.383411+0000", 2024-04-19T01:04:09.332 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0015193019999999999 2024-04-19T01:04:09.332 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.332 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.332 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:09.332 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.383463+0000", 2024-04-19T01:04:09.332 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.1372000000000002e-05 2024-04-19T01:04:09.332 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.332 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.332 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:09.332 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.385386+0000", 2024-04-19T01:04:09.332 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0019233500000000001 2024-04-19T01:04:09.332 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.332 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.333 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:09.333 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.385461+0000", 2024-04-19T01:04:09.333 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.4919999999999994e-05 2024-04-19T01:04:09.333 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.333 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:09.333 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.333 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.333 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.333 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_repop(client.4260.0:188 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'138, mlcod=24'130)", 2024-04-19T01:04:09.333 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:03:51.382350+0000", 2024-04-19T01:04:09.333 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 17.885335188999999, 2024-04-19T01:04:09.333 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0037732500000000001, 2024-04-19T01:04:09.333 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:09.334 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:09.334 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:09.334 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:09.334 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.334 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:09.334 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.382350+0000", 2024-04-19T01:04:09.334 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.334 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.334 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.334 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:09.334 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.382350+0000", 2024-04-19T01:04:09.334 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.334 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.334 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.335 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:09.335 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.382352+0000", 2024-04-19T01:04:09.335 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9530000000000002e-06 2024-04-19T01:04:09.335 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.335 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.335 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:09.335 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.382410+0000", 2024-04-19T01:04:09.335 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.8084999999999997e-05 2024-04-19T01:04:09.335 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.335 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.335 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:09.335 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.382438+0000", 2024-04-19T01:04:09.335 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.7954e-05 2024-04-19T01:04:09.335 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.335 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.336 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:09.336 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.382451+0000", 2024-04-19T01:04:09.336 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.2918999999999999e-05 2024-04-19T01:04:09.336 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.336 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.336 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:09.336 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.383965+0000", 2024-04-19T01:04:09.336 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001513978 2024-04-19T01:04:09.336 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.336 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.336 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:09.336 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.383995+0000", 2024-04-19T01:04:09.336 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.0023999999999999e-05 2024-04-19T01:04:09.336 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.336 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.337 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:09.337 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.386043+0000", 2024-04-19T01:04:09.337 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0020472939999999999 2024-04-19T01:04:09.337 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.337 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.337 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:09.337 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.386124+0000", 2024-04-19T01:04:09.337 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.1043000000000003e-05 2024-04-19T01:04:09.337 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.337 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:09.337 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.337 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.337 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.338 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_repop(client.4260.0:189 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'139, mlcod=24'130)", 2024-04-19T01:04:09.338 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:03:51.383015+0000", 2024-04-19T01:04:09.338 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 17.884670292999999, 2024-04-19T01:04:09.338 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0037579319999999999, 2024-04-19T01:04:09.338 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:09.338 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:09.338 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:09.338 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:09.338 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.338 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:09.338 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.383015+0000", 2024-04-19T01:04:09.338 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.338 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.339 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.339 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:09.339 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.383015+0000", 2024-04-19T01:04:09.339 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.339 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.339 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.339 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:09.339 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.383017+0000", 2024-04-19T01:04:09.339 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.3170000000000001e-06 2024-04-19T01:04:09.339 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.339 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.339 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:09.339 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.383028+0000", 2024-04-19T01:04:09.339 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.0951999999999999e-05 2024-04-19T01:04:09.339 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.340 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.340 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:09.340 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.383046+0000", 2024-04-19T01:04:09.340 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.8128999999999999e-05 2024-04-19T01:04:09.340 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.340 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.340 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:09.340 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.383056+0000", 2024-04-19T01:04:09.340 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.0569e-05 2024-04-19T01:04:09.340 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.340 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.340 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:09.340 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.385565+0000", 2024-04-19T01:04:09.340 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0025083670000000001 2024-04-19T01:04:09.341 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.341 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.341 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:09.341 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.385615+0000", 2024-04-19T01:04:09.341 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.0679e-05 2024-04-19T01:04:09.341 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.341 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.341 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:09.341 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.386717+0000", 2024-04-19T01:04:09.341 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0011022 2024-04-19T01:04:09.341 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.341 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.341 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:09.341 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.386773+0000", 2024-04-19T01:04:09.342 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.5719e-05 2024-04-19T01:04:09.342 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.342 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:09.342 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.342 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.342 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.342 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_repop(client.4260.0:190 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'140, mlcod=24'138)", 2024-04-19T01:04:09.342 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:03:51.387836+0000", 2024-04-19T01:04:09.342 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 17.879849818, 2024-04-19T01:04:09.342 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.005045852, 2024-04-19T01:04:09.342 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:09.342 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:09.342 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:09.343 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:09.343 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.343 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:09.343 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.387836+0000", 2024-04-19T01:04:09.343 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.343 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.343 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.343 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:09.343 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.387836+0000", 2024-04-19T01:04:09.343 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.343 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.343 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.343 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:09.343 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.387837+0000", 2024-04-19T01:04:09.344 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.44e-06 2024-04-19T01:04:09.344 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.344 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.344 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:09.344 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.387912+0000", 2024-04-19T01:04:09.344 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.4729999999999998e-05 2024-04-19T01:04:09.344 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.344 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.344 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:09.344 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.387939+0000", 2024-04-19T01:04:09.344 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.7005e-05 2024-04-19T01:04:09.344 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.344 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.344 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:09.344 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.387952+0000", 2024-04-19T01:04:09.345 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.3079e-05 2024-04-19T01:04:09.345 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.345 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.345 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:09.345 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.388067+0000", 2024-04-19T01:04:09.345 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000114616 2024-04-19T01:04:09.345 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.345 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.345 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:09.345 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.388103+0000", 2024-04-19T01:04:09.345 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.6087000000000002e-05 2024-04-19T01:04:09.345 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.345 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.345 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:09.345 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.392798+0000", 2024-04-19T01:04:09.346 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0046950059999999998 2024-04-19T01:04:09.346 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.346 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.346 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:09.346 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.392882+0000", 2024-04-19T01:04:09.346 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.3888999999999996e-05 2024-04-19T01:04:09.346 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.346 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:09.346 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.346 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.346 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.346 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_repop(client.4260.0:191 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'141, mlcod=24'138)", 2024-04-19T01:04:09.346 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:03:51.388788+0000", 2024-04-19T01:04:09.347 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 17.878897115000001, 2024-04-19T01:04:09.347 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0041476849999999999, 2024-04-19T01:04:09.347 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:09.347 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:09.347 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:09.347 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:09.347 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.347 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:09.347 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.388788+0000", 2024-04-19T01:04:09.347 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.347 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.347 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.347 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:09.347 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.388788+0000", 2024-04-19T01:04:09.348 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.348 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.348 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.348 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:09.348 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.388790+0000", 2024-04-19T01:04:09.348 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9530000000000002e-06 2024-04-19T01:04:09.348 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.348 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.348 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:09.348 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.388853+0000", 2024-04-19T01:04:09.348 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.2150999999999995e-05 2024-04-19T01:04:09.348 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.348 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.348 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:09.348 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.388880+0000", 2024-04-19T01:04:09.349 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.7951000000000001e-05 2024-04-19T01:04:09.349 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.349 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.349 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:09.349 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.388899+0000", 2024-04-19T01:04:09.349 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.8552000000000001e-05 2024-04-19T01:04:09.349 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.349 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.349 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:09.349 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.389565+0000", 2024-04-19T01:04:09.349 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00066563699999999996 2024-04-19T01:04:09.349 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.349 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.349 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:09.349 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.389605+0000", 2024-04-19T01:04:09.350 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.0302999999999997e-05 2024-04-19T01:04:09.350 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.350 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.350 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:09.350 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.392888+0000", 2024-04-19T01:04:09.350 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0032825250000000001 2024-04-19T01:04:09.350 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.350 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.350 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:09.350 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.392936+0000", 2024-04-19T01:04:09.350 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.8612999999999997e-05 2024-04-19T01:04:09.350 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.350 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:09.350 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.351 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.351 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.351 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_repop(client.4260.0:192 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'142, mlcod=24'138)", 2024-04-19T01:04:09.351 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:03:51.389547+0000", 2024-04-19T01:04:09.351 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 17.878138288999999, 2024-04-19T01:04:09.351 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0049295049999999998, 2024-04-19T01:04:09.351 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:09.351 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:09.351 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:09.351 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:09.351 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.351 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:09.351 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.389547+0000", 2024-04-19T01:04:09.351 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.351 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.352 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.352 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:09.352 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.389547+0000", 2024-04-19T01:04:09.352 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.352 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.352 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.352 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:09.352 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.389549+0000", 2024-04-19T01:04:09.352 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.283e-06 2024-04-19T01:04:09.352 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.352 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.352 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:09.352 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.389617+0000", 2024-04-19T01:04:09.352 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.8066999999999997e-05 2024-04-19T01:04:09.353 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.353 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.353 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:09.353 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.389637+0000", 2024-04-19T01:04:09.353 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0214999999999999e-05 2024-04-19T01:04:09.353 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.353 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.353 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:09.353 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.389647+0000", 2024-04-19T01:04:09.353 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.0271999999999999e-05 2024-04-19T01:04:09.353 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.353 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.353 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:09.354 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.391063+0000", 2024-04-19T01:04:09.354 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00141636 2024-04-19T01:04:09.354 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.354 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.354 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:09.354 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.391095+0000", 2024-04-19T01:04:09.354 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.1158000000000002e-05 2024-04-19T01:04:09.354 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.354 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.354 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:09.354 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.394409+0000", 2024-04-19T01:04:09.354 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0033140859999999999 2024-04-19T01:04:09.354 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.354 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.355 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:09.355 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.394477+0000", 2024-04-19T01:04:09.355 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.8064000000000004e-05 2024-04-19T01:04:09.355 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.355 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:09.355 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.355 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.355 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.355 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_repop(client.4260.0:193 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'143, mlcod=24'138)", 2024-04-19T01:04:09.355 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:03:51.390630+0000", 2024-04-19T01:04:09.355 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 17.877055488, 2024-04-19T01:04:09.355 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0045361359999999996, 2024-04-19T01:04:09.355 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:09.355 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:09.356 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:09.356 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:09.356 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.356 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:09.356 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.390630+0000", 2024-04-19T01:04:09.356 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.356 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.356 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.356 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:09.356 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.390630+0000", 2024-04-19T01:04:09.356 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.356 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.356 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.356 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:09.357 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.390632+0000", 2024-04-19T01:04:09.357 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9870000000000002e-06 2024-04-19T01:04:09.357 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.357 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.357 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:09.357 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.390690+0000", 2024-04-19T01:04:09.357 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.8445e-05 2024-04-19T01:04:09.357 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.357 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.357 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:09.357 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.390714+0000", 2024-04-19T01:04:09.357 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3084000000000001e-05 2024-04-19T01:04:09.357 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.357 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.357 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:09.358 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.390730+0000", 2024-04-19T01:04:09.358 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6365e-05 2024-04-19T01:04:09.358 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.358 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.358 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:09.358 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.393039+0000", 2024-04-19T01:04:09.358 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0023088119999999999 2024-04-19T01:04:09.358 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.358 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.358 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:09.358 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.393074+0000", 2024-04-19T01:04:09.358 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.5327000000000002e-05 2024-04-19T01:04:09.358 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.358 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.359 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:09.359 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.395074+0000", 2024-04-19T01:04:09.359 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0019999140000000002 2024-04-19T01:04:09.359 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.359 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.359 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:09.359 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:03:51.395166+0000", 2024-04-19T01:04:09.359 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.2201999999999996e-05 2024-04-19T01:04:09.359 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.359 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:09.359 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.359 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.359 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:09.359 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:09.361 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-19T01:04:09.362 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:04:09.359+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:04:09.462 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:04:09.461+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:04:09.520 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-19T01:04:09.528 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:09.528 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [], 2024-04-19T01:04:09.528 INFO:teuthology.orchestra.run.smithi110.stdout: "num_ops": 0 2024-04-19T01:04:09.528 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:09.543 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-19T01:04:09.563 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:04:09.561+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:04:09.659 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:09.659 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [], 2024-04-19T01:04:09.659 INFO:teuthology.orchestra.run.smithi110.stdout: "complaint_time": 30, 2024-04-19T01:04:09.659 INFO:teuthology.orchestra.run.smithi110.stdout: "num_blocked_ops": 0 2024-04-19T01:04:09.659 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:09.664 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:04:09.665+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:04:09.673 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-04-19T01:04:09.673 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T01:04:09.674 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-19T01:04:09.691 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-04-19T01:04:09.691 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T01:04:09.714 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-04-19T01:04:09.715 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T01:04:09.733 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-04-19T01:04:09.733 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T01:04:09.753 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-04-19T01:04:09.753 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T01:04:09.765 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:04:09.764+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:04:09.772 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-04-19T01:04:09.773 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T01:04:09.794 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.3 to 1 (10000) 2024-04-19T01:04:09.796 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-04-19T01:04:09.796 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T01:04:09.800 INFO:tasks.rados.rados.0.smithi155.stdout:60: done (4 left) 2024-04-19T01:04:09.800 INFO:tasks.rados.rados.0.smithi155.stdout:61: done (3 left) 2024-04-19T01:04:09.800 INFO:tasks.rados.rados.0.smithi155.stdout:62: done (2 left) 2024-04-19T01:04:09.800 INFO:tasks.rados.rados.0.smithi155.stdout:63: done (1 left) 2024-04-19T01:04:09.800 INFO:tasks.rados.rados.0.smithi155.stdout:64: done (0 left) 2024-04-19T01:04:09.801 INFO:tasks.rados.rados.0.smithi155.stdout:65: read oid 29 snap -1 2024-04-19T01:04:09.801 INFO:tasks.rados.rados.0.smithi155.stdout:65: expect (ObjNum 28 snap 0 seq_num 28) 2024-04-19T01:04:09.801 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 34 current snap is 5 2024-04-19T01:04:09.801 INFO:tasks.rados.rados.0.smithi155.stdout:66: seq_num 53 ranges {557056=524288,1081344=376832} 2024-04-19T01:04:09.807 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:09.807 INFO:teuthology.orchestra.run.smithi110.stdout: "size": 20, 2024-04-19T01:04:09.807 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 600, 2024-04-19T01:04:09.807 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [ 2024-04-19T01:04:09.807 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.807 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 38/31 ECSubWrite(tid=30, reqid=client.4604.0:168, at_version=38'14, trim_to=0'0, roll_forward_to=34'12))", 2024-04-19T01:04:09.807 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:08.807499+0000", 2024-04-19T01:04:09.807 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 0.99975878699999998, 2024-04-19T01:04:09.807 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0053266590000000001, 2024-04-19T01:04:09.807 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:09.808 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:09.808 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:09.808 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:09.808 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.808 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:09.808 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.807499+0000", 2024-04-19T01:04:09.808 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.808 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.808 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.808 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:09.808 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.807499+0000", 2024-04-19T01:04:09.808 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.808 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.808 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.809 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:09.809 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.807500+0000", 2024-04-19T01:04:09.809 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.5300000000000004e-07 2024-04-19T01:04:09.809 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.809 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.809 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:09.809 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.807654+0000", 2024-04-19T01:04:09.809 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015412600000000001 2024-04-19T01:04:09.809 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.809 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.809 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:09.809 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.807701+0000", 2024-04-19T01:04:09.809 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.6903999999999997e-05 2024-04-19T01:04:09.809 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.809 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.810 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:09.810 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.807728+0000", 2024-04-19T01:04:09.810 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.7382e-05 2024-04-19T01:04:09.810 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.810 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.810 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:09.810 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.807907+0000", 2024-04-19T01:04:09.810 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00017864399999999999 2024-04-19T01:04:09.810 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.810 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.810 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:09.810 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.807931+0000", 2024-04-19T01:04:09.810 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4212000000000001e-05 2024-04-19T01:04:09.810 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.811 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.811 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:09.811 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.812742+0000", 2024-04-19T01:04:09.811 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0048114459999999996 2024-04-19T01:04:09.811 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.811 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.811 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:09.812 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.812826+0000", 2024-04-19T01:04:09.812 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.3392000000000004e-05 2024-04-19T01:04:09.812 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.812 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:09.812 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.812 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.812 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.812 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.0s1 38/31 ECSubWrite(tid=11, reqid=client.4604.0:169, at_version=38'8, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:04:09.812 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:08.808715+0000", 2024-04-19T01:04:09.812 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 0.99854310800000001, 2024-04-19T01:04:09.812 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.004027821, 2024-04-19T01:04:09.812 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:09.812 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:09.812 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:09.812 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:09.813 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.813 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:09.813 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.808715+0000", 2024-04-19T01:04:09.813 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.813 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.813 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.813 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:09.813 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.808715+0000", 2024-04-19T01:04:09.813 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.813 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.813 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.813 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:09.813 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.808715+0000", 2024-04-19T01:04:09.813 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.2399999999999998e-07 2024-04-19T01:04:09.814 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.814 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.814 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:09.814 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.808795+0000", 2024-04-19T01:04:09.814 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.9392000000000002e-05 2024-04-19T01:04:09.814 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.814 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.814 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:09.814 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.808825+0000", 2024-04-19T01:04:09.814 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.0477999999999999e-05 2024-04-19T01:04:09.814 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.814 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.814 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:09.814 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.808847+0000", 2024-04-19T01:04:09.814 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2062999999999999e-05 2024-04-19T01:04:09.815 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.815 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.815 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:09.815 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.808954+0000", 2024-04-19T01:04:09.815 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000107123 2024-04-19T01:04:09.815 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.815 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.815 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:09.815 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.808965+0000", 2024-04-19T01:04:09.815 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.0458999999999999e-05 2024-04-19T01:04:09.815 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.816 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.816 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:09.816 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.812698+0000", 2024-04-19T01:04:09.816 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0037330419999999998 2024-04-19T01:04:09.816 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.816 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.816 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:09.816 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.812743+0000", 2024-04-19T01:04:09.816 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.4740000000000002e-05 2024-04-19T01:04:09.816 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.816 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:09.816 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.816 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.816 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.817 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.0s1 38/31 ECSubWrite(tid=12, reqid=client.4604.0:170, at_version=38'9, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:04:09.817 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:08.811091+0000", 2024-04-19T01:04:09.817 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 0.99616721399999997, 2024-04-19T01:04:09.817 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.003492364, 2024-04-19T01:04:09.817 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:09.817 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:09.817 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:09.817 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:09.817 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.817 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:09.817 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.811091+0000", 2024-04-19T01:04:09.817 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.817 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.817 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.817 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:09.818 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.811091+0000", 2024-04-19T01:04:09.818 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.818 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.818 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.818 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:09.818 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.811092+0000", 2024-04-19T01:04:09.818 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.0839999999999999e-06 2024-04-19T01:04:09.818 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.818 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.818 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:09.818 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.811112+0000", 2024-04-19T01:04:09.818 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0285e-05 2024-04-19T01:04:09.818 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.819 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.819 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:09.819 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.811125+0000", 2024-04-19T01:04:09.819 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.2951e-05 2024-04-19T01:04:09.819 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.819 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.819 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:09.819 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.811134+0000", 2024-04-19T01:04:09.819 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.3570000000000004e-06 2024-04-19T01:04:09.819 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.819 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.819 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:09.819 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.811226+0000", 2024-04-19T01:04:09.819 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.1768999999999995e-05 2024-04-19T01:04:09.819 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.820 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.820 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:09.820 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.811235+0000", 2024-04-19T01:04:09.820 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.166e-06 2024-04-19T01:04:09.820 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.820 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.820 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:09.820 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.814509+0000", 2024-04-19T01:04:09.820 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0032738289999999998 2024-04-19T01:04:09.821 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.821 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.821 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:09.821 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.814583+0000", 2024-04-19T01:04:09.821 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.3923000000000004e-05 2024-04-19T01:04:09.821 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.821 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:09.821 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.821 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.821 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.821 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.0s1 38/31 ECSubWrite(tid=13, reqid=client.4604.0:171, at_version=38'10, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:04:09.821 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:08.813792+0000", 2024-04-19T01:04:09.821 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 0.99346641800000002, 2024-04-19T01:04:09.821 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0013385859999999999, 2024-04-19T01:04:09.822 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:09.822 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:09.822 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:09.822 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:09.822 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.822 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:09.822 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.813792+0000", 2024-04-19T01:04:09.823 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.823 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.823 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.823 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:09.823 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.813792+0000", 2024-04-19T01:04:09.823 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.823 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.823 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.823 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:09.823 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.813792+0000", 2024-04-19T01:04:09.823 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.3099999999999997e-07 2024-04-19T01:04:09.823 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.823 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.823 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:09.824 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.813811+0000", 2024-04-19T01:04:09.824 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.8825999999999999e-05 2024-04-19T01:04:09.824 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.824 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.824 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:09.824 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.813827+0000", 2024-04-19T01:04:09.824 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5534000000000001e-05 2024-04-19T01:04:09.824 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.824 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.824 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:09.824 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.813836+0000", 2024-04-19T01:04:09.824 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.4180000000000003e-06 2024-04-19T01:04:09.824 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.824 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.825 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:09.825 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.813930+0000", 2024-04-19T01:04:09.825 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.4151e-05 2024-04-19T01:04:09.825 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.825 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.825 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:09.825 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.813953+0000", 2024-04-19T01:04:09.825 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3254000000000001e-05 2024-04-19T01:04:09.825 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.825 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.825 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:09.825 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.815048+0000", 2024-04-19T01:04:09.826 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001094069 2024-04-19T01:04:09.826 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.826 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.826 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:09.826 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.815130+0000", 2024-04-19T01:04:09.826 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.2602999999999995e-05 2024-04-19T01:04:09.826 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.826 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:09.826 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.826 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.826 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.826 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.cs4 38/31 ECSubWrite(tid=32, reqid=client.4604.0:172, at_version=38'15, trim_to=0'0, roll_forward_to=35'13))", 2024-04-19T01:04:09.826 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:08.813847+0000", 2024-04-19T01:04:09.827 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 0.99341106800000001, 2024-04-19T01:04:09.827 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0028621979999999998, 2024-04-19T01:04:09.827 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:09.827 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:09.827 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:09.827 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:09.827 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.827 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:09.827 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.813847+0000", 2024-04-19T01:04:09.827 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.827 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.827 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.827 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:09.827 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.813847+0000", 2024-04-19T01:04:09.828 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.828 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.828 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.828 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:09.828 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.813848+0000", 2024-04-19T01:04:09.828 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.4699999999999999e-06 2024-04-19T01:04:09.828 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.828 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.828 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:09.828 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.813948+0000", 2024-04-19T01:04:09.828 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.9730999999999998e-05 2024-04-19T01:04:09.829 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.829 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.829 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:09.829 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.813980+0000", 2024-04-19T01:04:09.829 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.1748999999999997e-05 2024-04-19T01:04:09.829 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.829 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.829 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:09.829 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.814008+0000", 2024-04-19T01:04:09.829 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.8061000000000001e-05 2024-04-19T01:04:09.829 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.829 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.829 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:09.829 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.814164+0000", 2024-04-19T01:04:09.829 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015618199999999999 2024-04-19T01:04:09.830 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.830 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.830 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:09.830 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.814189+0000", 2024-04-19T01:04:09.830 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5265e-05 2024-04-19T01:04:09.830 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.830 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.830 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:09.830 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.816655+0000", 2024-04-19T01:04:09.830 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0024654329999999999 2024-04-19T01:04:09.830 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.830 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.830 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:09.830 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.816709+0000", 2024-04-19T01:04:09.831 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.4307e-05 2024-04-19T01:04:09.831 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.831 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:09.831 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.831 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.831 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.831 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 38/31 ECSubWrite(tid=31, reqid=client.4604.0:168, at_version=0'0, trim_to=0'0, roll_forward_to=38'14))", 2024-04-19T01:04:09.831 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:08.815790+0000", 2024-04-19T01:04:09.831 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 0.99146769700000004, 2024-04-19T01:04:09.831 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001176194, 2024-04-19T01:04:09.831 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:09.831 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:09.831 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:09.831 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:09.831 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.832 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:09.832 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.815790+0000", 2024-04-19T01:04:09.832 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.832 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.832 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.832 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:09.832 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.815790+0000", 2024-04-19T01:04:09.832 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.833 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.833 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.833 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:09.833 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.815792+0000", 2024-04-19T01:04:09.833 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7999999999999999e-06 2024-04-19T01:04:09.833 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.833 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.833 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:09.833 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.815815+0000", 2024-04-19T01:04:09.833 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2467999999999999e-05 2024-04-19T01:04:09.833 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.834 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.834 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:09.834 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.815841+0000", 2024-04-19T01:04:09.834 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6628999999999999e-05 2024-04-19T01:04:09.834 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.834 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.834 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:09.834 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.815857+0000", 2024-04-19T01:04:09.834 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5372000000000001e-05 2024-04-19T01:04:09.834 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.834 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.834 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:09.834 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.815988+0000", 2024-04-19T01:04:09.834 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00013138000000000001 2024-04-19T01:04:09.834 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.835 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.835 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:09.835 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.816012+0000", 2024-04-19T01:04:09.835 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4542e-05 2024-04-19T01:04:09.835 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.835 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.835 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:09.835 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.816931+0000", 2024-04-19T01:04:09.835 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00091875400000000003 2024-04-19T01:04:09.835 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.835 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.835 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:09.835 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.816966+0000", 2024-04-19T01:04:09.835 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.5249000000000001e-05 2024-04-19T01:04:09.835 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.836 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:09.836 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.836 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.836 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.836 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.cs4 38/31 ECSubWrite(tid=33, reqid=client.4604.0:172, at_version=0'0, trim_to=0'0, roll_forward_to=38'15))", 2024-04-19T01:04:09.836 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:08.819610+0000", 2024-04-19T01:04:09.836 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 0.98764787499999995, 2024-04-19T01:04:09.836 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0012618429999999999, 2024-04-19T01:04:09.836 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:09.836 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:09.836 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:09.836 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:09.837 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.837 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:09.837 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.819610+0000", 2024-04-19T01:04:09.837 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.837 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.837 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.837 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:09.837 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.819610+0000", 2024-04-19T01:04:09.837 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.837 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.837 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.837 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:09.837 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.819612+0000", 2024-04-19T01:04:09.838 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.793e-06 2024-04-19T01:04:09.838 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.838 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.838 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:09.838 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.819637+0000", 2024-04-19T01:04:09.838 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5545e-05 2024-04-19T01:04:09.838 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.838 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.838 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:09.838 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.819676+0000", 2024-04-19T01:04:09.838 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.8695000000000001e-05 2024-04-19T01:04:09.838 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.838 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.839 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:09.839 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.819701+0000", 2024-04-19T01:04:09.839 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4664999999999998e-05 2024-04-19T01:04:09.839 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.839 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.839 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:09.839 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.819834+0000", 2024-04-19T01:04:09.839 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00013356699999999999 2024-04-19T01:04:09.839 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.839 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.839 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:09.839 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.819851+0000", 2024-04-19T01:04:09.839 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6962000000000001e-05 2024-04-19T01:04:09.839 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.839 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.840 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:09.840 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.820780+0000", 2024-04-19T01:04:09.840 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00092820100000000003 2024-04-19T01:04:09.840 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.840 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.840 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:09.840 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.820872+0000", 2024-04-19T01:04:09.840 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.2415000000000002e-05 2024-04-19T01:04:09.840 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.840 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:09.840 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.840 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.840 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.840 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.0s1 38/31 ECSubWrite(tid=14, reqid=client.4604.0:171, at_version=0'0, trim_to=0'0, roll_forward_to=38'10))", 2024-04-19T01:04:09.840 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:08.821136+0000", 2024-04-19T01:04:09.841 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 0.986122426, 2024-04-19T01:04:09.841 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0011498770000000001, 2024-04-19T01:04:09.841 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:09.841 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:09.841 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:09.841 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:09.841 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.841 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:09.841 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.821136+0000", 2024-04-19T01:04:09.841 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.841 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.841 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.841 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:09.841 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.821136+0000", 2024-04-19T01:04:09.842 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:09.842 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.842 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.842 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:09.842 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.821138+0000", 2024-04-19T01:04:09.842 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1330000000000002e-06 2024-04-19T01:04:09.842 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.842 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.842 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:09.842 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.821166+0000", 2024-04-19T01:04:09.842 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.8169000000000001e-05 2024-04-19T01:04:09.843 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.843 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.843 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:09.843 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.821195+0000", 2024-04-19T01:04:09.843 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.8702000000000001e-05 2024-04-19T01:04:09.843 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.843 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.843 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:09.843 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.821217+0000", 2024-04-19T01:04:09.843 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2282000000000001e-05 2024-04-19T01:04:09.844 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.844 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.844 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:09.844 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.821369+0000", 2024-04-19T01:04:09.844 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015251199999999999 2024-04-19T01:04:09.844 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.844 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.844 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:09.844 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.821397+0000", 2024-04-19T01:04:09.844 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.7719e-05 2024-04-19T01:04:09.844 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.844 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.844 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:09.844 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.822200+0000", 2024-04-19T01:04:09.844 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00080273699999999998 2024-04-19T01:04:09.845 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:09.845 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:09.845 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:09.845 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.822285+0000", 2024-04-19T01:04:09.845 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.5623000000000003e-05 2024-04-19T01:04:09.845 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.845 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:09.845 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.845 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:09.845 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:09.845 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:09.847 INFO:tasks.rados.rados.0.smithi155.stdout:66: writing smithi15528762-34 from 557056 to 1081344 tid 1 2024-04-19T01:04:09.847 INFO:tasks.rados.rados.0.smithi155.stdout:66: writing smithi15528762-34 from 1081344 to 1458176 tid 2 2024-04-19T01:04:09.847 INFO:tasks.rados.rados.0.smithi155.stdout:67: copy_from oid 39 from oid 9 current snap is 5 2024-04-19T01:04:09.847 INFO:tasks.rados.rados.0.smithi155.stdout:65: expect (ObjNum 28 snap 0 seq_num 28) 2024-04-19T01:04:09.848 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-04-19T01:04:09.848 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-04-19T01:04:09.849 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-04-19T01:04:09.849 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T01:04:09.849 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2024-04-19T01:04:09.849 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T01:04:09.866 INFO:tasks.rados.rados.0.smithi155.stdout:66: finishing write tid 1 to smithi15528762-34 2024-04-19T01:04:09.866 INFO:tasks.rados.rados.0.smithi155.stdout:66: finishing write tid 2 to smithi15528762-34 2024-04-19T01:04:09.866 INFO:tasks.rados.rados.0.smithi155.stdout:66: finishing write tid 3 to smithi15528762-34 2024-04-19T01:04:09.866 INFO:tasks.rados.rados.0.smithi155.stdout:66: finishing write tid 4 to smithi15528762-34 2024-04-19T01:04:09.866 INFO:tasks.rados.rados.0.smithi155.stdout:66: oid 34 updating version 0 to 18 2024-04-19T01:04:09.866 INFO:tasks.rados.rados.0.smithi155.stdout:66: oid 34 version 18 is already newer than 16 2024-04-19T01:04:09.866 INFO:tasks.rados.rados.0.smithi155.stdout:66: oid 34 version 18 is already newer than 17 2024-04-19T01:04:09.866 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 34 v 18 (ObjNum 53 snap 5 seq_num 53) dirty exists 2024-04-19T01:04:09.866 INFO:tasks.rados.rados.0.smithi155.stdout:66: left oid 34 (ObjNum 53 snap 5 seq_num 53) 2024-04-19T01:04:09.867 INFO:tasks.rados.rados.0.smithi155.stdout:67: finishing copy_from racing read to smithi15528762-39 2024-04-19T01:04:09.867 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:04:09.865+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:04:09.867 INFO:tasks.rados.rados.0.smithi155.stdout:67: finishing copy_from to smithi15528762-39 2024-04-19T01:04:09.867 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 39 v 2 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-04-19T01:04:09.867 INFO:tasks.rados.rados.0.smithi155.stdout:65: done (2 left) 2024-04-19T01:04:09.867 INFO:tasks.rados.rados.0.smithi155.stdout:66: done (1 left) 2024-04-19T01:04:09.868 INFO:tasks.rados.rados.0.smithi155.stdout:67: done (0 left) 2024-04-19T01:04:09.868 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 43 current snap is 5 2024-04-19T01:04:09.868 INFO:tasks.rados.rados.0.smithi155.stdout:68: seq_num 54 ranges {720896=540672,1261568=376832} 2024-04-19T01:04:09.868 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2024-04-19T01:04:09.868 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T01:04:09.883 INFO:tasks.rados.rados.0.smithi155.stdout:68: writing smithi15528762-43 from 720896 to 1261568 tid 1 2024-04-19T01:04:09.890 INFO:tasks.rados.rados.0.smithi155.stdout:68: writing smithi15528762-43 from 1261568 to 1638400 tid 2 2024-04-19T01:04:09.890 INFO:tasks.rados.rados.0.smithi155.stdout:69: snap_remove snap 1 2024-04-19T01:04:09.891 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2024-04-19T01:04:09.892 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T01:04:09.914 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2024-04-19T01:04:09.914 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T01:04:09.932 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2024-04-19T01:04:09.932 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-19T01:04:09.951 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2024-04-19T01:04:09.952 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-19T01:04:09.966 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:04:09.964+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:04:09.973 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:09.973 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:09.973 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2024-04-19T01:04:09.973 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-19T01:04:09.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:04:09.972+0000 7f7bab59e640 0 -- 172.21.15.33:0/2619187789 >> [v2:172.21.15.155:6807/741132602,v1:172.21.15.155:6810/741132602] conn(0x7f7bac2247f0 msgr2=0x7f7bac226be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-19T01:04:09.987 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-04-19T01:04:10.067 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:04:10.065+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:04:10.102 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:10.102 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:10.115 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-04-19T01:04:10.168 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:04:10.164+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:04:10.199 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2024-04-19T01:04:10.200 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-19T01:04:10.236 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:10.236 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:10.250 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-19T01:04:10.270 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:04:10.268+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:04:10.373 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:04:10.372+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:04:10.477 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:10.477 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:10.478 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-19T01:04:10.478 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:04:10.476+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:04:10.578 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:04:10.579+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:04:10.643 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:10.644 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:10.657 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-19T01:04:10.679 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:04:10.679+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:04:10.783 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:04:10.779+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:04:10.783 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:10.783 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:10.798 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-19T01:04:10.804 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.4 to 1 (10000) 2024-04-19T01:04:10.808 INFO:tasks.rados.rados.0.smithi155.stdout:69: done (1 left) 2024-04-19T01:04:10.808 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 40 current snap is 5 2024-04-19T01:04:10.808 INFO:tasks.rados.rados.0.smithi155.stdout:68: finishing write tid 1 to smithi15528762-43 2024-04-19T01:04:10.808 INFO:tasks.rados.rados.0.smithi155.stdout:68: finishing write tid 2 to smithi15528762-43 2024-04-19T01:04:10.808 INFO:tasks.rados.rados.0.smithi155.stdout:68: finishing write tid 3 to smithi15528762-43 2024-04-19T01:04:10.808 INFO:tasks.rados.rados.0.smithi155.stdout:68: finishing write tid 4 to smithi15528762-43 2024-04-19T01:04:10.808 INFO:tasks.rados.rados.0.smithi155.stdout:68: oid 43 updating version 0 to 22 2024-04-19T01:04:10.808 INFO:tasks.rados.rados.0.smithi155.stdout:68: oid 43 version 22 is already newer than 20 2024-04-19T01:04:10.808 INFO:tasks.rados.rados.0.smithi155.stdout:68: oid 43 version 22 is already newer than 21 2024-04-19T01:04:10.808 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 43 v 22 (ObjNum 54 snap 5 seq_num 54) dirty exists 2024-04-19T01:04:10.809 INFO:tasks.rados.rados.0.smithi155.stdout:68: left oid 43 (ObjNum 54 snap 5 seq_num 54) 2024-04-19T01:04:10.809 INFO:tasks.rados.rados.0.smithi155.stdout:70: seq_num 55 ranges {671744=557056,1228800=376832} 2024-04-19T01:04:10.823 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-19T01:04:10.828 INFO:tasks.rados.rados.0.smithi155.stdout:70: writing smithi15528762-40 from 671744 to 1228800 tid 1 2024-04-19T01:04:10.830 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-04-19T01:04:10.835 INFO:tasks.rados.rados.0.smithi155.stdout:70: writing smithi15528762-40 from 1228800 to 1605632 tid 2 2024-04-19T01:04:10.836 INFO:tasks.rados.rados.0.smithi155.stdout:68: done (1 left) 2024-04-19T01:04:10.836 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 19 current snap is 5 2024-04-19T01:04:10.836 INFO:tasks.rados.rados.0.smithi155.stdout:71: seq_num 56 ranges {311296=573440,884736=376832} 2024-04-19T01:04:10.852 INFO:tasks.rados.rados.0.smithi155.stdout:71: writing smithi15528762-19 from 311296 to 884736 tid 1 2024-04-19T01:04:10.859 INFO:tasks.rados.rados.0.smithi155.stdout:71: writing smithi15528762-19 from 884736 to 1261568 tid 2 2024-04-19T01:04:10.859 INFO:tasks.rados.rados.0.smithi155.stdout:72: read oid 22 snap 0 2024-04-19T01:04:10.859 INFO:tasks.rados.rados.0.smithi155.stdout:72: expect (ObjNum 21 snap 0 seq_num 21) 2024-04-19T01:04:10.859 INFO:tasks.rados.rados.0.smithi155.stdout:73: delete oid 4 current snap is 5 2024-04-19T01:04:10.859 INFO:tasks.rados.rados.0.smithi155.stdout:70: finishing write tid 1 to smithi15528762-40 2024-04-19T01:04:10.859 INFO:tasks.rados.rados.0.smithi155.stdout:70: finishing write tid 2 to smithi15528762-40 2024-04-19T01:04:10.859 INFO:tasks.rados.rados.0.smithi155.stdout:70: finishing write tid 3 to smithi15528762-40 2024-04-19T01:04:10.861 INFO:tasks.rados.rados.0.smithi155.stdout:70: finishing write tid 4 to smithi15528762-40 2024-04-19T01:04:10.861 INFO:tasks.rados.rados.0.smithi155.stdout:70: oid 40 updating version 0 to 9 2024-04-19T01:04:10.861 INFO:tasks.rados.rados.0.smithi155.stdout:70: oid 40 updating version 9 to 10 2024-04-19T01:04:10.861 INFO:tasks.rados.rados.0.smithi155.stdout:70: oid 40 version 10 is already newer than 8 2024-04-19T01:04:10.861 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 40 v 10 (ObjNum 55 snap 5 seq_num 55) dirty exists 2024-04-19T01:04:10.861 INFO:tasks.rados.rados.0.smithi155.stdout:70: left oid 40 (ObjNum 55 snap 5 seq_num 55) 2024-04-19T01:04:10.867 INFO:tasks.rados.rados.0.smithi155.stdout:72: expect (ObjNum 21 snap 0 seq_num 21) 2024-04-19T01:04:10.881 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:10.880+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:10.898 INFO:tasks.rados.rados.0.smithi155.stdout:71: finishing write tid 1 to smithi15528762-19 2024-04-19T01:04:10.898 INFO:tasks.rados.rados.0.smithi155.stdout:71: finishing write tid 2 to smithi15528762-19 2024-04-19T01:04:10.898 INFO:tasks.rados.rados.0.smithi155.stdout:71: finishing write tid 3 to smithi15528762-19 2024-04-19T01:04:10.898 INFO:tasks.rados.rados.0.smithi155.stdout:70: done (3 left) 2024-04-19T01:04:10.898 INFO:tasks.rados.rados.0.smithi155.stdout:72: done (2 left) 2024-04-19T01:04:10.898 INFO:tasks.rados.rados.0.smithi155.stdout:73: done (1 left) 2024-04-19T01:04:10.898 INFO:tasks.rados.rados.0.smithi155.stdout:74: delete oid 1 current snap is 5 2024-04-19T01:04:10.899 INFO:tasks.rados.rados.0.smithi155.stdout:71: finishing write tid 4 to smithi15528762-19 2024-04-19T01:04:10.899 INFO:tasks.rados.rados.0.smithi155.stdout:71: oid 19 updating version 0 to 8 2024-04-19T01:04:10.899 INFO:tasks.rados.rados.0.smithi155.stdout:71: oid 19 version 8 is already newer than 6 2024-04-19T01:04:10.899 INFO:tasks.rados.rados.0.smithi155.stdout:71: oid 19 version 8 is already newer than 7 2024-04-19T01:04:10.899 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 19 v 8 (ObjNum 56 snap 5 seq_num 56) dirty exists 2024-04-19T01:04:10.899 INFO:tasks.rados.rados.0.smithi155.stdout:71: left oid 19 (ObjNum 56 snap 5 seq_num 56) 2024-04-19T01:04:10.901 INFO:tasks.rados.rados.0.smithi155.stdout:71: done (1 left) 2024-04-19T01:04:10.901 INFO:tasks.rados.rados.0.smithi155.stdout:74: done (0 left) 2024-04-19T01:04:10.901 INFO:tasks.rados.rados.0.smithi155.stdout:75: rollback oid 1 current snap is 5 2024-04-19T01:04:10.901 INFO:tasks.rados.rados.0.smithi155.stdout:rollback oid 1 to 0 2024-04-19T01:04:10.901 INFO:tasks.rados.rados.0.smithi155.stdout:76: snap_remove snap 3 2024-04-19T01:04:10.929 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:10.929 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:10.943 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-19T01:04:10.982 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:04:10.983+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:04:11.083 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:04:11.084+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:04:11.109 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:11.110 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:11.124 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-19T01:04:11.185 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:04:11.185+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:04:11.286 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:04:11.288+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:04:11.288 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:11.288 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:11.303 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-19T01:04:11.387 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:04:11.385+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:04:11.470 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:11.471 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:11.484 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-19T01:04:11.488 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:04:11.487+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:04:11.589 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:04:11.587+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:04:11.600 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:11.600 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:11.615 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-19T01:04:11.690 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:04:11.691+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:04:11.732 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:11.732 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:11.746 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-04-19T01:04:11.791 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:04:11.789+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:04:11.814 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2024-04-19T01:04:11.814 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.5 to 1 (10000) 2024-04-19T01:04:11.826 INFO:tasks.rados.rados.0.smithi155.stdout:76: done (1 left) 2024-04-19T01:04:11.826 INFO:tasks.rados.rados.0.smithi155.stdout:77: read oid 46 snap -1 2024-04-19T01:04:11.826 INFO:tasks.rados.rados.0.smithi155.stdout:77: expect (ObjNum 45 snap 0 seq_num 45) 2024-04-19T01:04:11.826 INFO:tasks.rados.rados.0.smithi155.stdout:78: rollback oid 11 current snap is 5 2024-04-19T01:04:11.826 INFO:tasks.rados.rados.0.smithi155.stdout:rollback oid 11 to 2 2024-04-19T01:04:11.826 INFO:tasks.rados.rados.0.smithi155.stdout:75: finishing rollback tid 1 to smithi15528762-1 2024-04-19T01:04:11.826 INFO:tasks.rados.rados.0.smithi155.stdout:75: finishing rollback tid 2 to smithi15528762-1 2024-04-19T01:04:11.826 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 1 v 8 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-19T01:04:11.826 INFO:tasks.rados.rados.0.smithi155.stdout:75: done (2 left) 2024-04-19T01:04:11.827 INFO:tasks.rados.rados.0.smithi155.stdout:79: rmattr oid 18 current snap is 5 2024-04-19T01:04:11.827 INFO:tasks.rados.rados.0.smithi155.stdout:79: done (2 left) 2024-04-19T01:04:11.827 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 31 current snap is 5 2024-04-19T01:04:11.827 INFO:tasks.rados.rados.0.smithi155.stdout:80: seq_num 59 ranges {1392640=622592,2015232=376832} 2024-04-19T01:04:11.854 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-04-19T01:04:11.862 INFO:tasks.rados.rados.0.smithi155.stdout:80: writing smithi15528762-31 from 1392640 to 2015232 tid 1 2024-04-19T01:04:11.869 INFO:tasks.rados.rados.0.smithi155.stdout:80: writing smithi15528762-31 from 2015232 to 2392064 tid 2 2024-04-19T01:04:11.869 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 35 current snap is 5 2024-04-19T01:04:11.869 INFO:tasks.rados.rados.0.smithi155.stdout:81: seq_num 60 ranges {573440=638976,1212416=376832} 2024-04-19T01:04:11.874 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:11.874 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:11.886 INFO:tasks.rados.rados.0.smithi155.stdout:81: writing smithi15528762-35 from 573440 to 1212416 tid 1 2024-04-19T01:04:11.888 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-04-19T01:04:11.891 INFO:tasks.rados.rados.0.smithi155.stdout:81: writing smithi15528762-35 from 1212416 to 1589248 tid 2 2024-04-19T01:04:11.891 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 43 current snap is 5 2024-04-19T01:04:11.891 INFO:tasks.rados.rados.0.smithi155.stdout:82: seq_num 61 ranges {1638400=655360,2293760=376832} 2024-04-19T01:04:11.898 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:04:11.895+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:04:11.913 INFO:tasks.rados.rados.0.smithi155.stdout:82: writing smithi15528762-43 from 1638400 to 2293760 tid 1 2024-04-19T01:04:11.916 INFO:tasks.rados.rados.0.smithi155.stdout:82: writing smithi15528762-43 from 2293760 to 2670592 tid 2 2024-04-19T01:04:11.916 INFO:tasks.rados.rados.0.smithi155.stdout:83: snap_create 2024-04-19T01:04:11.916 INFO:tasks.rados.rados.0.smithi155.stdout:77: expect (ObjNum 45 snap 0 seq_num 45) 2024-04-19T01:04:11.971 INFO:tasks.rados.rados.0.smithi155.stdout:78: finishing rollback tid 0 to smithi15528762-11 2024-04-19T01:04:11.972 INFO:tasks.rados.rados.0.smithi155.stdout:78: finishing rollback tid 1 to smithi15528762-11 2024-04-19T01:04:11.972 INFO:tasks.rados.rados.0.smithi155.stdout:78: finishing rollback tid 2 to smithi15528762-11 2024-04-19T01:04:11.972 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 11 v 8 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-04-19T01:04:11.972 INFO:tasks.rados.rados.0.smithi155.stdout:80: finishing write tid 1 to smithi15528762-31 2024-04-19T01:04:11.972 INFO:tasks.rados.rados.0.smithi155.stdout:80: finishing write tid 2 to smithi15528762-31 2024-04-19T01:04:11.972 INFO:tasks.rados.rados.0.smithi155.stdout:80: finishing write tid 3 to smithi15528762-31 2024-04-19T01:04:11.972 INFO:tasks.rados.rados.0.smithi155.stdout:80: finishing write tid 4 to smithi15528762-31 2024-04-19T01:04:11.972 INFO:tasks.rados.rados.0.smithi155.stdout:80: oid 31 updating version 0 to 18 2024-04-19T01:04:11.972 INFO:tasks.rados.rados.0.smithi155.stdout:80: oid 31 version 18 is already newer than 17 2024-04-19T01:04:11.972 INFO:tasks.rados.rados.0.smithi155.stdout:80: oid 31 updating version 18 to 19 2024-04-19T01:04:11.972 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 31 v 19 (ObjNum 59 snap 5 seq_num 59) dirty exists 2024-04-19T01:04:11.973 INFO:tasks.rados.rados.0.smithi155.stdout:80: left oid 31 (ObjNum 59 snap 5 seq_num 59) 2024-04-19T01:04:11.973 INFO:tasks.rados.rados.0.smithi155.stdout:81: finishing write tid 1 to smithi15528762-35 2024-04-19T01:04:11.973 INFO:tasks.rados.rados.0.smithi155.stdout:81: finishing write tid 2 to smithi15528762-35 2024-04-19T01:04:11.973 INFO:tasks.rados.rados.0.smithi155.stdout:81: finishing write tid 3 to smithi15528762-35 2024-04-19T01:04:11.973 INFO:tasks.rados.rados.0.smithi155.stdout:81: finishing write tid 4 to smithi15528762-35 2024-04-19T01:04:11.973 INFO:tasks.rados.rados.0.smithi155.stdout:81: oid 35 updating version 0 to 6 2024-04-19T01:04:11.973 INFO:tasks.rados.rados.0.smithi155.stdout:81: oid 35 version 6 is already newer than 5 2024-04-19T01:04:11.973 INFO:tasks.rados.rados.0.smithi155.stdout:81: oid 35 version 6 is already newer than 4 2024-04-19T01:04:11.973 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 35 v 6 (ObjNum 60 snap 5 seq_num 60) dirty exists 2024-04-19T01:04:11.973 INFO:tasks.rados.rados.0.smithi155.stdout:81: left oid 35 (ObjNum 60 snap 5 seq_num 60) 2024-04-19T01:04:11.973 INFO:tasks.rados.rados.0.smithi155.stdout:82: finishing write tid 1 to smithi15528762-43 2024-04-19T01:04:11.973 INFO:tasks.rados.rados.0.smithi155.stdout:82: finishing write tid 2 to smithi15528762-43 2024-04-19T01:04:11.973 INFO:tasks.rados.rados.0.smithi155.stdout:82: finishing write tid 3 to smithi15528762-43 2024-04-19T01:04:11.973 INFO:tasks.rados.rados.0.smithi155.stdout:82: finishing write tid 4 to smithi15528762-43 2024-04-19T01:04:11.974 INFO:tasks.rados.rados.0.smithi155.stdout:82: oid 43 updating version 0 to 35 2024-04-19T01:04:11.974 INFO:tasks.rados.rados.0.smithi155.stdout:82: oid 43 updating version 35 to 36 2024-04-19T01:04:11.974 INFO:tasks.rados.rados.0.smithi155.stdout:82: oid 43 updating version 36 to 37 2024-04-19T01:04:11.974 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 43 v 37 (ObjNum 61 snap 5 seq_num 61) dirty exists 2024-04-19T01:04:11.974 INFO:tasks.rados.rados.0.smithi155.stdout:82: left oid 43 (ObjNum 61 snap 5 seq_num 61) 2024-04-19T01:04:11.993 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:04:11.991+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:04:12.015 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:12.016 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:12.031 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-04-19T01:04:12.094 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:04:12.092+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:04:12.151 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:12.151 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:12.165 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-19T01:04:12.212 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:04:12.213+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:04:12.227 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-19T01:04:12.296 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:12.297 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:12.297 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:04:12.295+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:04:12.310 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-19T01:04:12.373 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-04-19T01:04:12.374 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T01:04:12.395 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-04-19T01:04:12.395 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T01:04:12.397 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:04:12.395+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:04:12.418 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-04-19T01:04:12.418 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T01:04:12.440 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-04-19T01:04:12.440 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T01:04:12.457 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-04-19T01:04:12.457 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T01:04:12.474 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:04:12.474 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:04:12.474 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-04-19T01:04:12.474 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-04-19T01:04:12.474 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:04:12.478 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-04-19T01:04:12.479 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T01:04:12.486 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-19T01:04:12.501 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-04-19T01:04:12.502 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T01:04:12.522 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-04-19T01:04:12.522 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T01:04:12.531 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:04:12.533+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:04:12.543 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2024-04-19T01:04:12.544 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T01:04:12.566 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2024-04-19T01:04:12.567 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T01:04:12.589 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2024-04-19T01:04:12.589 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T01:04:12.599 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:04:12.597+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:04:12.603 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:04:12.604 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-04-19T01:04:12.604 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-04-19T01:04:12.604 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-04-19T01:04:12.604 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.604 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.0s3 38/31 ECSubWrite(tid=11, reqid=client.4604.0:169, at_version=38'8, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:04:12.604 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:08.807627+0000", 2024-04-19T01:04:12.604 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 3.7981727169999999, 2024-04-19T01:04:12.604 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0036417569999999998, 2024-04-19T01:04:12.604 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:12.604 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:12.605 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:12.605 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:12.605 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.605 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:12.605 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.807627+0000", 2024-04-19T01:04:12.605 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:12.605 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:12.605 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.605 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:12.605 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.807627+0000", 2024-04-19T01:04:12.605 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:12.605 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:12.605 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.606 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:12.606 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.807628+0000", 2024-04-19T01:04:12.606 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5699999999999998e-07 2024-04-19T01:04:12.606 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:12.606 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.606 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:12.606 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.807669+0000", 2024-04-19T01:04:12.606 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0889000000000001e-05 2024-04-19T01:04:12.606 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:12.607 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.607 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:12.607 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.807702+0000", 2024-04-19T01:04:12.607 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3096000000000003e-05 2024-04-19T01:04:12.607 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:12.607 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.607 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:12.607 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.807725+0000", 2024-04-19T01:04:12.607 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3329e-05 2024-04-19T01:04:12.607 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:12.607 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.607 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:12.608 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.807890+0000", 2024-04-19T01:04:12.608 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016469600000000001 2024-04-19T01:04:12.608 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:12.608 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.608 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:12.608 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.807918+0000", 2024-04-19T01:04:12.608 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8693000000000001e-05 2024-04-19T01:04:12.608 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:12.608 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.608 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:12.608 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.811184+0000", 2024-04-19T01:04:12.608 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0032654060000000002 2024-04-19T01:04:12.608 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:12.608 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.609 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:12.609 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.811269+0000", 2024-04-19T01:04:12.609 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5290999999999994e-05 2024-04-19T01:04:12.609 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:12.609 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:12.609 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:12.609 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:12.609 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.609 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.0s3 38/31 ECSubWrite(tid=12, reqid=client.4604.0:170, at_version=38'9, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:04:12.609 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:08.810226+0000", 2024-04-19T01:04:12.610 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 3.7955739509999997, 2024-04-19T01:04:12.610 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0020635419999999998, 2024-04-19T01:04:12.610 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:12.610 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:12.610 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:12.610 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:12.610 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.610 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:12.610 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.810226+0000", 2024-04-19T01:04:12.610 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:12.611 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:12.611 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.611 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:12.611 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.810226+0000", 2024-04-19T01:04:12.611 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:12.611 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:12.611 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.611 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:12.611 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.810228+0000", 2024-04-19T01:04:12.611 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8369999999999999e-06 2024-04-19T01:04:12.611 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:12.611 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.612 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:12.612 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.810304+0000", 2024-04-19T01:04:12.612 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6113999999999997e-05 2024-04-19T01:04:12.612 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:12.612 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.612 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:12.612 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.810342+0000", 2024-04-19T01:04:12.612 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7698999999999999e-05 2024-04-19T01:04:12.612 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:12.612 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.612 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:12.612 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.810368+0000", 2024-04-19T01:04:12.612 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6602999999999999e-05 2024-04-19T01:04:12.612 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:12.613 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.613 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:12.613 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.810532+0000", 2024-04-19T01:04:12.613 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016407300000000001 2024-04-19T01:04:12.613 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:12.613 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.613 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:12.613 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.810557+0000", 2024-04-19T01:04:12.613 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4312999999999999e-05 2024-04-19T01:04:12.613 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:12.613 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.613 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:12.613 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.812222+0000", 2024-04-19T01:04:12.614 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001665609 2024-04-19T01:04:12.614 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:12.614 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.614 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:12.614 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.812290+0000", 2024-04-19T01:04:12.614 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7293999999999996e-05 2024-04-19T01:04:12.614 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:12.614 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:12.614 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:12.614 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:12.614 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.614 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.0s3 38/31 ECSubWrite(tid=13, reqid=client.4604.0:171, at_version=38'10, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:04:12.615 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:08.812791+0000", 2024-04-19T01:04:12.615 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 3.793009257, 2024-04-19T01:04:12.615 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0015931840000000001, 2024-04-19T01:04:12.615 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:12.615 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:12.615 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:12.615 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:12.615 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.615 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:12.615 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.812791+0000", 2024-04-19T01:04:12.616 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:12.616 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:12.616 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.616 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:12.616 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.812791+0000", 2024-04-19T01:04:12.616 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:12.616 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:12.616 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.616 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:12.616 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.812792+0000", 2024-04-19T01:04:12.616 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5400000000000001e-06 2024-04-19T01:04:12.616 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:12.617 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.617 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:12.617 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.812825+0000", 2024-04-19T01:04:12.617 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2354999999999998e-05 2024-04-19T01:04:12.617 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:12.617 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.617 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:12.617 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.812862+0000", 2024-04-19T01:04:12.617 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7116000000000002e-05 2024-04-19T01:04:12.617 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:12.617 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.617 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:12.617 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.812890+0000", 2024-04-19T01:04:12.618 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7909e-05 2024-04-19T01:04:12.618 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:12.618 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.618 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:12.618 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.813038+0000", 2024-04-19T01:04:12.618 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014805999999999999 2024-04-19T01:04:12.618 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:12.619 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.619 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:12.619 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.813064+0000", 2024-04-19T01:04:12.619 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6346000000000002e-05 2024-04-19T01:04:12.619 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:12.619 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.619 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:12.619 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.814319+0000", 2024-04-19T01:04:12.620 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001255092 2024-04-19T01:04:12.620 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:12.620 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.620 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:12.620 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.814384+0000", 2024-04-19T01:04:12.620 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4765999999999996e-05 2024-04-19T01:04:12.620 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:12.620 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:12.620 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:12.620 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:12.621 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.621 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.0s3 38/31 ECSubWrite(tid=14, reqid=client.4604.0:171, at_version=0'0, trim_to=0'0, roll_forward_to=38'10))", 2024-04-19T01:04:12.621 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:08.820124+0000", 2024-04-19T01:04:12.621 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 3.7856763020000002, 2024-04-19T01:04:12.621 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001482253, 2024-04-19T01:04:12.621 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:12.621 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:12.621 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:12.621 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:12.621 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.621 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:12.622 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.820124+0000", 2024-04-19T01:04:12.622 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:12.622 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:12.622 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.622 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:12.622 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.820124+0000", 2024-04-19T01:04:12.622 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:12.622 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:12.622 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.622 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:12.622 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.820126+0000", 2024-04-19T01:04:12.622 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.784e-06 2024-04-19T01:04:12.623 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:12.623 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.623 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:12.623 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.820147+0000", 2024-04-19T01:04:12.623 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1951999999999999e-05 2024-04-19T01:04:12.623 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:12.623 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.623 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:12.623 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.820179+0000", 2024-04-19T01:04:12.623 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1936000000000001e-05 2024-04-19T01:04:12.623 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:12.623 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.623 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:12.623 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.820197+0000", 2024-04-19T01:04:12.624 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7802999999999999e-05 2024-04-19T01:04:12.624 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:12.624 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.624 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:12.624 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.820385+0000", 2024-04-19T01:04:12.624 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000188269 2024-04-19T01:04:12.624 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:12.624 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.624 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:12.624 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.820413+0000", 2024-04-19T01:04:12.624 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7932999999999998e-05 2024-04-19T01:04:12.624 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:12.624 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.624 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:12.624 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.821533+0000", 2024-04-19T01:04:12.625 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011198650000000001 2024-04-19T01:04:12.625 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:12.625 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:12.625 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:12.625 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.821606+0000", 2024-04-19T01:04:12.625 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2711000000000002e-05 2024-04-19T01:04:12.625 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:12.625 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:12.625 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:12.625 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:12.625 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:12.625 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:04:12.626 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2024-04-19T01:04:12.627 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T01:04:12.627 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-19T01:04:12.632 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2024-04-19T01:04:12.632 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-19T01:04:12.657 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2024-04-19T01:04:12.657 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-19T01:04:12.681 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2024-04-19T01:04:12.682 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-19T01:04:12.701 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:04:12.700+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:04:12.703 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2024-04-19T01:04:12.704 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-19T01:04:12.797 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:04:12.797 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:04:12.797 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-04-19T01:04:12.797 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:04:12.801 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:04:12.801+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:04:12.811 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-19T01:04:12.822 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.6 to 1 (10000) 2024-04-19T01:04:12.828 INFO:tasks.rados.rados.0.smithi155.stdout:77: done (5 left) 2024-04-19T01:04:12.828 INFO:tasks.rados.rados.0.smithi155.stdout:78: done (4 left) 2024-04-19T01:04:12.829 INFO:tasks.rados.rados.0.smithi155.stdout:80: done (3 left) 2024-04-19T01:04:12.829 INFO:tasks.rados.rados.0.smithi155.stdout:81: done (2 left) 2024-04-19T01:04:12.829 INFO:tasks.rados.rados.0.smithi155.stdout:82: done (1 left) 2024-04-19T01:04:12.829 INFO:tasks.rados.rados.0.smithi155.stdout:83: done (0 left) 2024-04-19T01:04:12.829 INFO:tasks.rados.rados.0.smithi155.stdout:84: read oid 19 snap -1 2024-04-19T01:04:12.829 INFO:tasks.rados.rados.0.smithi155.stdout:84: expect (ObjNum 56 snap 5 seq_num 56) 2024-04-19T01:04:12.829 INFO:tasks.rados.rados.0.smithi155.stdout:85: delete oid 39 current snap is 6 2024-04-19T01:04:12.838 INFO:tasks.rados.rados.0.smithi155.stdout:85: done (1 left) 2024-04-19T01:04:12.838 INFO:tasks.rados.rados.0.smithi155.stdout:86: snap_remove snap 4 2024-04-19T01:04:12.849 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-04-19T01:04:12.902 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:04:12.904+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:04:12.953 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:04:12.954 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:04:12.954 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-04-19T01:04:12.954 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-04-19T01:04:12.954 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:04:12.967 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-19T01:04:13.003 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:04:13.004+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:04:13.104 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:04:13.105+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:04:13.135 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:04:13.136 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-04-19T01:04:13.136 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-04-19T01:04:13.136 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [] 2024-04-19T01:04:13.136 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:04:13.150 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-19T01:04:13.210 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:04:13.208+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:04:13.309 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:04:13.307+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:04:13.312 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:04:13.313 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:04:13.313 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-04-19T01:04:13.313 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:04:13.326 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-19T01:04:13.410 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:04:13.409+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:04:13.443 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:04:13.443 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:04:13.443 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-04-19T01:04:13.444 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-04-19T01:04:13.444 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:04:13.458 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-19T01:04:13.511 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:13.512+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:13.611 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:04:13.609+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:04:13.628 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:04:13.628 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-04-19T01:04:13.628 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-04-19T01:04:13.628 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-04-19T01:04:13.628 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.628 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4260.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-19T01:04:13.629 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:03:50.620310+0000", 2024-04-19T01:04:13.629 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 23.005599490000002, 2024-04-19T01:04:13.629 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.15782495899999999, 2024-04-19T01:04:13.629 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:13.629 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:13.629 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:13.629 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:13.629 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4260", 2024-04-19T01:04:13.629 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/1127060541", 2024-04-19T01:04:13.629 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1 2024-04-19T01:04:13.629 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.629 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:13.629 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.629 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:13.630 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.620310+0000", 2024-04-19T01:04:13.630 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:13.630 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.630 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.630 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:13.630 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.620310+0000", 2024-04-19T01:04:13.630 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:13.630 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.630 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.630 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:13.630 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.620313+0000", 2024-04-19T01:04:13.631 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.757e-06 2024-04-19T01:04:13.631 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.631 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.631 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:13.631 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.620327+0000", 2024-04-19T01:04:13.631 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4029000000000001e-05 2024-04-19T01:04:13.631 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.631 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.631 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:13.631 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.620362+0000", 2024-04-19T01:04:13.631 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5201999999999999e-05 2024-04-19T01:04:13.631 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.631 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.632 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:13.632 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.620383+0000", 2024-04-19T01:04:13.632 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1049000000000001e-05 2024-04-19T01:04:13.632 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.632 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.632 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:13.632 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.620512+0000", 2024-04-19T01:04:13.632 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012820900000000001 2024-04-19T01:04:13.632 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.632 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.632 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for peered", 2024-04-19T01:04:13.632 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.620542+0000", 2024-04-19T01:04:13.632 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0264999999999999e-05 2024-04-19T01:04:13.632 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.633 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.633 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:13.633 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.774580+0000", 2024-04-19T01:04:13.633 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.15403776799999999 2024-04-19T01:04:13.633 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.633 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.633 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:13.633 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.775052+0000", 2024-04-19T01:04:13.633 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00047277300000000001 2024-04-19T01:04:13.633 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.633 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.634 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 5", 2024-04-19T01:04:13.634 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.775495+0000", 2024-04-19T01:04:13.634 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00044209400000000003 2024-04-19T01:04:13.634 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.634 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.634 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-04-19T01:04:13.634 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.777608+0000", 2024-04-19T01:04:13.634 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0021133649999999999 2024-04-19T01:04:13.634 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.635 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.635 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-04-19T01:04:13.635 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.777957+0000", 2024-04-19T01:04:13.635 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00034961399999999999 2024-04-19T01:04:13.635 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.635 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.635 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:13.635 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.778062+0000", 2024-04-19T01:04:13.635 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000104576 2024-04-19T01:04:13.636 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.636 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.636 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:13.636 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.778135+0000", 2024-04-19T01:04:13.636 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3257999999999996e-05 2024-04-19T01:04:13.636 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:13.636 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:13.636 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:13.636 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.636 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.636 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4260.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-19T01:04:13.636 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:03:51.044418+0000", 2024-04-19T01:04:13.636 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 22.581492075, 2024-04-19T01:04:13.636 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.019787530000000001, 2024-04-19T01:04:13.637 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:13.637 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:13.637 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:13.637 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:13.637 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4260", 2024-04-19T01:04:13.637 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/1127060541", 2024-04-19T01:04:13.637 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 110 2024-04-19T01:04:13.637 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.637 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:13.637 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.637 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:13.637 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.044418+0000", 2024-04-19T01:04:13.637 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:13.637 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.638 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.638 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:13.638 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.044418+0000", 2024-04-19T01:04:13.638 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:13.638 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.638 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.638 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:13.638 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.044419+0000", 2024-04-19T01:04:13.638 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.33e-07 2024-04-19T01:04:13.638 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.638 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.638 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:13.638 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.044426+0000", 2024-04-19T01:04:13.639 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.294e-06 2024-04-19T01:04:13.639 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.639 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.639 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:13.639 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.044439+0000", 2024-04-19T01:04:13.639 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2746e-05 2024-04-19T01:04:13.639 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.639 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.639 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:13.639 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.044451+0000", 2024-04-19T01:04:13.639 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1863e-05 2024-04-19T01:04:13.639 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.639 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.639 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:13.640 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.052559+0000", 2024-04-19T01:04:13.640 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0081084079999999992 2024-04-19T01:04:13.640 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.640 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.640 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:13.640 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.052680+0000", 2024-04-19T01:04:13.640 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000121123 2024-04-19T01:04:13.640 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.640 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.640 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 5", 2024-04-19T01:04:13.640 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.052882+0000", 2024-04-19T01:04:13.640 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000202096 2024-04-19T01:04:13.640 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.640 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.641 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-04-19T01:04:13.641 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.056750+0000", 2024-04-19T01:04:13.641 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0038676779999999998 2024-04-19T01:04:13.641 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.641 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.641 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-04-19T01:04:13.641 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.064013+0000", 2024-04-19T01:04:13.641 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.007262686 2024-04-19T01:04:13.641 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.641 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.641 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:13.641 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.064095+0000", 2024-04-19T01:04:13.641 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.2393999999999998e-05 2024-04-19T01:04:13.641 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.642 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.642 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:13.642 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.064205+0000", 2024-04-19T01:04:13.642 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000110309 2024-04-19T01:04:13.642 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:13.642 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:13.642 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:13.642 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.642 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.642 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4260.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-19T01:04:13.642 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:03:51.044465+0000", 2024-04-19T01:04:13.642 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 22.581444907000002, 2024-04-19T01:04:13.642 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.019930231, 2024-04-19T01:04:13.643 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:13.643 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:13.643 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:13.643 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:13.643 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4260", 2024-04-19T01:04:13.643 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/1127060541", 2024-04-19T01:04:13.643 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 111 2024-04-19T01:04:13.643 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.643 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:13.643 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.643 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:13.643 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.044465+0000", 2024-04-19T01:04:13.643 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:13.644 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.644 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.644 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:13.644 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.044465+0000", 2024-04-19T01:04:13.644 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:13.644 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.644 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.644 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:13.644 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.044466+0000", 2024-04-19T01:04:13.644 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.0999999999999997e-07 2024-04-19T01:04:13.644 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.644 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.645 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:13.645 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.044473+0000", 2024-04-19T01:04:13.645 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7370000000000006e-06 2024-04-19T01:04:13.645 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.645 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.645 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:13.645 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.044484+0000", 2024-04-19T01:04:13.645 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0665999999999999e-05 2024-04-19T01:04:13.646 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.646 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.646 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:13.646 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.044494+0000", 2024-04-19T01:04:13.646 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5659999999999996e-06 2024-04-19T01:04:13.646 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.646 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.646 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:13.646 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.053989+0000", 2024-04-19T01:04:13.646 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0094957119999999999 2024-04-19T01:04:13.646 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.646 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.646 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:13.647 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.054171+0000", 2024-04-19T01:04:13.647 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00018117 2024-04-19T01:04:13.647 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.647 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.647 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 5", 2024-04-19T01:04:13.647 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.054386+0000", 2024-04-19T01:04:13.647 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021585199999999999 2024-04-19T01:04:13.647 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.647 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.647 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-04-19T01:04:13.647 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.056872+0000", 2024-04-19T01:04:13.647 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0024858020000000001 2024-04-19T01:04:13.647 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.648 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.648 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-04-19T01:04:13.648 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.064223+0000", 2024-04-19T01:04:13.648 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0073510670000000002 2024-04-19T01:04:13.648 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.648 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.648 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:13.648 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.064296+0000", 2024-04-19T01:04:13.648 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2333e-05 2024-04-19T01:04:13.648 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.648 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.648 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:13.648 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.064395+0000", 2024-04-19T01:04:13.648 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9516000000000001e-05 2024-04-19T01:04:13.649 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:13.649 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:13.649 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:13.649 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.649 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.649 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4260.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-19T01:04:13.649 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:03:51.044529+0000", 2024-04-19T01:04:13.649 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 22.581380626000001, 2024-04-19T01:04:13.649 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.020041330999999999, 2024-04-19T01:04:13.649 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:13.649 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:13.649 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:13.650 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:13.650 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4260", 2024-04-19T01:04:13.650 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/1127060541", 2024-04-19T01:04:13.650 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 112 2024-04-19T01:04:13.650 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.650 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:13.650 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.650 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:13.650 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.044529+0000", 2024-04-19T01:04:13.650 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:13.650 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.650 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.650 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:13.651 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.044529+0000", 2024-04-19T01:04:13.651 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:13.651 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.651 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.651 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:13.651 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.044530+0000", 2024-04-19T01:04:13.651 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.7000000000000003e-07 2024-04-19T01:04:13.651 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.651 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.652 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:13.652 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.044663+0000", 2024-04-19T01:04:13.652 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013264900000000001 2024-04-19T01:04:13.652 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.652 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.652 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:13.652 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.044677+0000", 2024-04-19T01:04:13.652 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4270000000000001e-05 2024-04-19T01:04:13.652 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.652 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.652 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:13.652 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.044687+0000", 2024-04-19T01:04:13.652 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7249999999999992e-06 2024-04-19T01:04:13.653 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.653 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.653 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:13.653 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.055180+0000", 2024-04-19T01:04:13.653 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.010493135000000001 2024-04-19T01:04:13.653 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.653 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.653 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:13.653 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.055425+0000", 2024-04-19T01:04:13.653 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000244935 2024-04-19T01:04:13.653 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.653 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.654 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 5", 2024-04-19T01:04:13.654 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.055793+0000", 2024-04-19T01:04:13.654 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00036859300000000001 2024-04-19T01:04:13.654 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.654 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.654 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-04-19T01:04:13.654 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.060357+0000", 2024-04-19T01:04:13.654 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0045631919999999998 2024-04-19T01:04:13.654 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.654 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.654 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-04-19T01:04:13.654 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.064409+0000", 2024-04-19T01:04:13.654 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0040520310000000002 2024-04-19T01:04:13.655 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.655 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.655 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:13.655 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.064475+0000", 2024-04-19T01:04:13.655 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6583999999999998e-05 2024-04-19T01:04:13.655 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.655 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.655 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:13.655 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.064571+0000", 2024-04-19T01:04:13.656 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5346999999999994e-05 2024-04-19T01:04:13.656 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:13.656 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:13.656 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:13.656 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.656 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.656 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4260.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-19T01:04:13.656 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:03:51.044703+0000", 2024-04-19T01:04:13.656 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 22.581206792, 2024-04-19T01:04:13.656 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.020047727000000001, 2024-04-19T01:04:13.656 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:13.656 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:13.656 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:13.657 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:13.657 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4260", 2024-04-19T01:04:13.657 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/1127060541", 2024-04-19T01:04:13.657 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 113 2024-04-19T01:04:13.657 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.657 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:13.657 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.657 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:13.657 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.044703+0000", 2024-04-19T01:04:13.657 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:13.657 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.658 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.658 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:13.658 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.044703+0000", 2024-04-19T01:04:13.658 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:13.658 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.658 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.658 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:13.658 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.044704+0000", 2024-04-19T01:04:13.658 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.0400000000000005e-07 2024-04-19T01:04:13.658 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.658 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.658 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:13.659 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.044713+0000", 2024-04-19T01:04:13.659 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5349999999999993e-06 2024-04-19T01:04:13.659 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.659 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.659 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:13.659 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.044724+0000", 2024-04-19T01:04:13.659 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0200000000000001e-05 2024-04-19T01:04:13.659 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.659 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.659 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:13.659 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.044732+0000", 2024-04-19T01:04:13.660 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.0239999999999997e-06 2024-04-19T01:04:13.660 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.660 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.660 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:13.660 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.057173+0000", 2024-04-19T01:04:13.660 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.012441429 2024-04-19T01:04:13.660 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.660 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.660 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:13.660 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.057317+0000", 2024-04-19T01:04:13.660 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014336899999999999 2024-04-19T01:04:13.661 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.661 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.661 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 5", 2024-04-19T01:04:13.661 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.057579+0000", 2024-04-19T01:04:13.661 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00026230300000000003 2024-04-19T01:04:13.661 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.661 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.661 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-04-19T01:04:13.661 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.060507+0000", 2024-04-19T01:04:13.661 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0029276950000000001 2024-04-19T01:04:13.661 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.661 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.661 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-04-19T01:04:13.662 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.064587+0000", 2024-04-19T01:04:13.662 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0040801639999999998 2024-04-19T01:04:13.662 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.662 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.662 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:13.662 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.064657+0000", 2024-04-19T01:04:13.662 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0420999999999995e-05 2024-04-19T01:04:13.662 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.662 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.662 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:13.662 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.064751+0000", 2024-04-19T01:04:13.662 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3683000000000004e-05 2024-04-19T01:04:13.662 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:13.662 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:13.663 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:13.663 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.663 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.663 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4260.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-19T01:04:13.663 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:03:51.044749+0000", 2024-04-19T01:04:13.663 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 22.581161269999999, 2024-04-19T01:04:13.663 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.021225126, 2024-04-19T01:04:13.663 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:13.663 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:13.663 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:13.663 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:13.663 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4260", 2024-04-19T01:04:13.663 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/1127060541", 2024-04-19T01:04:13.663 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 114 2024-04-19T01:04:13.664 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.664 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:13.664 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.664 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:13.664 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.044749+0000", 2024-04-19T01:04:13.664 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:13.664 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.664 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.664 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:13.664 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.044749+0000", 2024-04-19T01:04:13.664 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:13.664 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.664 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.664 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:13.664 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.044749+0000", 2024-04-19T01:04:13.665 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4000000000000001e-07 2024-04-19T01:04:13.665 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.665 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.665 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:13.665 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.044755+0000", 2024-04-19T01:04:13.665 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6099999999999997e-06 2024-04-19T01:04:13.665 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.665 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.665 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:13.665 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.044765+0000", 2024-04-19T01:04:13.666 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0349e-05 2024-04-19T01:04:13.666 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.666 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.666 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:13.666 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.044773+0000", 2024-04-19T01:04:13.666 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.9500000000000001e-06 2024-04-19T01:04:13.666 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.666 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.666 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:13.666 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.058797+0000", 2024-04-19T01:04:13.666 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.014023450999999999 2024-04-19T01:04:13.666 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.666 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.667 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:13.667 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.059049+0000", 2024-04-19T01:04:13.667 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00025231500000000003 2024-04-19T01:04:13.667 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.667 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.667 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 5", 2024-04-19T01:04:13.667 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.059446+0000", 2024-04-19T01:04:13.667 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00039740499999999998 2024-04-19T01:04:13.667 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.667 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.667 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-04-19T01:04:13.667 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.063989+0000", 2024-04-19T01:04:13.667 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0045430030000000003 2024-04-19T01:04:13.667 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.668 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.668 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-04-19T01:04:13.668 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.065782+0000", 2024-04-19T01:04:13.668 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0017926999999999999 2024-04-19T01:04:13.668 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.668 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.668 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:13.668 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.065857+0000", 2024-04-19T01:04:13.668 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5375000000000003e-05 2024-04-19T01:04:13.668 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.668 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.668 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:13.668 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.065974+0000", 2024-04-19T01:04:13.668 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000116228 2024-04-19T01:04:13.669 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:13.669 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:13.669 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:13.669 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.669 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.669 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4260.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-19T01:04:13.669 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:03:51.045047+0000", 2024-04-19T01:04:13.669 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 22.580862374999999, 2024-04-19T01:04:13.669 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.021134024000000001, 2024-04-19T01:04:13.669 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:13.669 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:13.669 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:13.669 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:13.669 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4260", 2024-04-19T01:04:13.670 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/1127060541", 2024-04-19T01:04:13.670 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 115 2024-04-19T01:04:13.670 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.670 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:13.670 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.670 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:13.670 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.045047+0000", 2024-04-19T01:04:13.670 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:13.670 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.670 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.670 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:13.671 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.045047+0000", 2024-04-19T01:04:13.671 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:13.671 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.671 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.671 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:13.671 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.045049+0000", 2024-04-19T01:04:13.671 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7269999999999999e-06 2024-04-19T01:04:13.671 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.671 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.671 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:13.671 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.045274+0000", 2024-04-19T01:04:13.671 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000225169 2024-04-19T01:04:13.671 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.672 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.672 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:13.672 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.045298+0000", 2024-04-19T01:04:13.672 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4054999999999999e-05 2024-04-19T01:04:13.672 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.672 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.672 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:13.672 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.045316+0000", 2024-04-19T01:04:13.672 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7529000000000001e-05 2024-04-19T01:04:13.672 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.672 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.672 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:13.672 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.060619+0000", 2024-04-19T01:04:13.672 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.015303192 2024-04-19T01:04:13.673 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.673 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.673 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:13.673 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.060892+0000", 2024-04-19T01:04:13.673 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00027284299999999999 2024-04-19T01:04:13.673 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.673 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.673 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 5", 2024-04-19T01:04:13.673 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.061167+0000", 2024-04-19T01:04:13.673 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000275443 2024-04-19T01:04:13.674 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.674 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.674 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-04-19T01:04:13.674 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.064894+0000", 2024-04-19T01:04:13.674 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0037264030000000001 2024-04-19T01:04:13.674 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.674 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.674 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-04-19T01:04:13.674 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.065987+0000", 2024-04-19T01:04:13.674 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00109366 2024-04-19T01:04:13.674 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.674 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.674 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:13.675 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.066084+0000", 2024-04-19T01:04:13.675 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6866000000000004e-05 2024-04-19T01:04:13.675 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.675 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.675 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:13.675 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.066182+0000", 2024-04-19T01:04:13.675 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7137000000000002e-05 2024-04-19T01:04:13.675 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:13.675 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:13.675 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:13.675 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.675 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.675 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4260.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-19T01:04:13.675 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:03:51.045356+0000", 2024-04-19T01:04:13.676 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 22.580554352, 2024-04-19T01:04:13.676 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.021087847999999999, 2024-04-19T01:04:13.676 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:13.676 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:13.676 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:13.676 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:13.676 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4260", 2024-04-19T01:04:13.676 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/1127060541", 2024-04-19T01:04:13.676 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 116 2024-04-19T01:04:13.677 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.677 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:13.677 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.677 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:13.677 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.045356+0000", 2024-04-19T01:04:13.677 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:13.677 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.677 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.677 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:13.677 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.045356+0000", 2024-04-19T01:04:13.677 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:13.677 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.677 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.677 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:13.678 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.045357+0000", 2024-04-19T01:04:13.678 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7659999999999999e-06 2024-04-19T01:04:13.678 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.678 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.678 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:13.678 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.045369+0000", 2024-04-19T01:04:13.678 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1469e-05 2024-04-19T01:04:13.678 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.678 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.678 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:13.678 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.045411+0000", 2024-04-19T01:04:13.678 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2246000000000002e-05 2024-04-19T01:04:13.678 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.679 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.679 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:13.679 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.045427+0000", 2024-04-19T01:04:13.679 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5596000000000001e-05 2024-04-19T01:04:13.679 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.679 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.679 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:13.679 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.062599+0000", 2024-04-19T01:04:13.679 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.017172366000000001 2024-04-19T01:04:13.679 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.679 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.679 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:13.679 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.062821+0000", 2024-04-19T01:04:13.680 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00022237099999999999 2024-04-19T01:04:13.680 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.680 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.680 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 5", 2024-04-19T01:04:13.680 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.063197+0000", 2024-04-19T01:04:13.680 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00037538300000000002 2024-04-19T01:04:13.680 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.680 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.680 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-04-19T01:04:13.680 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.066195+0000", 2024-04-19T01:04:13.680 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0029984529999999999 2024-04-19T01:04:13.680 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.680 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.680 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-04-19T01:04:13.681 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.066291+0000", 2024-04-19T01:04:13.681 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6020999999999994e-05 2024-04-19T01:04:13.681 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.681 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.681 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:13.681 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.066377+0000", 2024-04-19T01:04:13.681 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5582999999999997e-05 2024-04-19T01:04:13.681 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.681 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.681 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:13.681 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.066443+0000", 2024-04-19T01:04:13.681 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6594000000000006e-05 2024-04-19T01:04:13.681 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:13.681 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:13.682 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:13.682 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.682 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.682 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4260.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-19T01:04:13.682 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:03:51.045451+0000", 2024-04-19T01:04:13.682 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 22.580458783000001, 2024-04-19T01:04:13.682 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.021453949, 2024-04-19T01:04:13.682 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:13.682 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:13.682 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:13.682 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:13.682 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4260", 2024-04-19T01:04:13.683 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/1127060541", 2024-04-19T01:04:13.683 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 117 2024-04-19T01:04:13.683 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.683 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:13.683 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.683 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:13.683 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.045451+0000", 2024-04-19T01:04:13.683 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:13.683 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.683 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.683 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:13.684 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.045451+0000", 2024-04-19T01:04:13.684 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:13.684 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.684 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.684 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:13.684 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.045452+0000", 2024-04-19T01:04:13.684 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3400000000000001e-06 2024-04-19T01:04:13.684 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.684 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.684 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:13.684 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.045463+0000", 2024-04-19T01:04:13.684 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0210000000000001e-05 2024-04-19T01:04:13.684 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.685 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.685 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:13.685 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.045487+0000", 2024-04-19T01:04:13.685 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4206000000000002e-05 2024-04-19T01:04:13.685 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.685 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.685 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:13.685 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.045522+0000", 2024-04-19T01:04:13.685 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5407999999999997e-05 2024-04-19T01:04:13.685 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.685 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.685 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:13.686 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.064995+0000", 2024-04-19T01:04:13.686 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.019472910999999999 2024-04-19T01:04:13.686 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.686 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.686 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:13.686 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.065130+0000", 2024-04-19T01:04:13.686 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013476199999999999 2024-04-19T01:04:13.686 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.686 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.687 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 5", 2024-04-19T01:04:13.687 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.065338+0000", 2024-04-19T01:04:13.687 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00020856 2024-04-19T01:04:13.687 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.687 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.687 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-04-19T01:04:13.687 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.066473+0000", 2024-04-19T01:04:13.687 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011346310000000001 2024-04-19T01:04:13.687 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.687 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.687 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-04-19T01:04:13.687 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.066767+0000", 2024-04-19T01:04:13.687 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000293436 2024-04-19T01:04:13.688 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.688 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.688 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:13.688 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.066849+0000", 2024-04-19T01:04:13.688 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.1966999999999996e-05 2024-04-19T01:04:13.688 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.688 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.688 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:13.688 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.066905+0000", 2024-04-19T01:04:13.688 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6518000000000003e-05 2024-04-19T01:04:13.688 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:13.688 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:13.689 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:13.689 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.689 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.689 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4260.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-19T01:04:13.689 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:03:51.154299+0000", 2024-04-19T01:04:13.689 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 22.471610496, 2024-04-19T01:04:13.689 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.018258566, 2024-04-19T01:04:13.689 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:13.689 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:13.689 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:13.689 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:13.689 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4260", 2024-04-19T01:04:13.689 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/1127060541", 2024-04-19T01:04:13.690 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 137 2024-04-19T01:04:13.690 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.690 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:13.690 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.690 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:13.690 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154299+0000", 2024-04-19T01:04:13.690 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:13.690 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.690 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.690 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:13.690 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154299+0000", 2024-04-19T01:04:13.690 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:13.691 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.691 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.691 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:13.691 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154300+0000", 2024-04-19T01:04:13.691 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5600000000000005e-07 2024-04-19T01:04:13.691 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.691 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.691 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:13.691 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154433+0000", 2024-04-19T01:04:13.692 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013333600000000001 2024-04-19T01:04:13.692 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.692 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.692 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:13.692 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154462+0000", 2024-04-19T01:04:13.692 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8198999999999999e-05 2024-04-19T01:04:13.692 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.692 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.692 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:13.692 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154472+0000", 2024-04-19T01:04:13.692 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9790000000000005e-06 2024-04-19T01:04:13.692 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.693 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.693 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:13.693 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.166367+0000", 2024-04-19T01:04:13.693 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.011895533 2024-04-19T01:04:13.693 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.693 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.693 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:13.693 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.166592+0000", 2024-04-19T01:04:13.693 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000224968 2024-04-19T01:04:13.693 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.693 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.693 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 5", 2024-04-19T01:04:13.693 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.166940+0000", 2024-04-19T01:04:13.694 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00034807399999999997 2024-04-19T01:04:13.694 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.694 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.694 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-04-19T01:04:13.694 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.170862+0000", 2024-04-19T01:04:13.694 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0039215830000000002 2024-04-19T01:04:13.694 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.694 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.694 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-04-19T01:04:13.694 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.172318+0000", 2024-04-19T01:04:13.694 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001456094 2024-04-19T01:04:13.694 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.694 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.694 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:13.695 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.172446+0000", 2024-04-19T01:04:13.695 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012857799999999999 2024-04-19T01:04:13.695 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.695 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.695 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:13.695 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.172558+0000", 2024-04-19T01:04:13.695 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011146600000000001 2024-04-19T01:04:13.695 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:13.695 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:13.695 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:13.695 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.695 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.695 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4260.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-19T01:04:13.695 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:03:51.154493+0000", 2024-04-19T01:04:13.696 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 22.471417147, 2024-04-19T01:04:13.696 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.020485405000000002, 2024-04-19T01:04:13.696 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:13.696 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:13.696 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:13.697 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:13.697 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4260", 2024-04-19T01:04:13.697 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/1127060541", 2024-04-19T01:04:13.697 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 138 2024-04-19T01:04:13.697 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.697 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:13.697 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.697 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:13.697 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154493+0000", 2024-04-19T01:04:13.697 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:13.698 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.698 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.698 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:13.698 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154493+0000", 2024-04-19T01:04:13.698 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:13.698 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.698 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.698 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:13.698 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154494+0000", 2024-04-19T01:04:13.698 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.8000000000000004e-07 2024-04-19T01:04:13.698 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.698 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.698 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:13.698 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154500+0000", 2024-04-19T01:04:13.699 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5300000000000002e-06 2024-04-19T01:04:13.699 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.699 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.699 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:13.699 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154511+0000", 2024-04-19T01:04:13.699 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1097000000000001e-05 2024-04-19T01:04:13.700 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.700 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.700 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:13.700 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154521+0000", 2024-04-19T01:04:13.700 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.825e-06 2024-04-19T01:04:13.700 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.700 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.700 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:13.700 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.169414+0000", 2024-04-19T01:04:13.700 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.014893257 2024-04-19T01:04:13.700 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.701 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.701 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:13.701 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.169568+0000", 2024-04-19T01:04:13.701 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015410099999999999 2024-04-19T01:04:13.701 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.701 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.701 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 5", 2024-04-19T01:04:13.701 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.169788+0000", 2024-04-19T01:04:13.701 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021970599999999999 2024-04-19T01:04:13.701 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.701 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.701 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-04-19T01:04:13.701 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.174702+0000", 2024-04-19T01:04:13.701 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0049139489999999999 2024-04-19T01:04:13.702 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.702 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.702 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-04-19T01:04:13.702 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.174740+0000", 2024-04-19T01:04:13.702 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7837999999999999e-05 2024-04-19T01:04:13.702 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.702 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.702 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:13.702 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.174870+0000", 2024-04-19T01:04:13.702 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012996899999999999 2024-04-19T01:04:13.702 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.702 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.702 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:13.703 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.174978+0000", 2024-04-19T01:04:13.703 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000108253 2024-04-19T01:04:13.703 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:13.703 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:13.703 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:13.703 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.703 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.703 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4260.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-19T01:04:13.703 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:03:51.154533+0000", 2024-04-19T01:04:13.703 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 22.471377131000001, 2024-04-19T01:04:13.703 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.022264368, 2024-04-19T01:04:13.703 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:13.703 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:13.703 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:13.704 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:13.704 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4260", 2024-04-19T01:04:13.704 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/1127060541", 2024-04-19T01:04:13.704 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 139 2024-04-19T01:04:13.704 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.704 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:13.704 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.704 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:13.704 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154533+0000", 2024-04-19T01:04:13.704 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:13.704 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.704 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.704 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:13.705 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154533+0000", 2024-04-19T01:04:13.705 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:13.705 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.705 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.705 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:13.705 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154533+0000", 2024-04-19T01:04:13.705 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.8999999999999996e-07 2024-04-19T01:04:13.705 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.705 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.705 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:13.705 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154539+0000", 2024-04-19T01:04:13.705 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6999999999999996e-06 2024-04-19T01:04:13.705 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.706 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.706 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:13.706 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154551+0000", 2024-04-19T01:04:13.706 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1565e-05 2024-04-19T01:04:13.706 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.706 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.706 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:13.706 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154558+0000", 2024-04-19T01:04:13.706 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7000000000000008e-06 2024-04-19T01:04:13.706 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.706 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.706 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:13.707 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.170967+0000", 2024-04-19T01:04:13.707 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.016408323999999998 2024-04-19T01:04:13.707 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.707 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.707 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:13.707 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.171183+0000", 2024-04-19T01:04:13.708 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000216226 2024-04-19T01:04:13.708 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.708 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.708 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 5", 2024-04-19T01:04:13.708 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.171547+0000", 2024-04-19T01:04:13.708 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00036430000000000002 2024-04-19T01:04:13.708 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.708 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.708 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-04-19T01:04:13.708 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.175108+0000", 2024-04-19T01:04:13.708 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.003560325 2024-04-19T01:04:13.708 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.708 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.709 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-04-19T01:04:13.709 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.176560+0000", 2024-04-19T01:04:13.709 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001452577 2024-04-19T01:04:13.709 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.709 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.709 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:13.709 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.176684+0000", 2024-04-19T01:04:13.709 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012410200000000001 2024-04-19T01:04:13.709 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.709 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.709 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:13.709 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.176797+0000", 2024-04-19T01:04:13.709 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000112859 2024-04-19T01:04:13.710 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:13.710 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:13.710 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:13.710 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.710 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.710 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4260.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-19T01:04:13.710 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:03:51.154570+0000", 2024-04-19T01:04:13.710 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 22.471340329, 2024-04-19T01:04:13.710 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.022711031999999999, 2024-04-19T01:04:13.710 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:13.710 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:13.710 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:13.711 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:13.711 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4260", 2024-04-19T01:04:13.711 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/1127060541", 2024-04-19T01:04:13.711 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 140 2024-04-19T01:04:13.711 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.711 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:13.711 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.711 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:13.711 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154570+0000", 2024-04-19T01:04:13.711 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:13.711 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.711 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.711 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:13.711 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154570+0000", 2024-04-19T01:04:13.711 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:13.712 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.712 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.712 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:13.712 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154570+0000", 2024-04-19T01:04:13.712 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.23e-07 2024-04-19T01:04:13.712 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.712 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.712 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:13.712 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154713+0000", 2024-04-19T01:04:13.712 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014260400000000001 2024-04-19T01:04:13.712 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.712 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.712 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:13.713 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154726+0000", 2024-04-19T01:04:13.713 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3179999999999999e-05 2024-04-19T01:04:13.713 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.713 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.713 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:13.713 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154735+0000", 2024-04-19T01:04:13.713 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.9060000000000008e-06 2024-04-19T01:04:13.713 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.713 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.713 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:13.713 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.172661+0000", 2024-04-19T01:04:13.713 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.017926464999999999 2024-04-19T01:04:13.713 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.713 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.714 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:13.714 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.172783+0000", 2024-04-19T01:04:13.714 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000121243 2024-04-19T01:04:13.714 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.714 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.714 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 5", 2024-04-19T01:04:13.714 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.172977+0000", 2024-04-19T01:04:13.714 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019476199999999999 2024-04-19T01:04:13.714 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.714 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.714 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-04-19T01:04:13.714 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.176824+0000", 2024-04-19T01:04:13.715 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0038468859999999999 2024-04-19T01:04:13.715 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.715 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.715 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-04-19T01:04:13.715 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.176944+0000", 2024-04-19T01:04:13.715 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000119925 2024-04-19T01:04:13.715 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.715 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.715 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:13.715 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.177103+0000", 2024-04-19T01:04:13.715 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015884499999999999 2024-04-19T01:04:13.715 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.715 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.716 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:13.716 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.177281+0000", 2024-04-19T01:04:13.716 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000177493 2024-04-19T01:04:13.716 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:13.716 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:13.716 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:13.716 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.716 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.716 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4260.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-19T01:04:13.716 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:03:51.154750+0000", 2024-04-19T01:04:13.716 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 22.471159943, 2024-04-19T01:04:13.716 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.022800526000000002, 2024-04-19T01:04:13.716 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:13.717 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:13.717 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:13.717 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:13.717 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4260", 2024-04-19T01:04:13.717 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/1127060541", 2024-04-19T01:04:13.717 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 141 2024-04-19T01:04:13.717 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.717 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:13.717 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.718 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:13.718 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154750+0000", 2024-04-19T01:04:13.718 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:13.718 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.718 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.718 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:13.718 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154750+0000", 2024-04-19T01:04:13.718 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:13.718 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.719 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.719 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:13.719 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154751+0000", 2024-04-19T01:04:13.719 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.7700000000000003e-07 2024-04-19T01:04:13.719 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.719 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.719 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:13.719 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154757+0000", 2024-04-19T01:04:13.719 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9669999999999998e-06 2024-04-19T01:04:13.719 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.719 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.720 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:13.720 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154767+0000", 2024-04-19T01:04:13.720 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.8349999999999999e-06 2024-04-19T01:04:13.720 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.720 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.720 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:13.720 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154774+0000", 2024-04-19T01:04:13.720 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7740000000000004e-06 2024-04-19T01:04:13.720 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.720 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.720 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:13.720 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.175219+0000", 2024-04-19T01:04:13.721 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.020444451999999998 2024-04-19T01:04:13.721 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.721 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.721 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:13.721 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.175431+0000", 2024-04-19T01:04:13.721 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021206900000000001 2024-04-19T01:04:13.721 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.721 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.721 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 5", 2024-04-19T01:04:13.721 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.175804+0000", 2024-04-19T01:04:13.721 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00037343599999999999 2024-04-19T01:04:13.721 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.722 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.722 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-04-19T01:04:13.722 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.177312+0000", 2024-04-19T01:04:13.722 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001507995 2024-04-19T01:04:13.722 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.722 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.722 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-04-19T01:04:13.722 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.177414+0000", 2024-04-19T01:04:13.723 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000101819 2024-04-19T01:04:13.723 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.723 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.723 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:13.723 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.177490+0000", 2024-04-19T01:04:13.723 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5946999999999996e-05 2024-04-19T01:04:13.723 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.723 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.723 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:13.723 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.177550+0000", 2024-04-19T01:04:13.723 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0355000000000001e-05 2024-04-19T01:04:13.723 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:13.723 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:13.724 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:13.724 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.724 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.724 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4260.0:162 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-19T01:04:13.724 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:03:51.270043+0000", 2024-04-19T01:04:13.724 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 22.355867326999999, 2024-04-19T01:04:13.724 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.017765901000000001, 2024-04-19T01:04:13.724 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:13.724 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:13.724 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:13.724 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:13.724 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4260", 2024-04-19T01:04:13.724 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/1127060541", 2024-04-19T01:04:13.725 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 162 2024-04-19T01:04:13.725 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.725 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:13.725 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.725 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:13.725 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.270043+0000", 2024-04-19T01:04:13.725 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:13.725 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.725 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.725 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:13.726 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.270043+0000", 2024-04-19T01:04:13.726 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:13.726 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.726 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.726 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:13.726 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.270043+0000", 2024-04-19T01:04:13.726 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4399999999999998e-07 2024-04-19T01:04:13.726 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.726 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.726 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:13.726 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.270050+0000", 2024-04-19T01:04:13.726 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1789999999999996e-06 2024-04-19T01:04:13.726 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.727 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.727 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:13.727 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.270061+0000", 2024-04-19T01:04:13.727 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0893000000000001e-05 2024-04-19T01:04:13.727 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.727 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.727 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:13.727 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.270068+0000", 2024-04-19T01:04:13.727 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7370000000000006e-06 2024-04-19T01:04:13.727 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.727 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.727 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:13.728 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.282291+0000", 2024-04-19T01:04:13.728 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.012222777000000001 2024-04-19T01:04:13.728 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.728 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.728 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:13.728 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.282509+0000", 2024-04-19T01:04:13.728 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000217665 2024-04-19T01:04:13.728 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.728 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.728 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 5", 2024-04-19T01:04:13.728 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.282885+0000", 2024-04-19T01:04:13.728 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000376177 2024-04-19T01:04:13.728 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.729 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.729 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-04-19T01:04:13.729 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.285626+0000", 2024-04-19T01:04:13.729 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.002741402 2024-04-19T01:04:13.729 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.729 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.729 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-04-19T01:04:13.729 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.287664+0000", 2024-04-19T01:04:13.729 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0020375189999999998 2024-04-19T01:04:13.729 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.729 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.729 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:13.730 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.287744+0000", 2024-04-19T01:04:13.730 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.0433999999999996e-05 2024-04-19T01:04:13.730 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.730 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.730 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:13.730 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.287808+0000", 2024-04-19T01:04:13.730 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4173999999999999e-05 2024-04-19T01:04:13.730 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:13.730 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:13.730 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:13.730 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.730 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.731 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4260.0:163 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-19T01:04:13.731 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:03:51.270080+0000", 2024-04-19T01:04:13.731 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 22.355830118, 2024-04-19T01:04:13.731 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.018936428000000002, 2024-04-19T01:04:13.731 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:13.731 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:13.731 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:13.731 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:13.731 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4260", 2024-04-19T01:04:13.731 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/1127060541", 2024-04-19T01:04:13.731 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 163 2024-04-19T01:04:13.731 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.732 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:13.732 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.732 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:13.732 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.270080+0000", 2024-04-19T01:04:13.732 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:13.732 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.732 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.732 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:13.732 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.270080+0000", 2024-04-19T01:04:13.732 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:13.732 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.732 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.733 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:13.733 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.270081+0000", 2024-04-19T01:04:13.733 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8299999999999996e-07 2024-04-19T01:04:13.734 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.734 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.734 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:13.735 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.270087+0000", 2024-04-19T01:04:13.735 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.9199999999999998e-06 2024-04-19T01:04:13.735 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.735 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.735 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:13.735 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.270097+0000", 2024-04-19T01:04:13.735 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.8199999999999992e-06 2024-04-19T01:04:13.735 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.735 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.735 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:13.735 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.270105+0000", 2024-04-19T01:04:13.736 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7659999999999998e-06 2024-04-19T01:04:13.736 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.736 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.736 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:13.736 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.283518+0000", 2024-04-19T01:04:13.736 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.01341326 2024-04-19T01:04:13.736 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.736 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.736 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:13.736 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.283744+0000", 2024-04-19T01:04:13.736 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00022607500000000001 2024-04-19T01:04:13.736 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.736 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.736 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 5", 2024-04-19T01:04:13.737 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.284073+0000", 2024-04-19T01:04:13.737 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00032838799999999999 2024-04-19T01:04:13.737 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.737 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.737 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-04-19T01:04:13.737 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.287904+0000", 2024-04-19T01:04:13.737 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0038312369999999999 2024-04-19T01:04:13.737 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.737 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.737 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-04-19T01:04:13.737 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.288878+0000", 2024-04-19T01:04:13.737 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00097425699999999997 2024-04-19T01:04:13.737 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.737 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.738 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:13.738 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.288953+0000", 2024-04-19T01:04:13.738 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4937000000000004e-05 2024-04-19T01:04:13.738 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.738 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.738 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:13.738 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.289016+0000", 2024-04-19T01:04:13.738 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2984999999999994e-05 2024-04-19T01:04:13.738 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:13.738 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:13.738 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:13.738 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.738 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.739 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4260.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-19T01:04:13.739 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:03:51.270137+0000", 2024-04-19T01:04:13.739 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 22.355772844000001, 2024-04-19T01:04:13.739 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.019598717000000002, 2024-04-19T01:04:13.739 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:13.739 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:13.739 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:13.739 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:13.739 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4260", 2024-04-19T01:04:13.739 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/1127060541", 2024-04-19T01:04:13.739 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 164 2024-04-19T01:04:13.740 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.740 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:13.740 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.740 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:13.740 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.270137+0000", 2024-04-19T01:04:13.740 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:13.740 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.740 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.740 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:13.740 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.270137+0000", 2024-04-19T01:04:13.740 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:13.740 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.741 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.741 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:13.741 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.270138+0000", 2024-04-19T01:04:13.741 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5999999999999991e-07 2024-04-19T01:04:13.741 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.741 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.741 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:13.741 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.270244+0000", 2024-04-19T01:04:13.741 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000106006 2024-04-19T01:04:13.741 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.741 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.741 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:13.741 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.270266+0000", 2024-04-19T01:04:13.741 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.209e-05 2024-04-19T01:04:13.742 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.742 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.742 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:13.742 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.270285+0000", 2024-04-19T01:04:13.742 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8831999999999999e-05 2024-04-19T01:04:13.742 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.742 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.742 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:13.742 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.285694+0000", 2024-04-19T01:04:13.742 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.015408948 2024-04-19T01:04:13.742 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.742 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.742 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:13.742 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.285822+0000", 2024-04-19T01:04:13.743 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012800499999999999 2024-04-19T01:04:13.743 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.743 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.743 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 5", 2024-04-19T01:04:13.743 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.286043+0000", 2024-04-19T01:04:13.743 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000221385 2024-04-19T01:04:13.743 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.743 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.743 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-04-19T01:04:13.744 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.289030+0000", 2024-04-19T01:04:13.744 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0029865099999999999 2024-04-19T01:04:13.744 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.744 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.744 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-04-19T01:04:13.744 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.289594+0000", 2024-04-19T01:04:13.744 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00056424999999999997 2024-04-19T01:04:13.744 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.744 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.744 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:13.744 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.289673+0000", 2024-04-19T01:04:13.744 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8969999999999998e-05 2024-04-19T01:04:13.744 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.745 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.745 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:13.745 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.289736+0000", 2024-04-19T01:04:13.745 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2761000000000004e-05 2024-04-19T01:04:13.745 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:13.745 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:13.745 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:13.745 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.745 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.745 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4260.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-19T01:04:13.745 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:03:51.270338+0000", 2024-04-19T01:04:13.745 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 22.355571428000001, 2024-04-19T01:04:13.745 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.020098516, 2024-04-19T01:04:13.745 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:13.746 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:13.746 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:13.746 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:13.746 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4260", 2024-04-19T01:04:13.746 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/1127060541", 2024-04-19T01:04:13.746 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 165 2024-04-19T01:04:13.746 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.746 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:13.746 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.746 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:13.746 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.270338+0000", 2024-04-19T01:04:13.746 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:13.747 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.747 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.747 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:13.747 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.270338+0000", 2024-04-19T01:04:13.747 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:13.747 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.747 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.747 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:13.747 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.270340+0000", 2024-04-19T01:04:13.747 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.57e-06 2024-04-19T01:04:13.747 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.747 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.747 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:13.747 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.270353+0000", 2024-04-19T01:04:13.748 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2602999999999999e-05 2024-04-19T01:04:13.748 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.748 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.748 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:13.748 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.270373+0000", 2024-04-19T01:04:13.748 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0072999999999999e-05 2024-04-19T01:04:13.748 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.748 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.748 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:13.748 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.270387+0000", 2024-04-19T01:04:13.748 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3906e-05 2024-04-19T01:04:13.748 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.748 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.749 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:13.749 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.287960+0000", 2024-04-19T01:04:13.749 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.017573391000000001 2024-04-19T01:04:13.749 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.749 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.749 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:13.749 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.288090+0000", 2024-04-19T01:04:13.749 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012981200000000001 2024-04-19T01:04:13.749 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.749 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.749 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 5", 2024-04-19T01:04:13.749 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.288287+0000", 2024-04-19T01:04:13.749 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019689999999999999 2024-04-19T01:04:13.749 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.750 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.750 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-04-19T01:04:13.750 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.289762+0000", 2024-04-19T01:04:13.750 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0014748949999999999 2024-04-19T01:04:13.750 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.750 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.750 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-04-19T01:04:13.750 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.290302+0000", 2024-04-19T01:04:13.750 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00054058000000000003 2024-04-19T01:04:13.750 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.750 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.750 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:13.750 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.290378+0000", 2024-04-19T01:04:13.751 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5981000000000003e-05 2024-04-19T01:04:13.751 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.751 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.751 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:13.751 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.290437+0000", 2024-04-19T01:04:13.751 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8804999999999997e-05 2024-04-19T01:04:13.751 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:13.751 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:13.751 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:13.751 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.751 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.751 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.9s1 39/31 ECSubWrite(tid=32, reqid=client.4604.0:181, at_version=39'16, trim_to=0'0, roll_forward_to=38'14))", 2024-04-19T01:04:13.751 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:09.821971+0000", 2024-04-19T01:04:13.751 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 3.8039389850000003, 2024-04-19T01:04:13.752 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.024133847999999999, 2024-04-19T01:04:13.752 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:13.752 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:13.752 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:13.752 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:13.752 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.752 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:13.752 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:09.821971+0000", 2024-04-19T01:04:13.752 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:13.752 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.752 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.752 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:13.752 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:09.821971+0000", 2024-04-19T01:04:13.753 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:13.753 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.753 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.753 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:13.753 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:09.821971+0000", 2024-04-19T01:04:13.753 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6000000000000004e-07 2024-04-19T01:04:13.753 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.753 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.753 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:13.753 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:09.822446+0000", 2024-04-19T01:04:13.753 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00047419200000000002 2024-04-19T01:04:13.753 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.753 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.753 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:13.754 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:09.822468+0000", 2024-04-19T01:04:13.754 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2507e-05 2024-04-19T01:04:13.754 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.754 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.754 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:13.754 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:09.822484+0000", 2024-04-19T01:04:13.754 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6368999999999999e-05 2024-04-19T01:04:13.754 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.754 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.754 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:13.754 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:09.822654+0000", 2024-04-19T01:04:13.755 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000169643 2024-04-19T01:04:13.755 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.755 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.755 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:13.755 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:09.822683+0000", 2024-04-19T01:04:13.755 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9232000000000001e-05 2024-04-19T01:04:13.755 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.755 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.755 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:13.755 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:09.846041+0000", 2024-04-19T01:04:13.755 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.023357902999999999 2024-04-19T01:04:13.755 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.755 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.756 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:13.756 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:09.846105+0000", 2024-04-19T01:04:13.756 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3441999999999993e-05 2024-04-19T01:04:13.756 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:13.756 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:13.756 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:13.756 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.756 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.756 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.9s1 39/31 ECSubWrite(tid=33, reqid=client.4604.0:182, at_version=39'17, trim_to=0'0, roll_forward_to=38'14))", 2024-04-19T01:04:13.756 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:09.827156+0000", 2024-04-19T01:04:13.756 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 3.7987537009999999, 2024-04-19T01:04:13.756 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.019012881999999998, 2024-04-19T01:04:13.756 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:13.756 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:13.756 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:13.757 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:13.757 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.757 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:13.757 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:09.827156+0000", 2024-04-19T01:04:13.757 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:13.757 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.757 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.757 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:13.757 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:09.827156+0000", 2024-04-19T01:04:13.757 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:13.757 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.757 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.757 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:13.757 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:09.827157+0000", 2024-04-19T01:04:13.758 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2359999999999999e-06 2024-04-19T01:04:13.758 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.758 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.758 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:13.758 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:09.827336+0000", 2024-04-19T01:04:13.758 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000178226 2024-04-19T01:04:13.758 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.758 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.758 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:13.758 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:09.827357+0000", 2024-04-19T01:04:13.758 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1795999999999999e-05 2024-04-19T01:04:13.758 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.758 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.758 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:13.758 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:09.827370+0000", 2024-04-19T01:04:13.759 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2427e-05 2024-04-19T01:04:13.759 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.759 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.759 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:13.759 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:09.836059+0000", 2024-04-19T01:04:13.759 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0086895340000000005 2024-04-19T01:04:13.759 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.759 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.759 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:13.759 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:09.836086+0000", 2024-04-19T01:04:13.759 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6659e-05 2024-04-19T01:04:13.759 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.759 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.759 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:13.759 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:09.846117+0000", 2024-04-19T01:04:13.760 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.010031022000000001 2024-04-19T01:04:13.760 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:13.760 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:13.760 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:13.760 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:09.846169+0000", 2024-04-19T01:04:13.760 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.1981999999999998e-05 2024-04-19T01:04:13.760 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:13.760 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:13.760 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:13.760 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:13.760 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:13.760 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:04:13.762 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-04-19T01:04:13.762 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:04:13.761+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:04:13.830 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.7 to 1 (10000) 2024-04-19T01:04:13.835 INFO:tasks.rados.rados.0.smithi155.stdout:86: done (1 left) 2024-04-19T01:04:13.836 INFO:tasks.rados.rados.0.smithi155.stdout:87: read oid 50 snap 0 2024-04-19T01:04:13.836 INFO:tasks.rados.rados.0.smithi155.stdout:87: expect (ObjNum 49 snap 0 seq_num 49) 2024-04-19T01:04:13.836 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 34 current snap is 6 2024-04-19T01:04:13.836 INFO:tasks.rados.rados.0.smithi155.stdout:88: seq_num 63 ranges {1458176=688128,2146304=376832} 2024-04-19T01:04:13.854 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-19T01:04:13.854 INFO:tasks.rados.rados.0.smithi155.stdout:88: writing smithi15528762-34 from 1458176 to 2146304 tid 1 2024-04-19T01:04:13.857 INFO:tasks.rados.rados.0.smithi155.stdout:88: writing smithi15528762-34 from 2146304 to 2523136 tid 2 2024-04-19T01:04:13.857 INFO:tasks.rados.rados.0.smithi155.stdout:89: read oid 41 snap -1 2024-04-19T01:04:13.857 INFO:tasks.rados.rados.0.smithi155.stdout:89: expect (ObjNum 40 snap 0 seq_num 40) 2024-04-19T01:04:13.858 INFO:tasks.rados.rados.0.smithi155.stdout:90: read oid 23 snap 0 2024-04-19T01:04:13.858 INFO:tasks.rados.rados.0.smithi155.stdout:90: expect (ObjNum 22 snap 0 seq_num 22) 2024-04-19T01:04:13.858 INFO:tasks.rados.rados.0.smithi155.stdout:91: read oid 22 snap -1 2024-04-19T01:04:13.858 INFO:tasks.rados.rados.0.smithi155.stdout:91: expect (ObjNum 21 snap 0 seq_num 21) 2024-04-19T01:04:13.858 INFO:tasks.rados.rados.0.smithi155.stdout:92: rollback oid 33 current snap is 6 2024-04-19T01:04:13.858 INFO:tasks.rados.rados.0.smithi155.stdout:rollback oid 33 to 5 2024-04-19T01:04:13.858 INFO:tasks.rados.rados.0.smithi155.stdout:84: expect (ObjNum 56 snap 5 seq_num 56) 2024-04-19T01:04:13.862 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:04:13.864+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:04:13.874 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-04-19T01:04:13.917 INFO:tasks.rados.rados.0.smithi155.stdout:87: expect (ObjNum 49 snap 0 seq_num 49) 2024-04-19T01:04:13.940 INFO:tasks.rados.rados.0.smithi155.stdout:90: expect (ObjNum 22 snap 0 seq_num 22) 2024-04-19T01:04:13.947 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:04:13.948 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:04:13.948 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-04-19T01:04:13.948 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:04:13.949 INFO:tasks.rados.rados.0.smithi155.stdout:91: expect (ObjNum 21 snap 0 seq_num 21) 2024-04-19T01:04:13.958 INFO:tasks.rados.rados.0.smithi155.stdout:92: finishing rollback tid 0 to smithi15528762-33 2024-04-19T01:04:13.958 INFO:tasks.rados.rados.0.smithi155.stdout:92: finishing rollback tid 1 to smithi15528762-33 2024-04-19T01:04:13.958 INFO:tasks.rados.rados.0.smithi155.stdout:92: finishing rollback tid 2 to smithi15528762-33 2024-04-19T01:04:13.958 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 33 v 10 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-04-19T01:04:13.959 INFO:tasks.rados.rados.0.smithi155.stdout:89: expect (ObjNum 40 snap 0 seq_num 40) 2024-04-19T01:04:13.962 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-04-19T01:04:13.963 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:04:13.959+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:04:13.976 INFO:tasks.rados.rados.0.smithi155.stdout:88: finishing write tid 1 to smithi15528762-34 2024-04-19T01:04:13.976 INFO:tasks.rados.rados.0.smithi155.stdout:88: finishing write tid 2 to smithi15528762-34 2024-04-19T01:04:13.976 INFO:tasks.rados.rados.0.smithi155.stdout:88: finishing write tid 3 to smithi15528762-34 2024-04-19T01:04:13.976 INFO:tasks.rados.rados.0.smithi155.stdout:84: done (6 left) 2024-04-19T01:04:13.977 INFO:tasks.rados.rados.0.smithi155.stdout:87: done (5 left) 2024-04-19T01:04:13.977 INFO:tasks.rados.rados.0.smithi155.stdout:89: done (4 left) 2024-04-19T01:04:13.977 INFO:tasks.rados.rados.0.smithi155.stdout:90: done (3 left) 2024-04-19T01:04:13.977 INFO:tasks.rados.rados.0.smithi155.stdout:91: done (2 left) 2024-04-19T01:04:13.977 INFO:tasks.rados.rados.0.smithi155.stdout:92: done (1 left) 2024-04-19T01:04:13.977 INFO:tasks.rados.rados.0.smithi155.stdout:93: copy_from oid 22 from oid 46 current snap is 6 2024-04-19T01:04:13.977 INFO:tasks.rados.rados.0.smithi155.stdout:88: finishing write tid 4 to smithi15528762-34 2024-04-19T01:04:13.977 INFO:tasks.rados.rados.0.smithi155.stdout:88: oid 34 updating version 0 to 43 2024-04-19T01:04:13.977 INFO:tasks.rados.rados.0.smithi155.stdout:88: oid 34 updating version 43 to 45 2024-04-19T01:04:13.977 INFO:tasks.rados.rados.0.smithi155.stdout:88: oid 34 version 45 is already newer than 44 2024-04-19T01:04:13.977 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 34 v 45 (ObjNum 63 snap 6 seq_num 63) dirty exists 2024-04-19T01:04:13.977 INFO:tasks.rados.rados.0.smithi155.stdout:88: left oid 34 (ObjNum 63 snap 6 seq_num 63) 2024-04-19T01:04:13.977 INFO:tasks.rados.rados.0.smithi155.stdout:88: done (1 left) 2024-04-19T01:04:13.977 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 1 current snap is 6 2024-04-19T01:04:13.977 INFO:tasks.rados.rados.0.smithi155.stdout:94: seq_num 64 ranges {32768=704512,737280=376832} 2024-04-19T01:04:13.980 INFO:tasks.rados.rados.0.smithi155.stdout:94: writing smithi15528762-1 from 32768 to 737280 tid 1 2024-04-19T01:04:13.982 INFO:tasks.rados.rados.0.smithi155.stdout:94: writing smithi15528762-1 from 737280 to 1114112 tid 2 2024-04-19T01:04:13.982 INFO:tasks.rados.rados.0.smithi155.stdout:95: read oid 37 snap -1 2024-04-19T01:04:13.982 INFO:tasks.rados.rados.0.smithi155.stdout:95: expect deleted 2024-04-19T01:04:13.982 INFO:tasks.rados.rados.0.smithi155.stdout:96: read oid 17 snap -1 2024-04-19T01:04:13.982 INFO:tasks.rados.rados.0.smithi155.stdout:96: expect (ObjNum 16 snap 0 seq_num 16) 2024-04-19T01:04:13.982 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 14 current snap is 6 2024-04-19T01:04:13.983 INFO:tasks.rados.rados.0.smithi155.stdout:97: seq_num 65 ranges {229376=720896,950272=376832} 2024-04-19T01:04:13.987 INFO:tasks.rados.rados.0.smithi155.stdout:97: writing smithi15528762-14 from 229376 to 950272 tid 1 2024-04-19T01:04:13.990 INFO:tasks.rados.rados.0.smithi155.stdout:97: writing smithi15528762-14 from 950272 to 1327104 tid 2 2024-04-19T01:04:13.990 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 34 current snap is 6 2024-04-19T01:04:13.990 INFO:tasks.rados.rados.0.smithi155.stdout:98: seq_num 66 ranges {2523136=737280,3260416=376832} 2024-04-19T01:04:14.009 INFO:tasks.rados.rados.0.smithi155.stdout:98: writing smithi15528762-34 from 2523136 to 3260416 tid 1 2024-04-19T01:04:14.012 INFO:tasks.rados.rados.0.smithi155.stdout:98: writing smithi15528762-34 from 3260416 to 3637248 tid 2 2024-04-19T01:04:14.012 INFO:tasks.rados.rados.0.smithi155.stdout:99: copy_from oid 2 from oid 12 current snap is 6 2024-04-19T01:04:14.013 INFO:tasks.rados.rados.0.smithi155.stdout:93: finishing copy_from racing read to smithi15528762-22 2024-04-19T01:04:14.013 INFO:tasks.rados.rados.0.smithi155.stdout:100: rollback oid 5 current snap is 6 2024-04-19T01:04:14.013 INFO:tasks.rados.rados.0.smithi155.stdout:rollback oid 5 to 0 2024-04-19T01:04:14.013 INFO:tasks.rados.rados.0.smithi155.stdout:95: done (7 left) 2024-04-19T01:04:14.013 INFO:tasks.rados.rados.0.smithi155.stdout:101: read oid 21 snap -1 2024-04-19T01:04:14.013 INFO:tasks.rados.rados.0.smithi155.stdout:101: expect (ObjNum 11 snap 0 seq_num 11) 2024-04-19T01:04:14.013 INFO:tasks.rados.rados.0.smithi155.stdout:102: snap_remove snap 2 2024-04-19T01:04:14.064 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:04:14.064+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:04:14.099 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:04:14.099 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:04:14.099 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-04-19T01:04:14.099 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-04-19T01:04:14.099 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:04:14.113 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-04-19T01:04:14.165 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:14.164+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:14.266 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:04:14.264+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:04:14.280 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:04:14.280 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-04-19T01:04:14.280 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-04-19T01:04:14.280 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-04-19T01:04:14.280 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.280 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:173 3.3s0 3:c34529b5:::smithi15528762-31:head [append 0~507904] snapc 5=[5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-19T01:04:14.281 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:08.816172+0000", 2024-04-19T01:04:14.281 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.4626916120000004, 2024-04-19T01:04:14.281 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.024750307999999999, 2024-04-19T01:04:14.281 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:14.281 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:14.281 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:14.281 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:14.281 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:04:14.281 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:14.281 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 173 2024-04-19T01:04:14.281 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.281 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:14.281 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.282 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:14.282 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816172+0000", 2024-04-19T01:04:14.282 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.282 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.282 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.282 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:14.282 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816172+0000", 2024-04-19T01:04:14.282 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.282 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.282 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.282 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:14.282 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816175+0000", 2024-04-19T01:04:14.283 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9589999999999997e-06 2024-04-19T01:04:14.283 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.283 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.283 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:14.283 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816766+0000", 2024-04-19T01:04:14.283 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00059020999999999998 2024-04-19T01:04:14.283 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.283 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.283 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:14.283 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816802+0000", 2024-04-19T01:04:14.283 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6140999999999998e-05 2024-04-19T01:04:14.284 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.284 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.284 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:14.284 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816820+0000", 2024-04-19T01:04:14.284 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7771999999999999e-05 2024-04-19T01:04:14.284 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.284 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.284 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:14.284 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816990+0000", 2024-04-19T01:04:14.284 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016993299999999999 2024-04-19T01:04:14.284 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.284 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.285 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op must wait for map", 2024-04-19T01:04:14.285 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.817016+0000", 2024-04-19T01:04:14.285 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6857000000000001e-05 2024-04-19T01:04:14.285 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.285 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.285 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:14.285 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.821660+0000", 2024-04-19T01:04:14.285 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0046440520000000001 2024-04-19T01:04:14.285 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.285 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.285 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:14.285 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.821835+0000", 2024-04-19T01:04:14.286 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000174112 2024-04-19T01:04:14.286 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.286 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.286 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:14.286 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.824020+0000", 2024-04-19T01:04:14.286 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.002185523 2024-04-19T01:04:14.286 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.286 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.286 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:14.286 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.839423+0000", 2024-04-19T01:04:14.286 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.015403261999999999 2024-04-19T01:04:14.286 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.287 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.287 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:14.287 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.840758+0000", 2024-04-19T01:04:14.287 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001335145 2024-04-19T01:04:14.287 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.287 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.287 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:14.287 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.840922+0000", 2024-04-19T01:04:14.287 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000163342 2024-04-19T01:04:14.287 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:14.287 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:14.287 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:14.287 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.288 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.288 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:174 3.3s0 3:c34529b5:::smithi15528762-31:head [append 0~376832] snapc 5=[5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-19T01:04:14.288 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:08.821669+0000", 2024-04-19T01:04:14.288 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.4571943640000002, 2024-04-19T01:04:14.288 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.019779560000000002, 2024-04-19T01:04:14.288 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:14.288 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:14.288 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:14.288 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:14.288 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:04:14.288 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:14.288 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 174 2024-04-19T01:04:14.289 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.289 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:14.289 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.289 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:14.289 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.821669+0000", 2024-04-19T01:04:14.289 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.289 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.289 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.289 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:14.289 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.821669+0000", 2024-04-19T01:04:14.289 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.289 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.289 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.290 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:14.290 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.821671+0000", 2024-04-19T01:04:14.290 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2539999999999999e-06 2024-04-19T01:04:14.290 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.290 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.290 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:14.290 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822078+0000", 2024-04-19T01:04:14.291 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00040720600000000002 2024-04-19T01:04:14.291 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.291 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.291 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:14.291 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822097+0000", 2024-04-19T01:04:14.291 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9011000000000001e-05 2024-04-19T01:04:14.291 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.291 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.291 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:14.291 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822111+0000", 2024-04-19T01:04:14.291 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4188e-05 2024-04-19T01:04:14.291 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.291 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.292 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:14.292 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.830924+0000", 2024-04-19T01:04:14.292 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0088129609999999994 2024-04-19T01:04:14.292 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.292 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.292 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:14.292 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.831084+0000", 2024-04-19T01:04:14.292 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000159889 2024-04-19T01:04:14.292 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.292 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.292 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:14.292 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.832503+0000", 2024-04-19T01:04:14.292 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001418655 2024-04-19T01:04:14.293 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.293 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.293 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:14.293 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.841099+0000", 2024-04-19T01:04:14.293 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0085957340000000007 2024-04-19T01:04:14.293 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.293 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.293 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:14.293 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.841345+0000", 2024-04-19T01:04:14.293 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00024665800000000002 2024-04-19T01:04:14.294 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.294 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.294 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:14.294 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.841448+0000", 2024-04-19T01:04:14.294 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010300400000000001 2024-04-19T01:04:14.294 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:14.294 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:14.294 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:14.294 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.295 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.295 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:175 3.3s0 3:c34529b5:::smithi15528762-31:head [setxattr _header (56) in=7b] snapc 5=[5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-19T01:04:14.295 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:08.822142+0000", 2024-04-19T01:04:14.295 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.4567213639999999, 2024-04-19T01:04:14.295 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.020224723999999999, 2024-04-19T01:04:14.295 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:14.296 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:14.296 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:14.296 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:14.296 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:04:14.296 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:14.296 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 175 2024-04-19T01:04:14.296 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.296 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:14.296 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.296 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:14.296 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822142+0000", 2024-04-19T01:04:14.296 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.297 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.297 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.297 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:14.297 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822142+0000", 2024-04-19T01:04:14.297 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.297 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.297 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.297 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:14.297 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822143+0000", 2024-04-19T01:04:14.297 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2699999999999999e-06 2024-04-19T01:04:14.297 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.297 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.298 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:14.298 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822151+0000", 2024-04-19T01:04:14.298 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4760000000000001e-06 2024-04-19T01:04:14.298 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.298 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.298 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:14.298 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822165+0000", 2024-04-19T01:04:14.298 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4138000000000001e-05 2024-04-19T01:04:14.298 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.298 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.298 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:14.298 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822178+0000", 2024-04-19T01:04:14.299 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3185e-05 2024-04-19T01:04:14.299 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.299 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.299 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:14.299 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.839463+0000", 2024-04-19T01:04:14.299 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.017285038999999999 2024-04-19T01:04:14.299 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.299 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.299 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:14.299 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.839580+0000", 2024-04-19T01:04:14.299 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000117601 2024-04-19T01:04:14.299 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.299 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.300 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:14.300 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.840161+0000", 2024-04-19T01:04:14.300 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00058043199999999997 2024-04-19T01:04:14.300 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.300 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.300 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:14.300 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.841469+0000", 2024-04-19T01:04:14.300 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001308065 2024-04-19T01:04:14.300 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.300 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.300 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:14.300 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.842264+0000", 2024-04-19T01:04:14.301 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00079508400000000005 2024-04-19T01:04:14.301 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.301 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.301 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:14.301 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.842366+0000", 2024-04-19T01:04:14.301 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000102434 2024-04-19T01:04:14.301 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:14.301 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:14.301 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:14.301 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.301 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.301 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:176 3.3s0 3:c34529b5:::smithi15528762-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-19T01:04:14.301 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:08.822193+0000", 2024-04-19T01:04:14.302 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.4566701489999998, 2024-04-19T01:04:14.302 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.023273202, 2024-04-19T01:04:14.302 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:14.302 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:14.302 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-04-19T01:04:14.302 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:14.302 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:04:14.302 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:14.302 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 176 2024-04-19T01:04:14.302 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.302 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:14.302 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.303 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:14.303 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822193+0000", 2024-04-19T01:04:14.303 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.303 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.303 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.303 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:14.303 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822193+0000", 2024-04-19T01:04:14.303 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.303 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.303 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.303 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:14.303 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822194+0000", 2024-04-19T01:04:14.303 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4200000000000004e-07 2024-04-19T01:04:14.304 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.304 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.304 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:14.304 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822198+0000", 2024-04-19T01:04:14.304 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.206e-06 2024-04-19T01:04:14.304 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.304 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.304 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:14.304 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822210+0000", 2024-04-19T01:04:14.304 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.217e-05 2024-04-19T01:04:14.304 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.304 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.305 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:14.305 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822220+0000", 2024-04-19T01:04:14.305 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0059e-05 2024-04-19T01:04:14.305 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.305 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.305 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:14.305 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.842930+0000", 2024-04-19T01:04:14.305 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.020709701 2024-04-19T01:04:14.305 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.305 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.305 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:14.305 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.843170+0000", 2024-04-19T01:04:14.305 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00024026700000000001 2024-04-19T01:04:14.306 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.306 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.306 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:14.306 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.845466+0000", 2024-04-19T01:04:14.306 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0022958570000000001 2024-04-19T01:04:14.306 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:14.306 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:14.306 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:14.306 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.306 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.306 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 38/31 ECSubWriteReply(tid=13, last_complete=38'9, committed=1, applied=1))", 2024-04-19T01:04:14.306 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:08.829564+0000", 2024-04-19T01:04:14.307 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.4492989280000002, 2024-04-19T01:04:14.307 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0094823760000000007, 2024-04-19T01:04:14.307 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:14.307 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:14.307 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:14.307 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:14.307 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.307 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:14.307 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.829564+0000", 2024-04-19T01:04:14.307 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.307 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.307 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.307 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:14.308 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.829564+0000", 2024-04-19T01:04:14.308 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.308 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.308 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.308 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:14.308 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.829566+0000", 2024-04-19T01:04:14.308 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1569999999999998e-06 2024-04-19T01:04:14.308 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.308 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.308 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:14.309 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.829576+0000", 2024-04-19T01:04:14.309 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.005e-05 2024-04-19T01:04:14.309 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.309 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.309 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:14.309 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.829605+0000", 2024-04-19T01:04:14.309 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8501999999999999e-05 2024-04-19T01:04:14.309 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.309 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.309 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:14.309 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.829620+0000", 2024-04-19T01:04:14.309 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5017e-05 2024-04-19T01:04:14.310 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.310 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.310 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:14.310 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.839023+0000", 2024-04-19T01:04:14.310 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0094034410000000002 2024-04-19T01:04:14.310 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.310 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.310 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:14.310 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.839047+0000", 2024-04-19T01:04:14.310 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3209000000000001e-05 2024-04-19T01:04:14.311 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:14.311 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:14.311 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:14.311 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.311 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.311 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 38/31 ECSubWriteReply(tid=13, last_complete=38'9, committed=1, applied=1))", 2024-04-19T01:04:14.311 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:08.830273+0000", 2024-04-19T01:04:14.311 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.4485898610000003, 2024-04-19T01:04:14.311 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0088936320000000003, 2024-04-19T01:04:14.311 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:14.311 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:14.311 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:14.312 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:14.312 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.312 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:14.312 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.830273+0000", 2024-04-19T01:04:14.312 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.312 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.312 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.312 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:14.312 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.830273+0000", 2024-04-19T01:04:14.312 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.312 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.312 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.313 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:14.313 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.830275+0000", 2024-04-19T01:04:14.313 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4619999999999999e-06 2024-04-19T01:04:14.313 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.313 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.313 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:14.313 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.830283+0000", 2024-04-19T01:04:14.313 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7959999999999996e-06 2024-04-19T01:04:14.313 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.313 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.313 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:14.313 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.830310+0000", 2024-04-19T01:04:14.313 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7036999999999999e-05 2024-04-19T01:04:14.314 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.314 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.314 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:14.314 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.830324+0000", 2024-04-19T01:04:14.314 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4857e-05 2024-04-19T01:04:14.314 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.314 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.314 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:14.314 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.839152+0000", 2024-04-19T01:04:14.314 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0088272990000000003 2024-04-19T01:04:14.314 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.314 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.315 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:14.315 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.839167+0000", 2024-04-19T01:04:14.315 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5180999999999999e-05 2024-04-19T01:04:14.315 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:14.315 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:14.315 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:14.315 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.315 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.315 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 38/31 ECSubWriteReply(tid=13, last_complete=38'9, committed=1, applied=1))", 2024-04-19T01:04:14.315 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:08.830538+0000", 2024-04-19T01:04:14.315 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.4483255980000003, 2024-04-19T01:04:14.315 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0085543189999999995, 2024-04-19T01:04:14.316 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:14.316 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:14.316 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:14.316 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:14.316 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.316 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:14.316 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.830538+0000", 2024-04-19T01:04:14.316 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.316 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.316 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.316 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:14.316 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.830538+0000", 2024-04-19T01:04:14.316 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.317 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.317 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.317 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:14.317 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.830540+0000", 2024-04-19T01:04:14.317 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.03e-06 2024-04-19T01:04:14.317 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.317 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.317 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:14.317 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.830549+0000", 2024-04-19T01:04:14.317 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6630000000000005e-06 2024-04-19T01:04:14.317 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.317 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.317 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:14.318 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.830573+0000", 2024-04-19T01:04:14.318 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3512000000000001e-05 2024-04-19T01:04:14.318 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.318 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.318 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:14.318 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.830592+0000", 2024-04-19T01:04:14.318 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8960000000000001e-05 2024-04-19T01:04:14.318 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.318 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.318 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:14.318 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.839076+0000", 2024-04-19T01:04:14.318 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0084842389999999993 2024-04-19T01:04:14.319 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.319 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.319 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:14.319 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.839092+0000", 2024-04-19T01:04:14.319 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5914999999999999e-05 2024-04-19T01:04:14.319 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:14.319 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:14.319 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:14.319 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.319 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.319 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 37/31 ECSubWriteReply(tid=13, last_complete=38'9, committed=1, applied=1))", 2024-04-19T01:04:14.319 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:08.831207+0000", 2024-04-19T01:04:14.320 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.4476558309999996, 2024-04-19T01:04:14.320 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0079223569999999997, 2024-04-19T01:04:14.320 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:14.320 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:14.320 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:14.320 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:14.321 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.321 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:14.322 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.831207+0000", 2024-04-19T01:04:14.322 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.322 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.322 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.322 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:14.323 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.831207+0000", 2024-04-19T01:04:14.323 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.323 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.323 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.323 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:14.323 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.831209+0000", 2024-04-19T01:04:14.323 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0329999999999998e-06 2024-04-19T01:04:14.324 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.324 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.324 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:14.324 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.831219+0000", 2024-04-19T01:04:14.324 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.8430000000000005e-06 2024-04-19T01:04:14.324 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.325 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.325 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:14.325 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.831242+0000", 2024-04-19T01:04:14.325 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3289999999999999e-05 2024-04-19T01:04:14.326 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.326 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.326 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:14.326 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.831257+0000", 2024-04-19T01:04:14.326 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4851999999999999e-05 2024-04-19T01:04:14.326 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.327 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.327 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:14.327 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.839115+0000", 2024-04-19T01:04:14.327 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0078572959999999997 2024-04-19T01:04:14.327 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.327 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.327 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:14.328 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.839130+0000", 2024-04-19T01:04:14.328 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5043e-05 2024-04-19T01:04:14.328 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:14.328 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:14.328 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:14.328 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.328 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.329 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 38/31 ECSubWriteReply(tid=14, last_complete=38'10, committed=1, applied=1))", 2024-04-19T01:04:14.329 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:08.836335+0000", 2024-04-19T01:04:14.329 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.4425282780000002, 2024-04-19T01:04:14.329 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0030963639999999999, 2024-04-19T01:04:14.330 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:14.330 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:14.330 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:14.330 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:14.330 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.330 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:14.331 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.836335+0000", 2024-04-19T01:04:14.331 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.331 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.331 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.331 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:14.332 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.836335+0000", 2024-04-19T01:04:14.332 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.332 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.333 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.333 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:14.333 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.836337+0000", 2024-04-19T01:04:14.333 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.88e-06 2024-04-19T01:04:14.333 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.333 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.333 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:14.334 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.836347+0000", 2024-04-19T01:04:14.334 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0467e-05 2024-04-19T01:04:14.334 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.334 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.334 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:14.334 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.836375+0000", 2024-04-19T01:04:14.335 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7772000000000001e-05 2024-04-19T01:04:14.335 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.335 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.335 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:14.335 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.836389+0000", 2024-04-19T01:04:14.335 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4260000000000001e-05 2024-04-19T01:04:14.336 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.336 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.336 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:14.336 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.839284+0000", 2024-04-19T01:04:14.336 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0028942949999999999 2024-04-19T01:04:14.336 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.337 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.337 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:14.337 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.839431+0000", 2024-04-19T01:04:14.337 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014768999999999999 2024-04-19T01:04:14.337 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:14.338 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:14.341 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:14.341 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.341 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.342 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 38/31 ECSubWriteReply(tid=14, last_complete=38'10, committed=1, applied=1))", 2024-04-19T01:04:14.342 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:08.836803+0000", 2024-04-19T01:04:14.342 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.4420601230000001, 2024-04-19T01:04:14.342 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0024004550000000001, 2024-04-19T01:04:14.342 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:14.342 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:14.342 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:14.342 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:14.342 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.342 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:14.342 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.836803+0000", 2024-04-19T01:04:14.343 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.343 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.343 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.343 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:14.343 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.836803+0000", 2024-04-19T01:04:14.343 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.343 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.343 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.343 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:14.343 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.836804+0000", 2024-04-19T01:04:14.343 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.1999999999999999e-07 2024-04-19T01:04:14.343 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.344 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.344 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:14.344 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.836809+0000", 2024-04-19T01:04:14.344 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0030000000000002e-06 2024-04-19T01:04:14.344 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.344 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.344 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:14.344 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.836826+0000", 2024-04-19T01:04:14.344 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7442999999999999e-05 2024-04-19T01:04:14.344 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.344 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.344 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:14.344 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.836836+0000", 2024-04-19T01:04:14.344 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0006999999999999e-05 2024-04-19T01:04:14.345 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.345 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.345 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:14.345 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.839188+0000", 2024-04-19T01:04:14.345 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.002352083 2024-04-19T01:04:14.345 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.345 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.345 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:14.345 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.839203+0000", 2024-04-19T01:04:14.345 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5199e-05 2024-04-19T01:04:14.345 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:14.345 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:14.345 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:14.345 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.345 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.346 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 38/31 ECSubWriteReply(tid=14, last_complete=38'10, committed=1, applied=1))", 2024-04-19T01:04:14.346 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:08.837349+0000", 2024-04-19T01:04:14.346 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.4415144299999998, 2024-04-19T01:04:14.346 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0020658690000000001, 2024-04-19T01:04:14.346 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:14.346 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:14.346 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:14.346 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:14.347 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.347 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:14.347 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.837349+0000", 2024-04-19T01:04:14.347 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.347 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.347 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.347 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:14.347 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.837349+0000", 2024-04-19T01:04:14.347 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.347 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.347 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.347 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:14.347 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.837350+0000", 2024-04-19T01:04:14.347 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7600000000000001e-06 2024-04-19T01:04:14.347 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.348 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.348 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:14.348 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.837362+0000", 2024-04-19T01:04:14.348 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.152e-05 2024-04-19T01:04:14.348 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.348 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.348 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:14.348 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.837385+0000", 2024-04-19T01:04:14.348 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3295999999999999e-05 2024-04-19T01:04:14.348 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.348 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.348 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:14.348 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.837396+0000", 2024-04-19T01:04:14.348 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0923e-05 2024-04-19T01:04:14.348 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.349 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.349 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:14.349 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.839382+0000", 2024-04-19T01:04:14.349 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001985667 2024-04-19T01:04:14.349 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.349 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.349 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:14.349 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.839415+0000", 2024-04-19T01:04:14.349 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2703000000000002e-05 2024-04-19T01:04:14.349 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:14.349 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:14.349 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:14.349 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.349 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.350 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.bs3 43/31 ECSubWrite(tid=38, reqid=osd.10.0:37, at_version=0'0, trim_to=0'0, roll_forward_to=43'14))", 2024-04-19T01:04:14.350 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:13.850120+0000", 2024-04-19T01:04:14.350 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.42874312599999997, 2024-04-19T01:04:14.350 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0042529059999999999, 2024-04-19T01:04:14.350 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:14.350 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:14.350 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:14.350 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:14.350 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.350 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:14.350 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.850120+0000", 2024-04-19T01:04:14.350 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.350 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.350 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.350 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:14.351 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.850120+0000", 2024-04-19T01:04:14.351 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.351 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.351 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.351 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:14.351 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.850121+0000", 2024-04-19T01:04:14.351 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2900000000000003e-07 2024-04-19T01:04:14.351 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.351 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.352 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:14.352 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.850134+0000", 2024-04-19T01:04:14.352 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3553999999999999e-05 2024-04-19T01:04:14.352 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.352 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.352 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:14.352 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.850146+0000", 2024-04-19T01:04:14.352 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1902000000000001e-05 2024-04-19T01:04:14.352 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.352 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.352 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:14.352 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.850155+0000", 2024-04-19T01:04:14.352 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2129999999999997e-06 2024-04-19T01:04:14.353 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.353 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.353 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:14.353 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.850434+0000", 2024-04-19T01:04:14.353 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00027879499999999998 2024-04-19T01:04:14.353 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.353 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.353 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:14.353 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.850446+0000", 2024-04-19T01:04:14.353 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.224e-05 2024-04-19T01:04:14.354 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.354 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.354 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:14.354 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.854327+0000", 2024-04-19T01:04:14.354 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0038808380000000002 2024-04-19T01:04:14.354 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.354 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.354 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:14.354 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.854373+0000", 2024-04-19T01:04:14.354 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5634999999999999e-05 2024-04-19T01:04:14.354 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:14.354 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:14.354 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:14.354 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.355 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.355 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:242 3.3s0 3:cbbd4e2f:::smithi15528762-23:2 [read 0~376832,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:14.355 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:13.859977+0000", 2024-04-19T01:04:14.355 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.41888652999999998, 2024-04-19T01:04:14.355 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0058929610000000004, 2024-04-19T01:04:14.355 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:14.355 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:14.355 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-04-19T01:04:14.355 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:14.355 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:04:14.355 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:14.355 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 242 2024-04-19T01:04:14.355 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.355 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:14.355 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.356 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:14.356 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.859977+0000", 2024-04-19T01:04:14.356 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.356 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.356 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.356 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:14.356 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.859977+0000", 2024-04-19T01:04:14.356 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.356 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.356 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.356 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:14.357 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.859979+0000", 2024-04-19T01:04:14.357 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0640000000000001e-06 2024-04-19T01:04:14.357 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.357 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.357 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:14.357 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.859986+0000", 2024-04-19T01:04:14.357 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2640000000000002e-06 2024-04-19T01:04:14.357 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.357 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.357 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:14.357 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.860003+0000", 2024-04-19T01:04:14.357 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7181e-05 2024-04-19T01:04:14.357 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.358 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.358 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:14.358 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.860011+0000", 2024-04-19T01:04:14.358 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5360000000000004e-06 2024-04-19T01:04:14.358 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.358 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.358 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:14.358 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.860096+0000", 2024-04-19T01:04:14.358 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5148999999999994e-05 2024-04-19T01:04:14.358 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.358 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.358 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:14.358 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.860245+0000", 2024-04-19T01:04:14.359 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014940999999999999 2024-04-19T01:04:14.359 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.359 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.359 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:14.359 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.865870+0000", 2024-04-19T01:04:14.359 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.005624357 2024-04-19T01:04:14.359 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:14.359 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:14.359 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:14.359 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.359 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.359 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:243 3.3s0 3:cbbd4e2f:::smithi15528762-23:2 [read 0~376832,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:14.359 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:13.860032+0000", 2024-04-19T01:04:14.359 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.41883084700000001, 2024-04-19T01:04:14.360 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0063463169999999998, 2024-04-19T01:04:14.360 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:14.360 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:14.360 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-04-19T01:04:14.360 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:14.360 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:04:14.360 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:14.360 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 243 2024-04-19T01:04:14.360 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.360 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:14.360 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.360 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:14.360 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.860032+0000", 2024-04-19T01:04:14.360 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.360 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.361 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.361 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:14.361 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.860032+0000", 2024-04-19T01:04:14.361 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.361 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.361 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.361 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:14.361 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.860033+0000", 2024-04-19T01:04:14.361 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7400000000000003e-07 2024-04-19T01:04:14.361 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.361 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.361 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:14.361 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.860039+0000", 2024-04-19T01:04:14.361 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.173e-06 2024-04-19T01:04:14.361 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.362 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.362 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:14.362 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.860050+0000", 2024-04-19T01:04:14.362 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0628000000000001e-05 2024-04-19T01:04:14.362 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.362 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.362 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:14.362 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.860060+0000", 2024-04-19T01:04:14.362 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9839999999999996e-06 2024-04-19T01:04:14.362 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.362 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.362 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:14.362 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.860977+0000", 2024-04-19T01:04:14.362 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00091721999999999995 2024-04-19T01:04:14.363 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.363 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.363 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:14.363 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.861128+0000", 2024-04-19T01:04:14.363 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000151096 2024-04-19T01:04:14.363 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.363 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.363 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:14.364 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.866379+0000", 2024-04-19T01:04:14.364 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0052506419999999998 2024-04-19T01:04:14.364 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:14.364 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:14.364 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:14.364 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.364 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.364 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:244 3.3s0 3:cbbd4e2f:::smithi15528762-23:2 [read 0~376832,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:14.364 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:13.860072+0000", 2024-04-19T01:04:14.364 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.41879127199999999, 2024-04-19T01:04:14.364 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0076976969999999999, 2024-04-19T01:04:14.364 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:14.365 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:14.365 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-04-19T01:04:14.365 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:14.365 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:04:14.365 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:14.365 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 244 2024-04-19T01:04:14.365 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.365 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:14.365 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.365 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:14.365 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.860072+0000", 2024-04-19T01:04:14.365 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.365 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.365 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.365 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:14.366 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.860072+0000", 2024-04-19T01:04:14.366 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.366 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.366 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.366 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:14.366 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.860072+0000", 2024-04-19T01:04:14.366 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2099999999999996e-07 2024-04-19T01:04:14.366 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.366 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.366 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:14.366 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.860077+0000", 2024-04-19T01:04:14.366 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2359999999999998e-06 2024-04-19T01:04:14.366 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.366 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.367 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:14.367 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.860088+0000", 2024-04-19T01:04:14.367 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1252e-05 2024-04-19T01:04:14.367 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.367 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.367 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:14.367 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.860095+0000", 2024-04-19T01:04:14.367 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.1559999999999998e-06 2024-04-19T01:04:14.367 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.368 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.368 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:14.368 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.861858+0000", 2024-04-19T01:04:14.368 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0017626600000000001 2024-04-19T01:04:14.368 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.368 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.368 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:14.368 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.861991+0000", 2024-04-19T01:04:14.368 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013294399999999999 2024-04-19T01:04:14.368 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.368 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.368 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:14.368 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.867770+0000", 2024-04-19T01:04:14.368 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0057788279999999997 2024-04-19T01:04:14.368 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:14.369 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:14.369 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:14.369 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.369 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.369 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpRead(3.3s0 43/31 ECSubRead(tid=30, to_read={3:cbbd4e2f:::smithi15528762-23:head=0,94208,0}, subchunks={3:cbbd4e2f:::smithi15528762-23:head=[0,1]}, attrs_to_read=))", 2024-04-19T01:04:14.369 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:13.860826+0000", 2024-04-19T01:04:14.369 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.41803731999999999, 2024-04-19T01:04:14.369 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00295372, 2024-04-19T01:04:14.369 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:14.369 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:14.369 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:14.369 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:14.370 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.370 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:14.370 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.860826+0000", 2024-04-19T01:04:14.370 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.370 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.370 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.370 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:14.370 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.860826+0000", 2024-04-19T01:04:14.370 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.370 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.370 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.370 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:14.370 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.860826+0000", 2024-04-19T01:04:14.371 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.371 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.371 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.371 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:14.371 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.860826+0000", 2024-04-19T01:04:14.371 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.371 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.371 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.371 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:14.371 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.860849+0000", 2024-04-19T01:04:14.371 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3625000000000002e-05 2024-04-19T01:04:14.371 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.371 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.371 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:14.371 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.860870+0000", 2024-04-19T01:04:14.372 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0437000000000001e-05 2024-04-19T01:04:14.372 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.372 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.372 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:14.372 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.862653+0000", 2024-04-19T01:04:14.372 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0017829269999999999 2024-04-19T01:04:14.372 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.372 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.372 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:14.372 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.863780+0000", 2024-04-19T01:04:14.372 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001126731 2024-04-19T01:04:14.372 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:14.372 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:14.372 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:14.372 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.373 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.373 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.8s3 43/31 ECSubWrite(tid=182, reqid=client.4604.0:248, at_version=43'8, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:04:14.373 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:13.861521+0000", 2024-04-19T01:04:14.373 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.41734185099999999, 2024-04-19T01:04:14.373 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0029438860000000002, 2024-04-19T01:04:14.373 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:14.373 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:14.373 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:14.373 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:14.373 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.373 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:14.373 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.861521+0000", 2024-04-19T01:04:14.373 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.374 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.374 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.374 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:14.374 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.861521+0000", 2024-04-19T01:04:14.374 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.374 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.374 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.375 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:14.375 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.861522+0000", 2024-04-19T01:04:14.375 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0899999999999997e-07 2024-04-19T01:04:14.375 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.375 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.375 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:14.375 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.861550+0000", 2024-04-19T01:04:14.375 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8674999999999999e-05 2024-04-19T01:04:14.375 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.375 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.375 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:14.375 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.861564+0000", 2024-04-19T01:04:14.375 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3732e-05 2024-04-19T01:04:14.375 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.376 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.376 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:14.376 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.861573+0000", 2024-04-19T01:04:14.376 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3789999999999995e-06 2024-04-19T01:04:14.376 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.376 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.376 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:14.376 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.861693+0000", 2024-04-19T01:04:14.376 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000119744 2024-04-19T01:04:14.376 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.376 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.376 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:14.376 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.861706+0000", 2024-04-19T01:04:14.376 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3315999999999999e-05 2024-04-19T01:04:14.376 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.377 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.377 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:14.377 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.864424+0000", 2024-04-19T01:04:14.377 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.002717508 2024-04-19T01:04:14.377 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.377 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.377 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:14.377 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.864465+0000", 2024-04-19T01:04:14.377 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1223000000000002e-05 2024-04-19T01:04:14.377 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:14.378 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:14.378 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:14.378 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.378 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.378 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpRead(3.3s0 43/31 ECSubRead(tid=31, to_read={3:cbbd4e2f:::smithi15528762-23:head=0,94208,0}, subchunks={3:cbbd4e2f:::smithi15528762-23:head=[0,1]}, attrs_to_read=))", 2024-04-19T01:04:14.378 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:13.861627+0000", 2024-04-19T01:04:14.378 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.41723597499999998, 2024-04-19T01:04:14.378 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0023213729999999998, 2024-04-19T01:04:14.379 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:14.379 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:14.379 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:14.379 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:14.379 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.379 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:14.379 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.861627+0000", 2024-04-19T01:04:14.379 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.379 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.379 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.379 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:14.379 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.861627+0000", 2024-04-19T01:04:14.380 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.380 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.380 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.380 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:14.380 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.861627+0000", 2024-04-19T01:04:14.380 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.380 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.380 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.380 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:14.380 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.861627+0000", 2024-04-19T01:04:14.380 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.380 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.380 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.380 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:14.381 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.861712+0000", 2024-04-19T01:04:14.381 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4476999999999998e-05 2024-04-19T01:04:14.381 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.381 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.381 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:14.381 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.861726+0000", 2024-04-19T01:04:14.381 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.393e-05 2024-04-19T01:04:14.381 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.381 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.381 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:14.381 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.863833+0000", 2024-04-19T01:04:14.381 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0021075709999999999 2024-04-19T01:04:14.381 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.381 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.382 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:14.382 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.863949+0000", 2024-04-19T01:04:14.382 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000115395 2024-04-19T01:04:14.382 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:14.382 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:14.382 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:14.382 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.382 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.382 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.8s3 43/31 ECSubWrite(tid=183, reqid=client.4604.0:249, at_version=43'9, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:04:14.382 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:13.863070+0000", 2024-04-19T01:04:14.382 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.41579281000000001, 2024-04-19T01:04:14.382 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001889855, 2024-04-19T01:04:14.382 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:14.382 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:14.382 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:14.383 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:14.383 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.383 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:14.383 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.863070+0000", 2024-04-19T01:04:14.383 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.383 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.383 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.383 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:14.383 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.863070+0000", 2024-04-19T01:04:14.383 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.383 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.383 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.383 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:14.383 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.863071+0000", 2024-04-19T01:04:14.384 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1340000000000001e-06 2024-04-19T01:04:14.384 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.384 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.384 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:14.384 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.863099+0000", 2024-04-19T01:04:14.384 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7735000000000001e-05 2024-04-19T01:04:14.384 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.384 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.384 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:14.384 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.863117+0000", 2024-04-19T01:04:14.385 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7537000000000002e-05 2024-04-19T01:04:14.385 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.385 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.385 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:14.385 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.863129+0000", 2024-04-19T01:04:14.385 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.181e-05 2024-04-19T01:04:14.385 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.385 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.385 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:14.385 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.863212+0000", 2024-04-19T01:04:14.385 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.3253999999999999e-05 2024-04-19T01:04:14.385 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.385 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.385 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:14.385 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.863225+0000", 2024-04-19T01:04:14.386 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2911000000000001e-05 2024-04-19T01:04:14.386 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.386 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.386 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:14.386 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.864925+0000", 2024-04-19T01:04:14.386 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0017004489999999999 2024-04-19T01:04:14.386 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.386 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.386 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:14.386 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.864960+0000", 2024-04-19T01:04:14.386 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5024999999999998e-05 2024-04-19T01:04:14.386 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:14.386 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:14.386 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:14.387 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.387 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.387 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpReadReply(3.3s0 43/31 ECSubReadReply(tid=32, attrs_read=0))", 2024-04-19T01:04:14.387 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:13.865526+0000", 2024-04-19T01:04:14.387 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.413337436, 2024-04-19T01:04:14.387 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0022681210000000001, 2024-04-19T01:04:14.387 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:14.387 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:14.387 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:14.387 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:14.387 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.387 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:14.388 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.865526+0000", 2024-04-19T01:04:14.388 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.388 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.388 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.388 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:14.388 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.865526+0000", 2024-04-19T01:04:14.388 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:14.388 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.388 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.388 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:14.388 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.865526+0000", 2024-04-19T01:04:14.388 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4799999999999999e-07 2024-04-19T01:04:14.389 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.389 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.389 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:14.389 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.865619+0000", 2024-04-19T01:04:14.389 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2825000000000007e-05 2024-04-19T01:04:14.389 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.389 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.389 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:14.389 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.865629+0000", 2024-04-19T01:04:14.389 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6840000000000008e-06 2024-04-19T01:04:14.389 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.389 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.389 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:14.389 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.865635+0000", 2024-04-19T01:04:14.390 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.409e-06 2024-04-19T01:04:14.390 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.390 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.390 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:14.390 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.866958+0000", 2024-04-19T01:04:14.390 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0013230329999999999 2024-04-19T01:04:14.390 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:14.390 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:14.390 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:14.390 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.867794+0000", 2024-04-19T01:04:14.390 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00083572200000000001 2024-04-19T01:04:14.390 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:14.390 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:14.390 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:14.391 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:14.391 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:14.391 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:04:14.392 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:04:14.393+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:04:14.392 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-19T01:04:14.493 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:04:14.493+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:04:14.525 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:14.525 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:04:14.525 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2024-04-19T01:04:14.525 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:14.539 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-19T01:04:14.594 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:14.595+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:14.659 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:14.659 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:04:14.659 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2024-04-19T01:04:14.659 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2024-04-19T01:04:14.659 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:14.673 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-19T01:04:14.695 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:04:14.695+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:04:14.727 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-19T01:04:14.795 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:04:14.791+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:04:14.805 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:14.805 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2024-04-19T01:04:14.805 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2024-04-19T01:04:14.805 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-04-19T01:04:14.805 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.805 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:172 3.cs0 3:356739ec:::smithi15528762-4:head [create,setxattr aSrcO`NnS[qadSjpQT[PSmfh\\PfYagpctmbQ_kXi (1753) in=40b,setxattr hratdsb^VNONN^^acUkPeXNZohcdVkmXXhdTeofs (2805) in=40b,setxattr abOo``aaqamjUn`hc`Qde]XU\\lqYVrmVdcimNs_a (35) in=40b,setxattr ZheopalZpQPnebTbena`gaWXtmaSTdQkji`aZYOi (34) in=40b,setxattr katXYockjtYtieO]`jgpSrZZ\\mq`gOYplPOVal`Y (842) in=40b,setxattr [^]lbsa\\UX[na]akS\\hYcdqU^]aWe_[U[OlaQXUY (1894) in=40b,setxattr We][_Xheab\\Ylg`q^hsi`XTX_tQS`Ng\\]aqZTnV[ (1473) in=40b,setxattr TTRSo^W`XoqVWpSjRgc[adm`ROSSVYUNlsilh\\kS (911) in=40b,setxattr bqbUQROpngcT`ndhrZQP`iraoUgs[ZZatacnZoc_ (2876) in=40b,setxattr qPPSibdg_e\\Y_V`kTQO^\\`_jr^jt\\aiadkepqYUj (3576) in=40b,setxattr q`N\\bPOqejbpq`QZVQfWgjScjfahlgXWodnRReWS (1017) in=40b,setxattr `pdT]PSaeeZ^jhWYc]iO`^tRXWcjedgQdke^adpf (7928) in=40b,setxattr matSmfe^casVRfWbNqkt_mZNWbhhh`\\Waf\\djeca (7157) in=40b,setxattr ]SWX\\hrhracj[TeSRYP[bZ[[lSSRghmfarTYpp]U (735) in=40b,setxattr WrnQrVa (13964) in=7b,setxattr PcP_]XU]rhapTs\\X`ojNmm_Qsjkkjecf`VgrdTrR (1542) in=40b,setxattr o^RO^oeZlji]\\\\UhVsnb]t_WXomaRr^higjSUShQ (1016) in=40b,setxattr Z_ofVVcfZdUV[UaRnTV_Ni^_rRNl`\\kt`]aaaoed (701) in=40b,setxattr X[m^[`bVlbgi``^mWZ]cl[S_nn[Ok\\ldS\\_h]]qk (1999) in=40b,setxattr fpmQVVnVPSmZmt`OeaT`adhoNjhpOb[`\\aPWQ^Rd (700) in=40b] snapc 5=[5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-19T01:04:14.805 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:08.804031+0000", 2024-04-19T01:04:14.805 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 6.0009278720000001, 2024-04-19T01:04:14.805 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014732528999999999, 2024-04-19T01:04:14.805 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:14.806 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:14.806 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:14.806 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:14.806 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:14.806 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:14.806 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 172 2024-04-19T01:04:14.806 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.806 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:14.806 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.806 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:14.806 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.804031+0000", 2024-04-19T01:04:14.806 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:14.806 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.806 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.807 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:14.807 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.804031+0000", 2024-04-19T01:04:14.807 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:14.807 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.807 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.807 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:14.807 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.804034+0000", 2024-04-19T01:04:14.807 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5389999999999999e-06 2024-04-19T01:04:14.807 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.807 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.807 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:14.808 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.804179+0000", 2024-04-19T01:04:14.808 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014485600000000001 2024-04-19T01:04:14.808 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.808 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.808 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:14.808 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.804208+0000", 2024-04-19T01:04:14.808 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8615999999999998e-05 2024-04-19T01:04:14.808 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.808 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.808 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:14.809 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.804223+0000", 2024-04-19T01:04:14.809 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5244e-05 2024-04-19T01:04:14.809 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.809 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.809 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:14.809 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.804362+0000", 2024-04-19T01:04:14.809 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013886000000000001 2024-04-19T01:04:14.809 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.809 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.809 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op must wait for map", 2024-04-19T01:04:14.809 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.804377+0000", 2024-04-19T01:04:14.809 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4832e-05 2024-04-19T01:04:14.809 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.810 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.810 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:14.810 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.809596+0000", 2024-04-19T01:04:14.810 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0052189250000000001 2024-04-19T01:04:14.810 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.810 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.810 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:14.810 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.809925+0000", 2024-04-19T01:04:14.810 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00032908900000000002 2024-04-19T01:04:14.810 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.810 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.810 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:14.810 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.813383+0000", 2024-04-19T01:04:14.810 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0034579929999999999 2024-04-19T01:04:14.810 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.811 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.811 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:14.811 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.815667+0000", 2024-04-19T01:04:14.811 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.002284632 2024-04-19T01:04:14.811 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.811 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.811 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:14.811 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.818639+0000", 2024-04-19T01:04:14.811 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.002972003 2024-04-19T01:04:14.811 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.811 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.811 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:14.811 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.818763+0000", 2024-04-19T01:04:14.812 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012394000000000001 2024-04-19T01:04:14.812 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:14.812 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:14.812 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:14.812 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.812 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.812 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.3s3 38/31 ECSubWrite(tid=13, reqid=client.4604.0:173, at_version=38'9, trim_to=0'0, roll_forward_to=34'7))", 2024-04-19T01:04:14.812 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:08.823945+0000", 2024-04-19T01:04:14.812 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.9810139119999999, 2024-04-19T01:04:14.812 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0057565109999999997, 2024-04-19T01:04:14.812 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:14.812 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:14.812 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:14.812 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:14.812 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.813 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:14.813 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.823945+0000", 2024-04-19T01:04:14.813 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:14.813 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.813 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.813 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:14.813 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.823945+0000", 2024-04-19T01:04:14.813 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:14.813 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.813 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.813 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:14.813 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.823945+0000", 2024-04-19T01:04:14.813 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8500000000000002e-07 2024-04-19T01:04:14.813 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.814 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.814 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:14.814 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.824342+0000", 2024-04-19T01:04:14.814 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00039621699999999998 2024-04-19T01:04:14.814 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.814 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.814 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:14.814 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.824358+0000", 2024-04-19T01:04:14.814 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6858e-05 2024-04-19T01:04:14.814 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.814 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.814 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:14.814 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.824369+0000", 2024-04-19T01:04:14.814 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1048e-05 2024-04-19T01:04:14.815 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.815 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.817 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:14.817 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.824583+0000", 2024-04-19T01:04:14.817 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000213384 2024-04-19T01:04:14.817 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.817 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.818 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:14.818 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.824601+0000", 2024-04-19T01:04:14.818 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8179999999999999e-05 2024-04-19T01:04:14.818 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.818 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.819 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:14.819 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.829657+0000", 2024-04-19T01:04:14.819 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0050557689999999999 2024-04-19T01:04:14.819 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.819 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.819 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:14.820 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.829701+0000", 2024-04-19T01:04:14.820 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4570000000000002e-05 2024-04-19T01:04:14.820 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:14.820 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:14.820 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:14.820 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.820 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.820 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.3s3 38/31 ECSubWrite(tid=14, reqid=client.4604.0:174, at_version=38'10, trim_to=0'0, roll_forward_to=34'7))", 2024-04-19T01:04:14.821 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:08.832902+0000", 2024-04-19T01:04:14.821 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.972056491, 2024-04-19T01:04:14.821 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.004101197, 2024-04-19T01:04:14.821 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:14.821 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:14.821 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:14.821 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:14.821 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.821 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:14.821 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.832902+0000", 2024-04-19T01:04:14.821 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:14.821 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.821 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.821 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:14.822 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.832902+0000", 2024-04-19T01:04:14.822 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:14.822 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.822 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.822 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:14.822 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.832904+0000", 2024-04-19T01:04:14.822 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.35e-06 2024-04-19T01:04:14.822 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.822 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.822 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:14.822 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.833080+0000", 2024-04-19T01:04:14.822 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00017685500000000001 2024-04-19T01:04:14.822 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.822 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.823 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:14.823 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.833145+0000", 2024-04-19T01:04:14.823 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4175000000000001e-05 2024-04-19T01:04:14.823 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.823 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.823 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:14.823 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.833194+0000", 2024-04-19T01:04:14.823 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.9258000000000002e-05 2024-04-19T01:04:14.823 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.823 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.823 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:14.823 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.833319+0000", 2024-04-19T01:04:14.823 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012463200000000001 2024-04-19T01:04:14.823 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.824 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.824 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:14.824 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.833332+0000", 2024-04-19T01:04:14.824 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3712e-05 2024-04-19T01:04:14.824 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.824 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.824 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:14.824 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.836957+0000", 2024-04-19T01:04:14.824 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0036249709999999998 2024-04-19T01:04:14.824 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.824 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.824 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:14.824 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.837003+0000", 2024-04-19T01:04:14.824 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6244e-05 2024-04-19T01:04:14.824 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:14.825 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:14.825 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:14.825 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.825 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.825 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:232 3.cs0 3:3233f066:::smithi15528762-50:2 [read 0~835584,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:14.825 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.838358+0000", 2024-04-19T01:04:14.825 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.96660116299999999, 2024-04-19T01:04:14.825 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011277769, 2024-04-19T01:04:14.825 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:14.825 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:14.825 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:14.825 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:14.825 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:14.825 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:14.826 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 232 2024-04-19T01:04:14.826 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.826 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:14.826 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.826 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:14.826 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838358+0000", 2024-04-19T01:04:14.827 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:14.827 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.827 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.827 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:14.827 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838358+0000", 2024-04-19T01:04:14.827 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:14.827 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.827 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.828 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:14.828 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838360+0000", 2024-04-19T01:04:14.828 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8770000000000002e-06 2024-04-19T01:04:14.828 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.828 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.828 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:14.828 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838388+0000", 2024-04-19T01:04:14.828 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7282000000000001e-05 2024-04-19T01:04:14.828 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.828 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.828 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:14.828 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838427+0000", 2024-04-19T01:04:14.828 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9242999999999997e-05 2024-04-19T01:04:14.829 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.829 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.829 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:14.829 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838444+0000", 2024-04-19T01:04:14.829 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7127000000000001e-05 2024-04-19T01:04:14.829 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.829 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.829 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:14.829 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838641+0000", 2024-04-19T01:04:14.829 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00019704499999999999 2024-04-19T01:04:14.829 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.829 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.829 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op must wait for map", 2024-04-19T01:04:14.829 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838653+0000", 2024-04-19T01:04:14.829 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2187999999999999e-05 2024-04-19T01:04:14.830 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.830 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.830 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:14.830 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.842741+0000", 2024-04-19T01:04:14.830 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0040879879999999999 2024-04-19T01:04:14.830 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.830 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.830 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:14.830 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.842898+0000", 2024-04-19T01:04:14.830 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015647299999999999 2024-04-19T01:04:14.830 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.831 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.831 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:14.831 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.849635+0000", 2024-04-19T01:04:14.831 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0067375459999999996 2024-04-19T01:04:14.831 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:14.831 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:14.831 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:14.831 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.831 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.831 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:233 3.cs0 3:3233f066:::smithi15528762-50:2 [read 0~835584,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:14.832 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.838487+0000", 2024-04-19T01:04:14.832 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.966472149, 2024-04-19T01:04:14.832 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013873264, 2024-04-19T01:04:14.832 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:14.832 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:14.832 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:14.832 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:14.832 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:14.832 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:14.833 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 233 2024-04-19T01:04:14.833 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.833 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:14.833 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.833 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:14.833 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838487+0000", 2024-04-19T01:04:14.833 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:14.833 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.833 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.833 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:14.834 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838487+0000", 2024-04-19T01:04:14.834 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:14.834 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.834 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.834 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:14.834 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838488+0000", 2024-04-19T01:04:14.834 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0380000000000001e-06 2024-04-19T01:04:14.834 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.835 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.835 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:14.835 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838495+0000", 2024-04-19T01:04:14.835 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.8299999999999996e-06 2024-04-19T01:04:14.835 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.836 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.836 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:14.836 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838509+0000", 2024-04-19T01:04:14.836 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3954000000000001e-05 2024-04-19T01:04:14.836 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.837 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.838 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:14.838 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838519+0000", 2024-04-19T01:04:14.838 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.9729999999999992e-06 2024-04-19T01:04:14.838 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.838 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.838 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:14.839 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838755+0000", 2024-04-19T01:04:14.839 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00023589999999999999 2024-04-19T01:04:14.839 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.839 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.839 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:04:14.839 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838764+0000", 2024-04-19T01:04:14.839 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.0429999999999996e-06 2024-04-19T01:04:14.840 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.840 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.840 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:14.840 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.844563+0000", 2024-04-19T01:04:14.840 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0057983749999999997 2024-04-19T01:04:14.840 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.840 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.841 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:14.841 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.844897+0000", 2024-04-19T01:04:14.841 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00033450199999999998 2024-04-19T01:04:14.841 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.841 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.842 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:14.842 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.852360+0000", 2024-04-19T01:04:14.842 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.007462649 2024-04-19T01:04:14.842 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:14.842 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:14.842 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:14.842 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.842 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.843 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:234 3.cs0 3:3233f066:::smithi15528762-50:2 [read 0~835584,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:14.843 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.838536+0000", 2024-04-19T01:04:14.843 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.96642266300000002, 2024-04-19T01:04:14.843 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.016682111999999999, 2024-04-19T01:04:14.843 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:14.843 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:14.844 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:14.844 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:14.844 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:14.844 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:14.844 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 234 2024-04-19T01:04:14.844 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.845 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:14.845 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.845 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:14.845 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838536+0000", 2024-04-19T01:04:14.845 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:14.845 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.845 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.845 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:14.846 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838536+0000", 2024-04-19T01:04:14.846 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:14.846 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.846 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.846 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:14.846 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838537+0000", 2024-04-19T01:04:14.846 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1200000000000002e-07 2024-04-19T01:04:14.846 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.846 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.846 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:14.846 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838542+0000", 2024-04-19T01:04:14.846 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3789999999999997e-06 2024-04-19T01:04:14.846 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.847 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.847 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:14.847 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838553+0000", 2024-04-19T01:04:14.847 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0986e-05 2024-04-19T01:04:14.847 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.847 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.847 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:14.847 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838564+0000", 2024-04-19T01:04:14.847 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0895e-05 2024-04-19T01:04:14.847 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.847 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.847 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:14.847 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.840056+0000", 2024-04-19T01:04:14.847 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014917229999999999 2024-04-19T01:04:14.848 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.848 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.848 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:04:14.848 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.840066+0000", 2024-04-19T01:04:14.848 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.9329999999999996e-06 2024-04-19T01:04:14.848 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.848 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.848 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:14.848 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.846974+0000", 2024-04-19T01:04:14.848 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0069077770000000004 2024-04-19T01:04:14.849 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.849 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.849 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:14.849 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.847132+0000", 2024-04-19T01:04:14.849 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015876400000000001 2024-04-19T01:04:14.849 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.849 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.849 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:14.849 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.855218+0000", 2024-04-19T01:04:14.849 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0080858430000000005 2024-04-19T01:04:14.849 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:14.849 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:14.849 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:14.849 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.850 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.850 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpReadReply(3.cs0 43/31 ECSubReadReply(tid=44, attrs_read=0))", 2024-04-19T01:04:14.850 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.849246+0000", 2024-04-19T01:04:14.850 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.95571237200000003, 2024-04-19T01:04:14.850 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0060257669999999996, 2024-04-19T01:04:14.850 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:14.851 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:14.851 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:14.851 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:14.851 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.851 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:14.851 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.849246+0000", 2024-04-19T01:04:14.851 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:14.851 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.852 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.852 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:14.852 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.849246+0000", 2024-04-19T01:04:14.852 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:14.852 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.852 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.852 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:14.852 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.849247+0000", 2024-04-19T01:04:14.853 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.9800000000000004e-07 2024-04-19T01:04:14.853 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.853 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.853 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:14.853 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.849453+0000", 2024-04-19T01:04:14.853 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00020607199999999999 2024-04-19T01:04:14.853 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.853 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.853 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:14.854 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.849465+0000", 2024-04-19T01:04:14.854 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1623999999999999e-05 2024-04-19T01:04:14.854 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.854 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.854 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:14.854 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.849473+0000", 2024-04-19T01:04:14.854 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1820000000000006e-06 2024-04-19T01:04:14.854 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.854 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.855 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:14.855 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.853988+0000", 2024-04-19T01:04:14.855 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0045149750000000001 2024-04-19T01:04:14.855 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.855 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.856 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:14.856 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.855272+0000", 2024-04-19T01:04:14.856 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001284416 2024-04-19T01:04:14.856 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:14.856 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:14.856 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:14.856 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.856 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.856 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds3 43/31 ECSubWrite(tid=36, reqid=osd.12.0:35, at_version=0'0, trim_to=0'0, roll_forward_to=43'12))", 2024-04-19T01:04:14.857 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.850430+0000", 2024-04-19T01:04:14.857 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.95452865799999997, 2024-04-19T01:04:14.857 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0039114850000000001, 2024-04-19T01:04:14.857 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:14.857 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:14.857 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:14.857 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:14.857 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.857 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:14.857 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.850430+0000", 2024-04-19T01:04:14.857 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:14.857 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.857 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.857 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:14.857 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.850430+0000", 2024-04-19T01:04:14.858 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:14.858 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.858 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.858 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:14.858 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.850431+0000", 2024-04-19T01:04:14.858 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1060000000000001e-06 2024-04-19T01:04:14.858 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.858 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.858 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:14.858 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.850447+0000", 2024-04-19T01:04:14.858 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5845000000000002e-05 2024-04-19T01:04:14.858 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.859 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.859 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:14.859 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.850463+0000", 2024-04-19T01:04:14.859 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6300999999999999e-05 2024-04-19T01:04:14.859 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.859 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.860 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:14.860 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.850495+0000", 2024-04-19T01:04:14.860 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.1965000000000003e-05 2024-04-19T01:04:14.860 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.860 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.860 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:14.860 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.850627+0000", 2024-04-19T01:04:14.860 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000131252 2024-04-19T01:04:14.861 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.861 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.861 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:14.861 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.850641+0000", 2024-04-19T01:04:14.861 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.47e-05 2024-04-19T01:04:14.861 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.861 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.861 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:14.861 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.854287+0000", 2024-04-19T01:04:14.861 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0036453470000000002 2024-04-19T01:04:14.861 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.862 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.862 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:14.862 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.854342+0000", 2024-04-19T01:04:14.862 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4969000000000002e-05 2024-04-19T01:04:14.862 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:14.862 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:14.862 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:14.862 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.862 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.862 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.3s3 43/31 ECSubWrite(tid=29, reqid=osd.15.0:28, at_version=0'0, trim_to=0'0, roll_forward_to=43'21))", 2024-04-19T01:04:14.862 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.852391+0000", 2024-04-19T01:04:14.863 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.95256775900000001, 2024-04-19T01:04:14.863 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0045986350000000002, 2024-04-19T01:04:14.863 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:14.863 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:14.863 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:14.863 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:14.863 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.863 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:14.863 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.852391+0000", 2024-04-19T01:04:14.863 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:14.864 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.864 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.864 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:14.864 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.852391+0000", 2024-04-19T01:04:14.864 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:14.864 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.864 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.864 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:14.864 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.852392+0000", 2024-04-19T01:04:14.864 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3239999999999999e-06 2024-04-19T01:04:14.864 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.864 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.864 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:14.864 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.852413+0000", 2024-04-19T01:04:14.865 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0800000000000001e-05 2024-04-19T01:04:14.865 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.865 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.865 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:14.865 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.852431+0000", 2024-04-19T01:04:14.865 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8144000000000001e-05 2024-04-19T01:04:14.865 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.865 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.865 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:14.865 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.852443+0000", 2024-04-19T01:04:14.866 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2231e-05 2024-04-19T01:04:14.866 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.866 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.866 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:14.866 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.852541+0000", 2024-04-19T01:04:14.866 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7597999999999996e-05 2024-04-19T01:04:14.866 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.866 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.866 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:14.866 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.852551+0000", 2024-04-19T01:04:14.866 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.5510000000000005e-06 2024-04-19T01:04:14.866 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.866 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.867 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:14.867 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.856947+0000", 2024-04-19T01:04:14.867 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0043961080000000001 2024-04-19T01:04:14.867 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.867 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.867 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:14.867 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.856990+0000", 2024-04-19T01:04:14.867 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2879000000000001e-05 2024-04-19T01:04:14.867 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:14.867 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:14.867 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:14.867 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.867 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.867 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.7s6 43/31 ECSubWrite(tid=9, reqid=osd.14.0:9, at_version=43'8, trim_to=0'0, roll_forward_to=41'6))", 2024-04-19T01:04:14.868 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.852850+0000", 2024-04-19T01:04:14.868 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.95210888199999999, 2024-04-19T01:04:14.868 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0050835699999999999, 2024-04-19T01:04:14.868 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:14.868 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:14.868 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:14.868 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:14.868 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.868 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:14.868 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.852850+0000", 2024-04-19T01:04:14.868 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:14.868 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.868 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.868 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:14.868 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.852850+0000", 2024-04-19T01:04:14.869 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:14.869 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.869 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.869 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:14.869 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.852850+0000", 2024-04-19T01:04:14.869 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.72e-07 2024-04-19T01:04:14.869 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.869 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.869 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:14.870 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.852891+0000", 2024-04-19T01:04:14.870 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0701999999999998e-05 2024-04-19T01:04:14.871 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.871 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.871 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:14.872 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.852911+0000", 2024-04-19T01:04:14.872 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0267000000000001e-05 2024-04-19T01:04:14.872 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.872 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.872 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:14.872 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.852957+0000", 2024-04-19T01:04:14.872 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6162000000000003e-05 2024-04-19T01:04:14.873 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.873 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.873 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:14.873 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.853100+0000", 2024-04-19T01:04:14.873 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000142776 2024-04-19T01:04:14.873 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.873 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.873 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:14.874 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.853111+0000", 2024-04-19T01:04:14.874 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0563e-05 2024-04-19T01:04:14.874 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.874 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.874 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:14.874 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.857878+0000", 2024-04-19T01:04:14.875 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0047673400000000001 2024-04-19T01:04:14.875 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.875 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.875 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:14.875 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.857933+0000", 2024-04-19T01:04:14.875 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.5387999999999999e-05 2024-04-19T01:04:14.876 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:14.876 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:14.876 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:14.876 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.876 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.877 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.2s2 43/31 ECSubWrite(tid=181, reqid=osd.11.0:180, at_version=0'0, trim_to=0'0, roll_forward_to=43'10))", 2024-04-19T01:04:14.877 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.854007+0000", 2024-04-19T01:04:14.877 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.95095215, 2024-04-19T01:04:14.877 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0039341310000000004, 2024-04-19T01:04:14.877 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:14.877 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:14.877 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:14.878 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:14.878 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.878 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:14.878 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.854007+0000", 2024-04-19T01:04:14.878 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:14.878 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.878 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.879 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:14.879 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.854007+0000", 2024-04-19T01:04:14.879 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:14.879 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.879 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.879 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:14.879 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.854008+0000", 2024-04-19T01:04:14.879 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0780000000000001e-06 2024-04-19T01:04:14.880 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.880 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.880 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:14.880 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.854022+0000", 2024-04-19T01:04:14.880 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4279e-05 2024-04-19T01:04:14.880 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.880 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.880 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:14.880 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.854040+0000", 2024-04-19T01:04:14.880 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8122e-05 2024-04-19T01:04:14.880 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.880 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.880 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:14.880 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.854051+0000", 2024-04-19T01:04:14.881 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1e-05 2024-04-19T01:04:14.881 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.881 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.881 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:14.881 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.854352+0000", 2024-04-19T01:04:14.881 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000300921 2024-04-19T01:04:14.881 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.881 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.881 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:14.881 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.854364+0000", 2024-04-19T01:04:14.881 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2128e-05 2024-04-19T01:04:14.881 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.881 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.881 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:14.881 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.857870+0000", 2024-04-19T01:04:14.882 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003505963 2024-04-19T01:04:14.882 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.882 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.882 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:14.882 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.857941+0000", 2024-04-19T01:04:14.882 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.0640000000000001e-05 2024-04-19T01:04:14.882 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:14.882 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:14.882 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:14.882 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.882 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.882 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds3 43/31 ECSubWrite(tid=38, reqid=client.4604.0:253, at_version=43'14, trim_to=0'0, roll_forward_to=43'12))", 2024-04-19T01:04:14.882 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.987301+0000", 2024-04-19T01:04:14.882 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.81765821000000005, 2024-04-19T01:04:14.883 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0078715780000000006, 2024-04-19T01:04:14.883 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:14.883 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:14.883 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:14.883 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:14.883 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.883 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:14.883 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.987301+0000", 2024-04-19T01:04:14.883 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:14.883 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.883 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.883 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:14.883 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.987301+0000", 2024-04-19T01:04:14.883 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:14.883 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.884 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.884 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:14.884 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.987302+0000", 2024-04-19T01:04:14.884 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5409999999999999e-06 2024-04-19T01:04:14.884 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.884 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.884 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:14.884 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.987425+0000", 2024-04-19T01:04:14.884 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012300599999999999 2024-04-19T01:04:14.884 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.884 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.884 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:14.884 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.987444+0000", 2024-04-19T01:04:14.884 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9117999999999999e-05 2024-04-19T01:04:14.885 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.885 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.885 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:14.885 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.987452+0000", 2024-04-19T01:04:14.885 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.0790000000000001e-06 2024-04-19T01:04:14.885 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.885 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.885 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:14.885 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.987539+0000", 2024-04-19T01:04:14.885 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.6765e-05 2024-04-19T01:04:14.885 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.885 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.885 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:14.885 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.987552+0000", 2024-04-19T01:04:14.886 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3166000000000001e-05 2024-04-19T01:04:14.886 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.886 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.886 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:14.886 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.995135+0000", 2024-04-19T01:04:14.886 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0075824159999999998 2024-04-19T01:04:14.886 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.886 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.886 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:14.886 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.995172+0000", 2024-04-19T01:04:14.887 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7487000000000002e-05 2024-04-19T01:04:14.887 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:14.887 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:14.887 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:14.887 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.887 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.887 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:263 3.fs0 3:f8a60714:::smithi15528762-14:head [append 0~720896] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:14.887 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.990535+0000", 2024-04-19T01:04:14.887 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.814423377, 2024-04-19T01:04:14.887 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.043978092000000003, 2024-04-19T01:04:14.888 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:14.888 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:14.888 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:14.888 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:14.888 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:14.888 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:14.888 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 263 2024-04-19T01:04:14.889 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.889 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:14.889 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.889 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:14.889 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.990535+0000", 2024-04-19T01:04:14.889 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:14.890 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.890 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.890 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:14.890 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.990535+0000", 2024-04-19T01:04:14.890 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:14.890 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.891 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.891 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:14.891 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.990537+0000", 2024-04-19T01:04:14.891 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4160000000000001e-06 2024-04-19T01:04:14.891 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.891 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.892 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:14.892 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991425+0000", 2024-04-19T01:04:14.892 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00088775099999999999 2024-04-19T01:04:14.892 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.892 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.892 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:14.893 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991440+0000", 2024-04-19T01:04:14.893 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5305999999999999e-05 2024-04-19T01:04:14.893 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.893 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.893 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:14.893 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991447+0000", 2024-04-19T01:04:14.894 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.4760000000000001e-06 2024-04-19T01:04:14.894 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.894 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.894 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:14.894 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991516+0000", 2024-04-19T01:04:14.894 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.9041999999999994e-05 2024-04-19T01:04:14.895 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.895 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.895 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:14.895 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991620+0000", 2024-04-19T01:04:14.895 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000103282 2024-04-19T01:04:14.895 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.895 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.896 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:14.896 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993018+0000", 2024-04-19T01:04:14.896 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001397976 2024-04-19T01:04:14.896 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.896 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.896 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:14.896 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.004440+0000", 2024-04-19T01:04:14.896 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011422261 2024-04-19T01:04:14.896 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.896 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.897 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:14.897 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.034454+0000", 2024-04-19T01:04:14.897 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.030013782999999999 2024-04-19T01:04:14.897 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.897 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.897 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:14.897 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.034513+0000", 2024-04-19T01:04:14.897 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.9799000000000001e-05 2024-04-19T01:04:14.897 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:14.897 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:14.897 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:14.897 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.897 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.897 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:264 3.fs0 3:f8a60714:::smithi15528762-14:head [append 0~376832] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:14.897 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.992666+0000", 2024-04-19T01:04:14.898 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.812292605, 2024-04-19T01:04:14.898 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.041981560000000001, 2024-04-19T01:04:14.898 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:14.898 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:14.898 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:14.898 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:14.898 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:14.898 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:14.898 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 264 2024-04-19T01:04:14.898 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.898 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:14.898 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.898 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:14.898 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.992666+0000", 2024-04-19T01:04:14.899 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:14.899 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.899 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.899 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:14.899 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.992666+0000", 2024-04-19T01:04:14.899 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:14.899 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.899 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.899 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:14.899 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.992668+0000", 2024-04-19T01:04:14.899 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.483e-06 2024-04-19T01:04:14.899 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.899 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.899 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:14.899 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993021+0000", 2024-04-19T01:04:14.900 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000352958 2024-04-19T01:04:14.900 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.900 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.900 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:14.900 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993031+0000", 2024-04-19T01:04:14.900 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0723000000000001e-05 2024-04-19T01:04:14.900 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.900 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.900 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:14.900 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993040+0000", 2024-04-19T01:04:14.900 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.507e-06 2024-04-19T01:04:14.900 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.900 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.900 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:14.900 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.000363+0000", 2024-04-19T01:04:14.901 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0073236819999999998 2024-04-19T01:04:14.901 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.901 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.901 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:14.901 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.000482+0000", 2024-04-19T01:04:14.901 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011817299999999999 2024-04-19T01:04:14.901 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.901 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.901 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:14.901 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.001332+0000", 2024-04-19T01:04:14.901 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000849886 2024-04-19T01:04:14.901 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.901 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.901 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:14.901 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.006292+0000", 2024-04-19T01:04:14.902 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0049605090000000001 2024-04-19T01:04:14.902 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.902 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.902 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:14.902 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.034602+0000", 2024-04-19T01:04:14.902 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.028310076999999999 2024-04-19T01:04:14.902 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.902 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.902 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:14.902 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.034648+0000", 2024-04-19T01:04:14.902 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5562000000000002e-05 2024-04-19T01:04:14.902 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:14.902 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:14.902 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:14.902 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.903 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.903 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:265 3.fs0 3:f8a60714:::smithi15528762-14:head [setxattr _header (56) in=7b] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:14.903 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.993067+0000", 2024-04-19T01:04:14.903 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.81189136699999997, 2024-04-19T01:04:14.903 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.041773988999999997, 2024-04-19T01:04:14.903 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:14.903 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:14.903 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:14.903 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:14.903 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:14.903 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:14.903 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 265 2024-04-19T01:04:14.903 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.904 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:14.904 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.904 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:14.904 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993067+0000", 2024-04-19T01:04:14.904 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:14.904 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.904 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.904 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:14.904 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993067+0000", 2024-04-19T01:04:14.904 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:14.904 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.904 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.904 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:14.904 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993068+0000", 2024-04-19T01:04:14.904 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2699999999999999e-07 2024-04-19T01:04:14.905 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.905 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.905 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:14.905 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993072+0000", 2024-04-19T01:04:14.905 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0539999999999996e-06 2024-04-19T01:04:14.905 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.905 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.905 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:14.905 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993080+0000", 2024-04-19T01:04:14.905 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6140000000000003e-06 2024-04-19T01:04:14.905 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.905 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.905 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:14.905 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993119+0000", 2024-04-19T01:04:14.905 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8754000000000002e-05 2024-04-19T01:04:14.906 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.906 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.906 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:14.906 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.004460+0000", 2024-04-19T01:04:14.906 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011341188 2024-04-19T01:04:14.906 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.906 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.906 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:14.906 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.004518+0000", 2024-04-19T01:04:14.906 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8306e-05 2024-04-19T01:04:14.906 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.906 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.906 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:14.906 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.004774+0000", 2024-04-19T01:04:14.907 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00025565400000000002 2024-04-19T01:04:14.907 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.907 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.907 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:14.907 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.006318+0000", 2024-04-19T01:04:14.907 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0015444409999999999 2024-04-19T01:04:14.907 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.907 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.907 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:14.907 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.034787+0000", 2024-04-19T01:04:14.907 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.028468566000000001 2024-04-19T01:04:14.907 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.907 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.907 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:14.907 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.034841+0000", 2024-04-19T01:04:14.908 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4685000000000002e-05 2024-04-19T01:04:14.908 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:14.908 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:14.908 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:14.908 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.908 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.908 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:266 3.fs0 3:f8a60714:::smithi15528762-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:14.908 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.993144+0000", 2024-04-19T01:04:14.908 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.81181431400000004, 2024-04-19T01:04:14.908 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.042235719999999997, 2024-04-19T01:04:14.908 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:14.908 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:14.908 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:14.908 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:14.909 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:14.909 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:14.909 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 266 2024-04-19T01:04:14.909 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.909 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:14.909 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.909 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:14.909 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993144+0000", 2024-04-19T01:04:14.909 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:14.909 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.909 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.909 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:14.909 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993144+0000", 2024-04-19T01:04:14.909 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:14.909 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.910 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.910 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:14.910 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993145+0000", 2024-04-19T01:04:14.910 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5099999999999999e-07 2024-04-19T01:04:14.910 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.910 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.910 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:14.910 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993149+0000", 2024-04-19T01:04:14.910 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4539999999999999e-06 2024-04-19T01:04:14.910 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.910 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.910 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:14.910 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993157+0000", 2024-04-19T01:04:14.910 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.6349999999999995e-06 2024-04-19T01:04:14.910 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.911 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.911 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:14.911 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993166+0000", 2024-04-19T01:04:14.911 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2709999999999998e-06 2024-04-19T01:04:14.911 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.911 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.911 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:14.911 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.004924+0000", 2024-04-19T01:04:14.911 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011757834 2024-04-19T01:04:14.911 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.911 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.911 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:04:14.911 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.004971+0000", 2024-04-19T01:04:14.911 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6947000000000003e-05 2024-04-19T01:04:14.911 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.912 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.912 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:14.912 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.035366+0000", 2024-04-19T01:04:14.912 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.03039501 2024-04-19T01:04:14.912 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.912 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.912 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:14.912 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.035380+0000", 2024-04-19T01:04:14.912 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3918e-05 2024-04-19T01:04:14.912 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:14.912 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:14.912 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:14.912 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.912 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.912 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds3 43/31 ECSubWrite(tid=39, reqid=client.4604.0:254, at_version=43'15, trim_to=0'0, roll_forward_to=43'12))", 2024-04-19T01:04:14.913 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.999286+0000", 2024-04-19T01:04:14.913 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.805672622, 2024-04-19T01:04:14.913 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0064635049999999996, 2024-04-19T01:04:14.913 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:14.913 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:14.913 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:14.913 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:14.913 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.913 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:14.913 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.999286+0000", 2024-04-19T01:04:14.913 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:14.913 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.913 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.913 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:14.913 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.999286+0000", 2024-04-19T01:04:14.914 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:14.914 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.914 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.914 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:14.914 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.999287+0000", 2024-04-19T01:04:14.914 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.3799999999999996e-07 2024-04-19T01:04:14.914 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.914 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.914 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:14.914 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.999341+0000", 2024-04-19T01:04:14.914 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4228000000000003e-05 2024-04-19T01:04:14.914 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.914 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.914 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:14.915 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.999356+0000", 2024-04-19T01:04:14.915 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5023e-05 2024-04-19T01:04:14.915 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.915 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.915 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:14.915 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.999367+0000", 2024-04-19T01:04:14.915 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1107000000000001e-05 2024-04-19T01:04:14.915 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.915 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.915 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:14.915 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.999449+0000", 2024-04-19T01:04:14.915 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1198000000000004e-05 2024-04-19T01:04:14.915 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.915 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.915 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:14.916 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.999460+0000", 2024-04-19T01:04:14.916 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1866e-05 2024-04-19T01:04:14.916 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.916 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.916 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:14.916 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.005708+0000", 2024-04-19T01:04:14.916 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0062479629999999996 2024-04-19T01:04:14.916 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.916 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.916 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:14.916 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.005750+0000", 2024-04-19T01:04:14.916 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1182e-05 2024-04-19T01:04:14.916 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:14.916 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:14.917 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:14.917 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.917 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.917 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 43/31 ECSubWriteReply(tid=47, last_complete=43'8, committed=1, applied=1))", 2024-04-19T01:04:14.917 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.999971+0000", 2024-04-19T01:04:14.917 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.80498771499999999, 2024-04-19T01:04:14.917 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0044699099999999997, 2024-04-19T01:04:14.917 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:14.917 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:14.917 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:14.917 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:14.917 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.917 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:14.917 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.999971+0000", 2024-04-19T01:04:14.918 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:14.918 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.918 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.918 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:14.918 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.999971+0000", 2024-04-19T01:04:14.918 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:14.918 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.918 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.918 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:14.918 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.999972+0000", 2024-04-19T01:04:14.918 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7699999999999992e-07 2024-04-19T01:04:14.918 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.918 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.918 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:14.919 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.999980+0000", 2024-04-19T01:04:14.919 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.8499999999999994e-06 2024-04-19T01:04:14.919 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.919 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.919 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:14.919 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.000009+0000", 2024-04-19T01:04:14.919 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9067000000000001e-05 2024-04-19T01:04:14.919 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.919 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.919 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:14.919 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.000017+0000", 2024-04-19T01:04:14.919 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.0839999999999992e-06 2024-04-19T01:04:14.919 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.919 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.919 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:14.920 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.004423+0000", 2024-04-19T01:04:14.920 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0044061170000000002 2024-04-19T01:04:14.920 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.920 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.920 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:14.920 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.004441+0000", 2024-04-19T01:04:14.920 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7815000000000001e-05 2024-04-19T01:04:14.920 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:14.920 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:14.920 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:14.920 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.920 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.920 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds3 43/31 ECSubWrite(tid=40, reqid=client.4604.0:255, at_version=43'16, trim_to=0'0, roll_forward_to=43'12))", 2024-04-19T01:04:14.920 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:14.001901+0000", 2024-04-19T01:04:14.921 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.80305738400000004, 2024-04-19T01:04:14.921 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0038741520000000001, 2024-04-19T01:04:14.921 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:14.921 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:14.921 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:14.921 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:14.921 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.921 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:14.921 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.001901+0000", 2024-04-19T01:04:14.921 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:14.921 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.922 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.922 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:14.922 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.001901+0000", 2024-04-19T01:04:14.922 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:14.922 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.922 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.922 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:14.922 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.001903+0000", 2024-04-19T01:04:14.922 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.283e-06 2024-04-19T01:04:14.922 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.922 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.922 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:14.922 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.001926+0000", 2024-04-19T01:04:14.922 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3062999999999999e-05 2024-04-19T01:04:14.923 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.923 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.923 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:14.923 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.001946+0000", 2024-04-19T01:04:14.923 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0021e-05 2024-04-19T01:04:14.923 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.923 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.923 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:14.923 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.001955+0000", 2024-04-19T01:04:14.923 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.6039999999999999e-06 2024-04-19T01:04:14.923 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.923 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.923 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:14.923 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.002661+0000", 2024-04-19T01:04:14.923 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00070593299999999995 2024-04-19T01:04:14.924 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.924 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.924 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:14.924 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.002672+0000", 2024-04-19T01:04:14.924 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.031e-05 2024-04-19T01:04:14.924 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.924 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.924 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:14.924 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.005754+0000", 2024-04-19T01:04:14.924 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0030827699999999999 2024-04-19T01:04:14.924 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.924 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.924 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:14.924 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.005776+0000", 2024-04-19T01:04:14.924 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1168000000000001e-05 2024-04-19T01:04:14.925 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:14.925 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:14.925 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:14.925 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.925 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.925 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds3 43/31 ECSubWrite(tid=42, reqid=client.4604.0:271, at_version=43'18, trim_to=0'0, roll_forward_to=43'14))", 2024-04-19T01:04:14.925 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:14.018665+0000", 2024-04-19T01:04:14.925 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.78629422699999996, 2024-04-19T01:04:14.925 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0035152120000000002, 2024-04-19T01:04:14.925 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:14.925 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:14.925 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:14.925 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:14.925 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.926 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:14.926 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.018665+0000", 2024-04-19T01:04:14.926 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:14.926 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.926 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.926 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:14.926 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.018665+0000", 2024-04-19T01:04:14.926 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:14.926 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.926 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.926 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:14.926 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.018665+0000", 2024-04-19T01:04:14.926 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.85e-07 2024-04-19T01:04:14.926 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.927 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.927 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:14.927 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.018719+0000", 2024-04-19T01:04:14.927 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3148e-05 2024-04-19T01:04:14.927 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.927 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.927 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:14.927 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.018749+0000", 2024-04-19T01:04:14.927 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0161000000000001e-05 2024-04-19T01:04:14.927 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.927 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.927 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:14.927 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.018759+0000", 2024-04-19T01:04:14.927 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0492999999999999e-05 2024-04-19T01:04:14.927 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.928 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.928 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:14.928 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.018852+0000", 2024-04-19T01:04:14.928 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.3189000000000005e-05 2024-04-19T01:04:14.928 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.928 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.928 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:14.928 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.018863+0000", 2024-04-19T01:04:14.928 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0574000000000001e-05 2024-04-19T01:04:14.928 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.928 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.928 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:14.928 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.022132+0000", 2024-04-19T01:04:14.928 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003268749 2024-04-19T01:04:14.928 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:14.929 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:14.929 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:14.929 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.022180+0000", 2024-04-19T01:04:14.929 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8013000000000003e-05 2024-04-19T01:04:14.929 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:14.929 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:14.929 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:14.929 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:14.929 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:14.929 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:14.932 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-19T01:04:14.932 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2024-04-19T01:04:14.933 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.8 to 1 (10000) 2024-04-19T01:04:14.934 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-04-19T01:04:14.934 INFO:tasks.rados.rados.0.smithi155.stdout:102: done (8 left) 2024-04-19T01:04:14.934 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 43 current snap is 6 2024-04-19T01:04:14.935 INFO:tasks.rados.rados.0.smithi155.stdout:103: seq_num 67 ranges {2670592=753664,3424256=376832} 2024-04-19T01:04:14.935 INFO:tasks.rados.rados.0.smithi155.stdout:103: writing smithi15528762-43 from 2670592 to 3424256 tid 1 2024-04-19T01:04:14.935 INFO:tasks.rados.rados.0.smithi155.stdout:103: writing smithi15528762-43 from 3424256 to 3801088 tid 2 2024-04-19T01:04:14.935 INFO:tasks.rados.rados.0.smithi155.stdout:104: copy_from oid 26 from oid 40 current snap is 6 2024-04-19T01:04:14.935 INFO:tasks.rados.rados.0.smithi155.stdout:105: read oid 37 snap -1 2024-04-19T01:04:14.935 INFO:tasks.rados.rados.0.smithi155.stdout:105: expect deleted 2024-04-19T01:04:14.935 INFO:tasks.rados.rados.0.smithi155.stdout:106: delete oid 8 current snap is 6 2024-04-19T01:04:14.935 INFO:tasks.rados.rados.0.smithi155.stdout:93: finishing copy_from to smithi15528762-22 2024-04-19T01:04:14.935 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 22 v 6 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-04-19T01:04:14.935 INFO:tasks.rados.rados.0.smithi155.stdout:96: expect (ObjNum 16 snap 0 seq_num 16) 2024-04-19T01:04:14.935 INFO:tasks.rados.rados.0.smithi155.stdout:94: finishing write tid 1 to smithi15528762-1 2024-04-19T01:04:14.935 INFO:tasks.rados.rados.0.smithi155.stdout:99: finishing copy_from racing read to smithi15528762-2 2024-04-19T01:04:14.935 INFO:tasks.rados.rados.0.smithi155.stdout:100: finishing rollback tid 0 to smithi15528762-5 2024-04-19T01:04:14.935 INFO:tasks.rados.rados.0.smithi155.stdout:94: finishing write tid 2 to smithi15528762-1 2024-04-19T01:04:14.935 INFO:tasks.rados.rados.0.smithi155.stdout:100: finishing rollback tid 1 to smithi15528762-5 2024-04-19T01:04:14.936 INFO:tasks.rados.rados.0.smithi155.stdout:94: finishing write tid 3 to smithi15528762-1 2024-04-19T01:04:14.936 INFO:tasks.rados.rados.0.smithi155.stdout:99: finishing copy_from to smithi15528762-2 2024-04-19T01:04:14.936 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 2 v 2 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-04-19T01:04:14.936 INFO:tasks.rados.rados.0.smithi155.stdout:100: finishing rollback tid 2 to smithi15528762-5 2024-04-19T01:04:14.936 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 5 v 18 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-04-19T01:04:14.936 INFO:tasks.rados.rados.0.smithi155.stdout:94: finishing write tid 4 to smithi15528762-1 2024-04-19T01:04:14.936 INFO:tasks.rados.rados.0.smithi155.stdout:94: oid 1 updating version 0 to 15 2024-04-19T01:04:14.936 INFO:tasks.rados.rados.0.smithi155.stdout:94: oid 1 version 15 is already newer than 14 2024-04-19T01:04:14.936 INFO:tasks.rados.rados.0.smithi155.stdout:94: oid 1 updating version 15 to 16 2024-04-19T01:04:14.936 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 1 v 16 (ObjNum 64 snap 6 seq_num 64) dirty exists 2024-04-19T01:04:14.936 INFO:tasks.rados.rados.0.smithi155.stdout:94: left oid 1 (ObjNum 64 snap 6 seq_num 64) 2024-04-19T01:04:14.936 INFO:tasks.rados.rados.0.smithi155.stdout:97: finishing write tid 1 to smithi15528762-14 2024-04-19T01:04:14.936 INFO:tasks.rados.rados.0.smithi155.stdout:97: finishing write tid 2 to smithi15528762-14 2024-04-19T01:04:14.936 INFO:tasks.rados.rados.0.smithi155.stdout:97: finishing write tid 3 to smithi15528762-14 2024-04-19T01:04:14.936 INFO:tasks.rados.rados.0.smithi155.stdout:101: expect (ObjNum 11 snap 0 seq_num 11) 2024-04-19T01:04:14.937 INFO:tasks.rados.rados.0.smithi155.stdout:97: finishing write tid 4 to smithi15528762-14 2024-04-19T01:04:14.937 INFO:tasks.rados.rados.0.smithi155.stdout:97: oid 14 updating version 0 to 8 2024-04-19T01:04:14.937 INFO:tasks.rados.rados.0.smithi155.stdout:97: oid 14 updating version 8 to 10 2024-04-19T01:04:14.937 INFO:tasks.rados.rados.0.smithi155.stdout:97: oid 14 version 10 is already newer than 9 2024-04-19T01:04:14.937 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 14 v 10 (ObjNum 65 snap 6 seq_num 65) dirty exists 2024-04-19T01:04:14.937 INFO:tasks.rados.rados.0.smithi155.stdout:97: left oid 14 (ObjNum 65 snap 6 seq_num 65) 2024-04-19T01:04:14.937 INFO:tasks.rados.rados.0.smithi155.stdout:98: finishing write tid 1 to smithi15528762-34 2024-04-19T01:04:14.937 INFO:tasks.rados.rados.0.smithi155.stdout:98: finishing write tid 2 to smithi15528762-34 2024-04-19T01:04:14.937 INFO:tasks.rados.rados.0.smithi155.stdout:98: finishing write tid 3 to smithi15528762-34 2024-04-19T01:04:14.937 INFO:tasks.rados.rados.0.smithi155.stdout:98: finishing write tid 4 to smithi15528762-34 2024-04-19T01:04:14.937 INFO:tasks.rados.rados.0.smithi155.stdout:98: oid 34 updating version 0 to 47 2024-04-19T01:04:14.937 INFO:tasks.rados.rados.0.smithi155.stdout:98: oid 34 updating version 47 to 48 2024-04-19T01:04:14.937 INFO:tasks.rados.rados.0.smithi155.stdout:98: oid 34 version 48 is already newer than 46 2024-04-19T01:04:14.937 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 34 v 48 (ObjNum 66 snap 6 seq_num 66) dirty exists 2024-04-19T01:04:14.937 INFO:tasks.rados.rados.0.smithi155.stdout:98: left oid 34 (ObjNum 66 snap 6 seq_num 66) 2024-04-19T01:04:14.938 INFO:tasks.rados.rados.0.smithi155.stdout:93: done (11 left) 2024-04-19T01:04:14.938 INFO:tasks.rados.rados.0.smithi155.stdout:94: done (10 left) 2024-04-19T01:04:14.938 INFO:tasks.rados.rados.0.smithi155.stdout:96: done (9 left) 2024-04-19T01:04:14.938 INFO:tasks.rados.rados.0.smithi155.stdout:97: done (8 left) 2024-04-19T01:04:14.938 INFO:tasks.rados.rados.0.smithi155.stdout:98: done (7 left) 2024-04-19T01:04:14.938 INFO:tasks.rados.rados.0.smithi155.stdout:99: done (6 left) 2024-04-19T01:04:14.938 INFO:tasks.rados.rados.0.smithi155.stdout:100: done (5 left) 2024-04-19T01:04:14.938 INFO:tasks.rados.rados.0.smithi155.stdout:101: done (4 left) 2024-04-19T01:04:14.938 INFO:tasks.rados.rados.0.smithi155.stdout:105: done (3 left) 2024-04-19T01:04:14.938 INFO:tasks.rados.rados.0.smithi155.stdout:106: done (2 left) 2024-04-19T01:04:14.938 INFO:tasks.rados.rados.0.smithi155.stdout:107: rmattr oid 7 current snap is 6 2024-04-19T01:04:14.938 INFO:tasks.rados.rados.0.smithi155.stdout:107: done (2 left) 2024-04-19T01:04:14.938 INFO:tasks.rados.rados.0.smithi155.stdout:108: setattr oid 17 current snap is 6 2024-04-19T01:04:14.938 INFO:tasks.rados.rados.0.smithi155.stdout:109: snap_remove snap 0 2024-04-19T01:04:14.939 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-04-19T01:04:14.939 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T01:04:14.940 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-04-19T01:04:14.940 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T01:04:14.940 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-04-19T01:04:14.940 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T01:04:14.941 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:14.932+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:14.948 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-04-19T01:04:14.949 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T01:04:14.966 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-04-19T01:04:14.966 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T01:04:15.003 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-04-19T01:04:15.003 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T01:04:15.025 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-04-19T01:04:15.025 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T01:04:15.031 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:15.031+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:15.046 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-04-19T01:04:15.046 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T01:04:15.063 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2024-04-19T01:04:15.063 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T01:04:15.080 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:15.080 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:15.084 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2024-04-19T01:04:15.084 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T01:04:15.096 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-19T01:04:15.105 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2024-04-19T01:04:15.106 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T01:04:15.126 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2024-04-19T01:04:15.127 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T01:04:15.132 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:04:15.131+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:04:15.144 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2024-04-19T01:04:15.145 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-19T01:04:15.180 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2024-04-19T01:04:15.180 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-19T01:04:15.200 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2024-04-19T01:04:15.201 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-19T01:04:15.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:04:15.199+0000 7fa90d8e4640 0 -- 172.21.15.33:0/3975323448 >> [v2:172.21.15.155:6807/741132602,v1:172.21.15.155:6810/741132602] conn(0x7fa908224080 msgr2=0x7fa908226470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-19T01:04:15.222 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2024-04-19T01:04:15.222 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-19T01:04:15.232 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:15.232 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:15.233 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:04:15.231+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:04:15.250 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-19T01:04:15.334 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:04:15.335+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:04:15.415 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:15.415 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:15.428 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-19T01:04:15.436 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:04:15.437+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:04:15.537 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:04:15.536+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:04:15.554 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:15.555 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:15.567 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-19T01:04:15.638 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:04:15.640+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:04:15.691 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:15.692 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:15.706 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-19T01:04:15.740 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:15.739+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:15.834 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:15.835 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:15.840 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:04:15.840+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:04:15.849 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.9 to 1 (10000) 2024-04-19T01:04:15.850 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-04-19T01:04:15.857 INFO:tasks.rados.rados.0.smithi155.stdout:109: done (3 left) 2024-04-19T01:04:15.858 INFO:tasks.rados.rados.0.smithi155.stdout:110: snap_create 2024-04-19T01:04:15.858 INFO:tasks.rados.rados.0.smithi155.stdout:103: finishing write tid 1 to smithi15528762-43 2024-04-19T01:04:15.858 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 17 v 35 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-04-19T01:04:15.858 INFO:tasks.rados.rados.0.smithi155.stdout:103: finishing write tid 2 to smithi15528762-43 2024-04-19T01:04:15.858 INFO:tasks.rados.rados.0.smithi155.stdout:103: finishing write tid 3 to smithi15528762-43 2024-04-19T01:04:15.858 INFO:tasks.rados.rados.0.smithi155.stdout:103: finishing write tid 4 to smithi15528762-43 2024-04-19T01:04:15.858 INFO:tasks.rados.rados.0.smithi155.stdout:103: oid 43 updating version 0 to 52 2024-04-19T01:04:15.858 INFO:tasks.rados.rados.0.smithi155.stdout:103: oid 43 version 52 is already newer than 51 2024-04-19T01:04:15.858 INFO:tasks.rados.rados.0.smithi155.stdout:103: oid 43 version 52 is already newer than 50 2024-04-19T01:04:15.858 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 43 v 52 (ObjNum 67 snap 6 seq_num 67) dirty exists 2024-04-19T01:04:15.858 INFO:tasks.rados.rados.0.smithi155.stdout:103: left oid 43 (ObjNum 67 snap 6 seq_num 67) 2024-04-19T01:04:15.858 INFO:tasks.rados.rados.0.smithi155.stdout:104: finishing copy_from racing read to smithi15528762-26 2024-04-19T01:04:15.859 INFO:tasks.rados.rados.0.smithi155.stdout:104: finishing copy_from to smithi15528762-26 2024-04-19T01:04:15.859 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 26 v 10 (ObjNum 55 snap 5 seq_num 55) dirty exists 2024-04-19T01:04:15.885 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-04-19T01:04:15.976 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:15.976 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:15.976 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:04:15.971+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:04:15.991 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-04-19T01:04:16.043 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:04:16.045+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:04:16.145 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:04:16.143+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:04:16.146 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:16.146 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:16.159 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-04-19T01:04:16.246 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:16.248+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:16.267 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:16.267 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:16.279 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-19T01:04:16.347 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:04:16.345+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:04:16.411 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:16.411 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:16.425 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-19T01:04:16.448 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:04:16.443+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:04:16.493 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.10 to 1 (10000) 2024-04-19T01:04:16.500 INFO:tasks.rados.rados.0.smithi155.stdout:103: done (3 left) 2024-04-19T01:04:16.500 INFO:tasks.rados.rados.0.smithi155.stdout:104: done (2 left) 2024-04-19T01:04:16.500 INFO:tasks.rados.rados.0.smithi155.stdout:108: done (1 left) 2024-04-19T01:04:16.500 INFO:tasks.rados.rados.0.smithi155.stdout:110: done (0 left) 2024-04-19T01:04:16.500 INFO:tasks.rados.rados.0.smithi155.stdout:111: read oid 13 snap -1 2024-04-19T01:04:16.500 INFO:tasks.rados.rados.0.smithi155.stdout:111: expect (ObjNum 12 snap 0 seq_num 12) 2024-04-19T01:04:16.500 INFO:tasks.rados.rados.0.smithi155.stdout:112: snap_create 2024-04-19T01:04:16.509 INFO:tasks.rados.rados.0.smithi155.stdout:111: expect (ObjNum 12 snap 0 seq_num 12) 2024-04-19T01:04:16.524 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-04-19T01:04:16.549 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:04:16.548+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:04:16.550 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:16.550 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:16.565 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-19T01:04:16.650 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:04:16.649+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:04:16.689 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:16.690 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:16.704 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-19T01:04:16.750 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:04:16.751+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:04:16.822 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:16.822 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:16.837 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-19T01:04:16.851 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:04:16.847+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:04:16.940 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-19T01:04:16.952 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:04:16.952+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:04:16.994 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:16.994 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:17.008 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-19T01:04:17.053 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:04:17.053+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:04:17.126 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:17.126 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:17.140 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-19T01:04:17.186 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:04:17.187+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:04:17.245 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-19T01:04:17.260 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:17.261 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:17.274 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-19T01:04:17.290 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:04:17.287+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:04:17.356 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:04:17.356+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:04:17.384 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-04-19T01:04:17.384 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T01:04:17.403 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-04-19T01:04:17.403 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T01:04:17.423 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-04-19T01:04:17.424 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T01:04:17.444 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-04-19T01:04:17.444 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T01:04:17.445 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:17.445 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:17.457 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:17.455+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:17.458 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-19T01:04:17.461 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-04-19T01:04:17.461 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T01:04:17.479 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-04-19T01:04:17.480 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T01:04:17.495 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.11 to 1 (10000) 2024-04-19T01:04:17.496 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2024-04-19T01:04:17.501 INFO:tasks.rados.rados.0.smithi155.stdout:111: done (1 left) 2024-04-19T01:04:17.502 INFO:tasks.rados.rados.0.smithi155.stdout:112: done (0 left) 2024-04-19T01:04:17.502 INFO:tasks.rados.rados.0.smithi155.stdout:113: delete oid 8 current snap is 8 2024-04-19T01:04:17.504 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-04-19T01:04:17.505 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T01:04:17.509 INFO:tasks.rados.rados.0.smithi155.stdout:113: done (0 left) 2024-04-19T01:04:17.510 INFO:tasks.rados.rados.0.smithi155.stdout:114: setattr oid 48 current snap is 8 2024-04-19T01:04:17.514 INFO:tasks.rados.rados.0.smithi155.stdout:115: rollback oid 16 current snap is 8 2024-04-19T01:04:17.514 INFO:tasks.rados.rados.0.smithi155.stdout:rollback oid 16 to 6 2024-04-19T01:04:17.514 INFO:tasks.rados.rados.0.smithi155.stdout:116: delete oid 45 current snap is 8 2024-04-19T01:04:17.518 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 12 1 2024-04-19T01:04:17.520 INFO:tasks.rados.rados.0.smithi155.stdout:116: done (2 left) 2024-04-19T01:04:17.520 INFO:tasks.rados.rados.0.smithi155.stdout:117: rollback oid 1 current snap is 8 2024-04-19T01:04:17.520 INFO:tasks.rados.rados.0.smithi155.stdout:rollback oid 1 to 5 2024-04-19T01:04:17.521 INFO:tasks.rados.rados.0.smithi155.stdout:118: delete oid 32 current snap is 8 2024-04-19T01:04:17.530 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-04-19T01:04:17.530 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T01:04:17.531 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 48 v 27 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-04-19T01:04:17.532 INFO:tasks.rados.rados.0.smithi155.stdout:117: finishing rollback tid 0 to smithi15528762-1 2024-04-19T01:04:17.533 INFO:tasks.rados.rados.0.smithi155.stdout:117: finishing rollback tid 1 to smithi15528762-1 2024-04-19T01:04:17.533 INFO:tasks.rados.rados.0.smithi155.stdout:117: finishing rollback tid 2 to smithi15528762-1 2024-04-19T01:04:17.533 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 1 v 30 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-19T01:04:17.533 INFO:tasks.rados.rados.0.smithi155.stdout:115: finishing rollback tid 0 to smithi15528762-16 2024-04-19T01:04:17.534 INFO:tasks.rados.rados.0.smithi155.stdout:115: finishing rollback tid 1 to smithi15528762-16 2024-04-19T01:04:17.535 INFO:tasks.rados.rados.0.smithi155.stdout:115: finishing rollback tid 2 to smithi15528762-16 2024-04-19T01:04:17.535 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 16 v 14 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-04-19T01:04:17.536 INFO:tasks.rados.rados.0.smithi155.stdout:114: done (3 left) 2024-04-19T01:04:17.536 INFO:tasks.rados.rados.0.smithi155.stdout:115: done (2 left) 2024-04-19T01:04:17.536 INFO:tasks.rados.rados.0.smithi155.stdout:117: done (1 left) 2024-04-19T01:04:17.536 INFO:tasks.rados.rados.0.smithi155.stdout:118: done (0 left) 2024-04-19T01:04:17.536 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 5 current snap is 8 2024-04-19T01:04:17.536 INFO:tasks.rados.rados.0.smithi155.stdout:119: seq_num 72 ranges {81920=442368,524288=720896,1245184=49152} 2024-04-19T01:04:17.544 INFO:tasks.rados.rados.0.smithi155.stdout:119: writing smithi15528762-5 from 81920 to 524288 tid 1 2024-04-19T01:04:17.548 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2024-04-19T01:04:17.548 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T01:04:17.558 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:04:17.555+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:04:17.560 INFO:tasks.rados.rados.0.smithi155.stdout:119: writing smithi15528762-5 from 524288 to 1245184 tid 2 2024-04-19T01:04:17.561 INFO:tasks.rados.rados.0.smithi155.stdout:119: writing smithi15528762-5 from 1245184 to 1294336 tid 3 2024-04-19T01:04:17.561 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 27 current snap is 8 2024-04-19T01:04:17.561 INFO:tasks.rados.rados.0.smithi155.stdout:120: seq_num 73 ranges {442368=458752,901120=475136,1376256=294912} 2024-04-19T01:04:17.567 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2024-04-19T01:04:17.568 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T01:04:17.576 INFO:tasks.rados.rados.0.smithi155.stdout:120: writing smithi15528762-27 from 442368 to 901120 tid 1 2024-04-19T01:04:17.585 INFO:tasks.rados.rados.0.smithi155.stdout:120: writing smithi15528762-27 from 901120 to 1376256 tid 2 2024-04-19T01:04:17.590 INFO:tasks.rados.rados.0.smithi155.stdout:120: writing smithi15528762-27 from 1376256 to 1671168 tid 3 2024-04-19T01:04:17.590 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 6 current snap is 8 2024-04-19T01:04:17.590 INFO:tasks.rados.rados.0.smithi155.stdout:121: seq_num 74 ranges {98304=475136,573440=622592,1196032=147456} 2024-04-19T01:04:17.591 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2024-04-19T01:04:17.591 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T01:04:17.599 INFO:tasks.rados.rados.0.smithi155.stdout:121: writing smithi15528762-6 from 98304 to 573440 tid 1 2024-04-19T01:04:17.606 INFO:tasks.rados.rados.0.smithi155.stdout:121: writing smithi15528762-6 from 573440 to 1196032 tid 2 2024-04-19T01:04:17.609 INFO:tasks.rados.rados.0.smithi155.stdout:121: writing smithi15528762-6 from 1196032 to 1343488 tid 3 2024-04-19T01:04:17.609 INFO:tasks.rados.rados.0.smithi155.stdout:122: read oid 47 snap 5 2024-04-19T01:04:17.609 INFO:tasks.rados.rados.0.smithi155.stdout:122: expect (ObjNum 46 snap 0 seq_num 46) 2024-04-19T01:04:17.609 INFO:tasks.rados.rados.0.smithi155.stdout:123: delete oid 31 current snap is 8 2024-04-19T01:04:17.609 INFO:tasks.rados.rados.0.smithi155.stdout:119: finishing write tid 1 to smithi15528762-5 2024-04-19T01:04:17.609 INFO:tasks.rados.rados.0.smithi155.stdout:119: finishing write tid 2 to smithi15528762-5 2024-04-19T01:04:17.609 INFO:tasks.rados.rados.0.smithi155.stdout:119: finishing write tid 3 to smithi15528762-5 2024-04-19T01:04:17.609 INFO:tasks.rados.rados.0.smithi155.stdout:119: finishing write tid 4 to smithi15528762-5 2024-04-19T01:04:17.609 INFO:tasks.rados.rados.0.smithi155.stdout:119: finishing write tid 5 to smithi15528762-5 2024-04-19T01:04:17.609 INFO:tasks.rados.rados.0.smithi155.stdout:119: oid 5 updating version 0 to 30 2024-04-19T01:04:17.610 INFO:tasks.rados.rados.0.smithi155.stdout:119: oid 5 version 30 is already newer than 28 2024-04-19T01:04:17.610 INFO:tasks.rados.rados.0.smithi155.stdout:119: oid 5 updating version 30 to 31 2024-04-19T01:04:17.610 INFO:tasks.rados.rados.0.smithi155.stdout:119: oid 5 version 31 is already newer than 29 2024-04-19T01:04:17.610 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 5 v 31 (ObjNum 72 snap 8 seq_num 72) dirty exists 2024-04-19T01:04:17.610 INFO:tasks.rados.rados.0.smithi155.stdout:119: left oid 5 (ObjNum 72 snap 8 seq_num 72) 2024-04-19T01:04:17.610 INFO:tasks.rados.rados.0.smithi155.stdout:120: finishing write tid 1 to smithi15528762-27 2024-04-19T01:04:17.610 INFO:tasks.rados.rados.0.smithi155.stdout:120: finishing write tid 2 to smithi15528762-27 2024-04-19T01:04:17.610 INFO:tasks.rados.rados.0.smithi155.stdout:120: finishing write tid 3 to smithi15528762-27 2024-04-19T01:04:17.610 INFO:tasks.rados.rados.0.smithi155.stdout:120: finishing write tid 4 to smithi15528762-27 2024-04-19T01:04:17.611 INFO:tasks.rados.rados.0.smithi155.stdout:120: finishing write tid 5 to smithi15528762-27 2024-04-19T01:04:17.611 INFO:tasks.rados.rados.0.smithi155.stdout:120: oid 27 updating version 0 to 19 2024-04-19T01:04:17.611 INFO:tasks.rados.rados.0.smithi155.stdout:120: oid 27 updating version 19 to 20 2024-04-19T01:04:17.611 INFO:tasks.rados.rados.0.smithi155.stdout:120: oid 27 updating version 20 to 21 2024-04-19T01:04:17.611 INFO:tasks.rados.rados.0.smithi155.stdout:120: oid 27 version 21 is already newer than 18 2024-04-19T01:04:17.611 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 27 v 21 (ObjNum 73 snap 8 seq_num 73) dirty exists 2024-04-19T01:04:17.611 INFO:tasks.rados.rados.0.smithi155.stdout:120: left oid 27 (ObjNum 73 snap 8 seq_num 73) 2024-04-19T01:04:17.615 INFO:tasks.rados.rados.0.smithi155.stdout:119: done (4 left) 2024-04-19T01:04:17.615 INFO:tasks.rados.rados.0.smithi155.stdout:120: done (3 left) 2024-04-19T01:04:17.615 INFO:tasks.rados.rados.0.smithi155.stdout:123: done (2 left) 2024-04-19T01:04:17.615 INFO:tasks.rados.rados.0.smithi155.stdout:124: setattr oid 32 current snap is 8 2024-04-19T01:04:17.619 INFO:tasks.rados.rados.0.smithi155.stdout:122: expect (ObjNum 46 snap 0 seq_num 46) 2024-04-19T01:04:17.621 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2024-04-19T01:04:17.621 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T01:04:17.639 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2024-04-19T01:04:17.640 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-19T01:04:17.645 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:17.646 INFO:teuthology.orchestra.run.smithi110.stdout: "size": 20, 2024-04-19T01:04:17.646 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 600, 2024-04-19T01:04:17.646 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [ 2024-04-19T01:04:17.646 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.646 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 39/31 ECSubWrite(tid=32, reqid=client.4604.0:181, at_version=39'16, trim_to=0'0, roll_forward_to=38'14))", 2024-04-19T01:04:17.646 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:09.820805+0000", 2024-04-19T01:04:17.646 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 7.8261416150000001, 2024-04-19T01:04:17.646 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.013829808000000001, 2024-04-19T01:04:17.646 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:17.646 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:17.646 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:17.646 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:17.647 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.647 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:17.647 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.820805+0000", 2024-04-19T01:04:17.647 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:17.647 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.647 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.647 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:17.647 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.820805+0000", 2024-04-19T01:04:17.647 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:17.648 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.648 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.648 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:17.648 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.820806+0000", 2024-04-19T01:04:17.648 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.2699999999999999e-07 2024-04-19T01:04:17.648 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.648 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.648 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:17.648 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.821061+0000", 2024-04-19T01:04:17.648 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000255133 2024-04-19T01:04:17.648 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.648 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.648 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:17.648 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.821256+0000", 2024-04-19T01:04:17.649 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00019562400000000001 2024-04-19T01:04:17.649 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.649 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.649 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:17.649 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.821283+0000", 2024-04-19T01:04:17.649 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6712000000000001e-05 2024-04-19T01:04:17.649 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.649 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.649 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:17.649 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.821504+0000", 2024-04-19T01:04:17.649 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00022120199999999999 2024-04-19T01:04:17.649 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.649 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.649 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:17.649 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.821557+0000", 2024-04-19T01:04:17.650 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.2534000000000002e-05 2024-04-19T01:04:17.650 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.650 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.650 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:17.651 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.834585+0000", 2024-04-19T01:04:17.651 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.013027617999999999 2024-04-19T01:04:17.651 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.651 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.651 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:17.651 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.834635+0000", 2024-04-19T01:04:17.652 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.0257999999999999e-05 2024-04-19T01:04:17.652 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:17.652 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:17.652 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:17.652 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.652 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.652 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 39/31 ECSubWrite(tid=33, reqid=client.4604.0:182, at_version=39'17, trim_to=0'0, roll_forward_to=38'14))", 2024-04-19T01:04:17.652 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:09.827672+0000", 2024-04-19T01:04:17.652 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 7.819274278, 2024-04-19T01:04:17.652 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0073674639999999998, 2024-04-19T01:04:17.652 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:17.652 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:17.652 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:17.652 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:17.653 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.653 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:17.653 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.827672+0000", 2024-04-19T01:04:17.653 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:17.653 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.653 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.653 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:17.653 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.827672+0000", 2024-04-19T01:04:17.653 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:17.653 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.653 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.653 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:17.653 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.827673+0000", 2024-04-19T01:04:17.653 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.17e-06 2024-04-19T01:04:17.653 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.654 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.654 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:17.654 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.827757+0000", 2024-04-19T01:04:17.654 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.3929000000000002e-05 2024-04-19T01:04:17.654 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.654 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.654 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:17.654 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.827779+0000", 2024-04-19T01:04:17.654 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1509999999999999e-05 2024-04-19T01:04:17.654 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.654 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.654 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:17.654 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.827792+0000", 2024-04-19T01:04:17.654 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.3021e-05 2024-04-19T01:04:17.655 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.655 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.655 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:17.655 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.832258+0000", 2024-04-19T01:04:17.655 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0044660740000000004 2024-04-19T01:04:17.655 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.655 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.656 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:17.656 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.832264+0000", 2024-04-19T01:04:17.656 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.5679999999999999e-06 2024-04-19T01:04:17.656 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.656 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.656 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:17.656 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.835020+0000", 2024-04-19T01:04:17.656 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0027565430000000002 2024-04-19T01:04:17.656 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.656 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.656 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:17.656 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.835040+0000", 2024-04-19T01:04:17.656 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9649000000000001e-05 2024-04-19T01:04:17.656 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:17.657 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:17.657 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:17.657 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.657 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.657 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 43/31 ECSubWrite(tid=70, reqid=client.4604.0:235, at_version=43'43, trim_to=0'0, roll_forward_to=41'37))", 2024-04-19T01:04:17.657 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.870038+0000", 2024-04-19T01:04:17.657 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 3.7769082530000002, 2024-04-19T01:04:17.657 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.020219719000000001, 2024-04-19T01:04:17.657 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:17.657 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:17.657 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:17.657 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:17.657 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.657 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:17.658 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.870038+0000", 2024-04-19T01:04:17.658 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:17.658 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.658 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.658 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:17.658 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.870038+0000", 2024-04-19T01:04:17.658 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:17.658 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.658 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.658 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:17.658 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.870040+0000", 2024-04-19T01:04:17.658 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.81e-06 2024-04-19T01:04:17.658 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.658 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.658 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:17.659 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.870453+0000", 2024-04-19T01:04:17.659 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000412965 2024-04-19T01:04:17.659 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.659 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.659 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:17.659 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.870494+0000", 2024-04-19T01:04:17.659 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.0938e-05 2024-04-19T01:04:17.659 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.659 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.659 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:17.659 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.870510+0000", 2024-04-19T01:04:17.659 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5458999999999999e-05 2024-04-19T01:04:17.659 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.659 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.660 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:17.660 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.870799+0000", 2024-04-19T01:04:17.660 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00028974699999999998 2024-04-19T01:04:17.660 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.660 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.660 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:17.660 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.870832+0000", 2024-04-19T01:04:17.660 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.2434999999999997e-05 2024-04-19T01:04:17.660 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.660 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.660 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:17.660 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.890185+0000", 2024-04-19T01:04:17.660 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.019353467999999999 2024-04-19T01:04:17.660 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.660 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.661 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:17.661 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.890258+0000", 2024-04-19T01:04:17.661 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.2897000000000004e-05 2024-04-19T01:04:17.661 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:17.661 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:17.661 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:17.661 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.661 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.661 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 43/31 ECSubWrite(tid=72, reqid=client.4604.0:236, at_version=43'44, trim_to=0'0, roll_forward_to=41'37))", 2024-04-19T01:04:17.661 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.890428+0000", 2024-04-19T01:04:17.661 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 3.7565183490000003, 2024-04-19T01:04:17.661 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0076663950000000003, 2024-04-19T01:04:17.661 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:17.661 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:17.662 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:17.662 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:17.662 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.662 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:17.662 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.890428+0000", 2024-04-19T01:04:17.662 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:17.662 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.662 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.662 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:17.662 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.890428+0000", 2024-04-19T01:04:17.662 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:17.662 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.662 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.662 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:17.663 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.890430+0000", 2024-04-19T01:04:17.663 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0040000000000002e-06 2024-04-19T01:04:17.663 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.663 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.663 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:17.663 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.890673+0000", 2024-04-19T01:04:17.663 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000242425 2024-04-19T01:04:17.663 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.663 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.663 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:17.663 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.890711+0000", 2024-04-19T01:04:17.663 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.8717999999999998e-05 2024-04-19T01:04:17.663 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.663 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.664 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:17.664 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.890735+0000", 2024-04-19T01:04:17.664 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4006e-05 2024-04-19T01:04:17.664 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.664 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.664 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:17.664 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.890909+0000", 2024-04-19T01:04:17.664 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00017384200000000001 2024-04-19T01:04:17.664 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.664 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.664 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:17.664 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.890938+0000", 2024-04-19T01:04:17.665 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.8958999999999998e-05 2024-04-19T01:04:17.665 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.665 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.665 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:17.665 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.898014+0000", 2024-04-19T01:04:17.665 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0070753470000000001 2024-04-19T01:04:17.665 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.665 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.666 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:17.666 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.898095+0000", 2024-04-19T01:04:17.666 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.1094000000000007e-05 2024-04-19T01:04:17.666 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:17.666 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:17.666 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:17.666 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.666 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.666 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.ds2 43/31 ECSubWrite(tid=38, reqid=client.4604.0:253, at_version=43'14, trim_to=0'0, roll_forward_to=43'12))", 2024-04-19T01:04:17.666 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.992478+0000", 2024-04-19T01:04:17.666 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 3.6544690480000002, 2024-04-19T01:04:17.666 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.025708154, 2024-04-19T01:04:17.667 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:17.667 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:17.667 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:17.667 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:17.667 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.667 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:17.667 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.992478+0000", 2024-04-19T01:04:17.667 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:17.667 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.667 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.667 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:17.667 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.992478+0000", 2024-04-19T01:04:17.667 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:17.667 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.667 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.668 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:17.668 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.992480+0000", 2024-04-19T01:04:17.668 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9870000000000002e-06 2024-04-19T01:04:17.668 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.668 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.668 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:17.668 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.992860+0000", 2024-04-19T01:04:17.668 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00038019600000000002 2024-04-19T01:04:17.668 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.668 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.668 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:17.668 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.992891+0000", 2024-04-19T01:04:17.668 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.0982000000000003e-05 2024-04-19T01:04:17.668 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.669 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.669 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:17.669 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.992906+0000", 2024-04-19T01:04:17.669 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5503e-05 2024-04-19T01:04:17.669 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.669 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.669 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:17.669 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.993064+0000", 2024-04-19T01:04:17.669 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015782900000000001 2024-04-19T01:04:17.669 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.669 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.669 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:17.669 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.993111+0000", 2024-04-19T01:04:17.669 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.7138000000000002e-05 2024-04-19T01:04:17.669 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.670 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.670 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:17.670 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.018092+0000", 2024-04-19T01:04:17.670 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.024980822999999999 2024-04-19T01:04:17.670 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.670 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.670 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:17.670 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.018186+0000", 2024-04-19T01:04:17.670 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.3696000000000006e-05 2024-04-19T01:04:17.670 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:17.670 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:17.671 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:17.671 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.671 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.671 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.ds2 43/31 ECSubWrite(tid=39, reqid=client.4604.0:254, at_version=43'15, trim_to=0'0, roll_forward_to=43'12))", 2024-04-19T01:04:17.671 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.996882+0000", 2024-04-19T01:04:17.671 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 3.650064929, 2024-04-19T01:04:17.671 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.032356653999999999, 2024-04-19T01:04:17.671 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:17.671 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:17.671 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:17.671 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:17.671 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.671 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:17.671 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.996882+0000", 2024-04-19T01:04:17.672 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:17.672 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.672 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.672 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:17.672 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.996882+0000", 2024-04-19T01:04:17.672 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:17.672 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.672 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.672 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:17.672 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.996884+0000", 2024-04-19T01:04:17.672 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2029999999999999e-06 2024-04-19T01:04:17.672 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.672 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.672 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:17.673 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.997205+0000", 2024-04-19T01:04:17.673 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00032076899999999999 2024-04-19T01:04:17.673 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.673 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.673 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:17.673 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.997249+0000", 2024-04-19T01:04:17.673 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.4654000000000003e-05 2024-04-19T01:04:17.673 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.673 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.673 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:17.673 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.997273+0000", 2024-04-19T01:04:17.673 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3589000000000001e-05 2024-04-19T01:04:17.673 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.673 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.674 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:17.674 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.007691+0000", 2024-04-19T01:04:17.674 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.010418495999999999 2024-04-19T01:04:17.674 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.674 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.674 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:17.674 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.007717+0000", 2024-04-19T01:04:17.674 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5876000000000001e-05 2024-04-19T01:04:17.674 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.674 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.674 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:17.674 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.029151+0000", 2024-04-19T01:04:17.674 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.021433651000000001 2024-04-19T01:04:17.674 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.675 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.675 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:17.675 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.029238+0000", 2024-04-19T01:04:17.675 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.7416000000000005e-05 2024-04-19T01:04:17.675 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:17.675 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:17.675 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:17.675 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.675 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.675 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.ds2 43/31 ECSubWrite(tid=40, reqid=client.4604.0:255, at_version=43'16, trim_to=0'0, roll_forward_to=43'12))", 2024-04-19T01:04:17.675 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.002736+0000", 2024-04-19T01:04:17.675 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 3.64421069, 2024-04-19T01:04:17.676 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.026570379000000002, 2024-04-19T01:04:17.676 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:17.676 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:17.676 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:17.676 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:17.676 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.676 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:17.677 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.002736+0000", 2024-04-19T01:04:17.677 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:17.677 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.677 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.677 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:17.677 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.002736+0000", 2024-04-19T01:04:17.677 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:17.677 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.677 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.677 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:17.677 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.002738+0000", 2024-04-19T01:04:17.677 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0899999999999999e-06 2024-04-19T01:04:17.677 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.677 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.678 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:17.678 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.002782+0000", 2024-04-19T01:04:17.678 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.3611e-05 2024-04-19T01:04:17.678 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.678 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.678 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:17.678 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.002823+0000", 2024-04-19T01:04:17.678 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.0954999999999997e-05 2024-04-19T01:04:17.678 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.678 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.678 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:17.678 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.002845+0000", 2024-04-19T01:04:17.678 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2775000000000001e-05 2024-04-19T01:04:17.679 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.679 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.679 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:17.679 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.019866+0000", 2024-04-19T01:04:17.679 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.017021162999999999 2024-04-19T01:04:17.679 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.679 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.679 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:17.679 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.019893+0000", 2024-04-19T01:04:17.679 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6176000000000001e-05 2024-04-19T01:04:17.679 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.679 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.679 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:17.679 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.029258+0000", 2024-04-19T01:04:17.679 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0093652990000000005 2024-04-19T01:04:17.680 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.680 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.681 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:17.681 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.029306+0000", 2024-04-19T01:04:17.681 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.8310000000000003e-05 2024-04-19T01:04:17.681 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:17.681 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:17.681 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:17.681 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.681 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.681 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.ds2 43/31 ECSubWrite(tid=42, reqid=client.4604.0:271, at_version=43'18, trim_to=0'0, roll_forward_to=43'14))", 2024-04-19T01:04:17.681 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.019519+0000", 2024-04-19T01:04:17.681 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 3.6274279840000001, 2024-04-19T01:04:17.682 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.010349381, 2024-04-19T01:04:17.682 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:17.682 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:17.682 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:17.682 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:17.682 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.682 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:17.682 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.019519+0000", 2024-04-19T01:04:17.682 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:17.682 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.682 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.682 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:17.682 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.019519+0000", 2024-04-19T01:04:17.682 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:17.683 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.683 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.683 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:17.683 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.019520+0000", 2024-04-19T01:04:17.683 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.424e-06 2024-04-19T01:04:17.683 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.683 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.683 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:17.683 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.019678+0000", 2024-04-19T01:04:17.683 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015818599999999999 2024-04-19T01:04:17.683 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.683 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.683 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:17.683 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.019702+0000", 2024-04-19T01:04:17.684 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3561999999999999e-05 2024-04-19T01:04:17.684 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.684 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.684 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:17.684 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.019717+0000", 2024-04-19T01:04:17.684 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.4793e-05 2024-04-19T01:04:17.684 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.684 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.684 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:17.684 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.021792+0000", 2024-04-19T01:04:17.684 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0020751739999999999 2024-04-19T01:04:17.684 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.684 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.685 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:17.685 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.021855+0000", 2024-04-19T01:04:17.685 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.3436999999999996e-05 2024-04-19T01:04:17.685 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.685 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.685 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:17.685 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.029768+0000", 2024-04-19T01:04:17.685 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0079128429999999993 2024-04-19T01:04:17.685 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.685 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.685 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:17.685 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.029868+0000", 2024-04-19T01:04:17.685 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.9962000000000003e-05 2024-04-19T01:04:17.685 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:17.686 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:17.686 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:17.686 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.686 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.686 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 43/31 ECSubWrite(tid=76, reqid=client.4604.0:267, at_version=43'46, trim_to=0'0, roll_forward_to=43'45))", 2024-04-19T01:04:17.686 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.020527+0000", 2024-04-19T01:04:17.686 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 3.6264196819999999, 2024-04-19T01:04:17.686 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.016212788999999998, 2024-04-19T01:04:17.686 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:17.686 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:17.686 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:17.686 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:17.686 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.686 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:17.687 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.020527+0000", 2024-04-19T01:04:17.687 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:17.687 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.687 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.687 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:17.687 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.020527+0000", 2024-04-19T01:04:17.687 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:17.687 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.687 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.687 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:17.687 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.020529+0000", 2024-04-19T01:04:17.687 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.046e-06 2024-04-19T01:04:17.687 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.687 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.687 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:17.688 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.020873+0000", 2024-04-19T01:04:17.688 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00034371800000000002 2024-04-19T01:04:17.688 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.688 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.688 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:17.688 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.020902+0000", 2024-04-19T01:04:17.688 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.8903000000000001e-05 2024-04-19T01:04:17.688 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.688 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.688 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:17.688 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.020924+0000", 2024-04-19T01:04:17.688 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2892e-05 2024-04-19T01:04:17.688 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.688 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.689 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:17.689 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.021071+0000", 2024-04-19T01:04:17.689 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00014612300000000001 2024-04-19T01:04:17.689 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.689 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.689 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:17.689 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.021113+0000", 2024-04-19T01:04:17.689 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.2573e-05 2024-04-19T01:04:17.689 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.689 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.689 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:17.689 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.036641+0000", 2024-04-19T01:04:17.689 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.015527501000000001 2024-04-19T01:04:17.689 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.689 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.690 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:17.690 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.036740+0000", 2024-04-19T01:04:17.690 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.9032999999999999e-05 2024-04-19T01:04:17.690 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:17.690 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:17.690 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:17.690 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.690 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.690 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 43/31 ECSubWrite(tid=77, reqid=client.4604.0:268, at_version=43'47, trim_to=0'0, roll_forward_to=43'45))", 2024-04-19T01:04:17.690 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.040150+0000", 2024-04-19T01:04:17.690 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 3.6067964780000001, 2024-04-19T01:04:17.690 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0062732179999999997, 2024-04-19T01:04:17.690 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:17.690 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:17.691 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:17.691 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:17.691 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.691 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:17.691 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.040150+0000", 2024-04-19T01:04:17.691 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:17.691 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.691 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.691 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:17.691 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.040150+0000", 2024-04-19T01:04:17.691 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:17.691 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.691 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.691 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:17.692 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.040152+0000", 2024-04-19T01:04:17.692 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.8530000000000001e-06 2024-04-19T01:04:17.692 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.692 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.692 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:17.692 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.040264+0000", 2024-04-19T01:04:17.692 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000112014 2024-04-19T01:04:17.692 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.692 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.692 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:17.692 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.040293+0000", 2024-04-19T01:04:17.692 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.9371999999999999e-05 2024-04-19T01:04:17.693 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.693 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.693 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:17.693 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.040307+0000", 2024-04-19T01:04:17.693 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.3702e-05 2024-04-19T01:04:17.693 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.693 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.693 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:17.693 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.040450+0000", 2024-04-19T01:04:17.693 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00014292100000000001 2024-04-19T01:04:17.693 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.693 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.693 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:17.694 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.040465+0000", 2024-04-19T01:04:17.694 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.4792e-05 2024-04-19T01:04:17.694 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.694 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.694 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:17.694 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.046330+0000", 2024-04-19T01:04:17.694 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.005865515 2024-04-19T01:04:17.694 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.694 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.694 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:17.694 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.046423+0000", 2024-04-19T01:04:17.694 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.3048999999999996e-05 2024-04-19T01:04:17.694 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:17.694 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:17.695 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:17.695 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.695 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.695 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.1s4 44/31 ECSubWrite(tid=15, reqid=client.4604.0:289, at_version=44'8, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:04:17.695 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.875457+0000", 2024-04-19T01:04:17.695 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 2.7714893649999999, 2024-04-19T01:04:17.695 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.036567641999999997, 2024-04-19T01:04:17.695 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:17.695 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:17.695 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:17.695 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:17.695 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.695 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:17.695 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875457+0000", 2024-04-19T01:04:17.695 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:17.697 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.697 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.697 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:17.697 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875457+0000", 2024-04-19T01:04:17.697 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:17.697 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.697 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.697 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:17.697 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875458+0000", 2024-04-19T01:04:17.697 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.8400000000000004e-07 2024-04-19T01:04:17.698 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.698 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.698 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:17.698 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875519+0000", 2024-04-19T01:04:17.698 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.1377000000000005e-05 2024-04-19T01:04:17.698 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.698 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.698 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:17.698 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875588+0000", 2024-04-19T01:04:17.698 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.9099999999999999e-05 2024-04-19T01:04:17.698 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.698 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.698 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:17.699 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875615+0000", 2024-04-19T01:04:17.699 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6902000000000001e-05 2024-04-19T01:04:17.699 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.699 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.699 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:17.699 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875846+0000", 2024-04-19T01:04:17.699 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00023076699999999999 2024-04-19T01:04:17.699 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.699 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.699 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:17.699 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875874+0000", 2024-04-19T01:04:17.699 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.7725e-05 2024-04-19T01:04:17.699 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.699 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.699 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:17.700 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.911826+0000", 2024-04-19T01:04:17.700 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.035952348000000002 2024-04-19T01:04:17.700 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.700 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.700 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:17.700 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.912025+0000", 2024-04-19T01:04:17.700 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000198739 2024-04-19T01:04:17.700 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:17.700 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:17.700 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:17.700 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.700 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.700 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 44/31 ECSubWrite(tid=82, reqid=client.4604.0:280, at_version=44'50, trim_to=0'0, roll_forward_to=43'48))", 2024-04-19T01:04:17.700 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.880504+0000", 2024-04-19T01:04:17.701 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 2.7664422709999998, 2024-04-19T01:04:17.701 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.041548157000000002, 2024-04-19T01:04:17.701 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:17.701 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:17.701 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:17.701 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:17.701 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.701 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:17.701 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.880504+0000", 2024-04-19T01:04:17.701 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:17.701 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.701 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.701 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:17.701 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.880504+0000", 2024-04-19T01:04:17.701 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:17.702 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.702 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.702 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:17.702 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.880506+0000", 2024-04-19T01:04:17.702 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9439999999999999e-06 2024-04-19T01:04:17.702 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.702 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.702 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:17.702 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.880844+0000", 2024-04-19T01:04:17.702 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00033782799999999998 2024-04-19T01:04:17.702 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.702 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.702 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:17.702 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.880877+0000", 2024-04-19T01:04:17.703 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.2524999999999998e-05 2024-04-19T01:04:17.703 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.703 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.703 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:17.703 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.880899+0000", 2024-04-19T01:04:17.703 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2656000000000001e-05 2024-04-19T01:04:17.703 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.703 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.703 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:17.703 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.881129+0000", 2024-04-19T01:04:17.703 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00022929600000000001 2024-04-19T01:04:17.703 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.703 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.703 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:17.704 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.881155+0000", 2024-04-19T01:04:17.704 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6729000000000001e-05 2024-04-19T01:04:17.704 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.704 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.704 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:17.704 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.921810+0000", 2024-04-19T01:04:17.704 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.040654807000000001 2024-04-19T01:04:17.704 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.704 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.704 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:17.704 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.922052+0000", 2024-04-19T01:04:17.704 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000242372 2024-04-19T01:04:17.704 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:17.704 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:17.704 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:17.705 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.705 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.705 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.es5 44/31 ECSubWrite(tid=83, reqid=osd.13.0:81, at_version=0'0, trim_to=0'0, roll_forward_to=44'14))", 2024-04-19T01:04:17.705 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.887116+0000", 2024-04-19T01:04:17.705 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 2.7598309090000002, 2024-04-19T01:04:17.705 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0081527319999999993, 2024-04-19T01:04:17.705 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:17.705 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:17.705 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:17.705 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:17.705 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.705 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:17.705 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.887116+0000", 2024-04-19T01:04:17.705 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:17.706 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.706 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.706 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:17.706 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.887116+0000", 2024-04-19T01:04:17.706 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:17.706 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.706 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.706 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:17.706 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.887118+0000", 2024-04-19T01:04:17.706 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1560000000000002e-06 2024-04-19T01:04:17.706 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.706 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.706 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:17.706 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.887144+0000", 2024-04-19T01:04:17.707 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6495999999999998e-05 2024-04-19T01:04:17.707 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.707 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.707 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:17.707 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.887178+0000", 2024-04-19T01:04:17.707 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.3241999999999998e-05 2024-04-19T01:04:17.707 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.707 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.707 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:17.707 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.887200+0000", 2024-04-19T01:04:17.707 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2665000000000001e-05 2024-04-19T01:04:17.707 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.707 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.707 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:17.708 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.887400+0000", 2024-04-19T01:04:17.708 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000199278 2024-04-19T01:04:17.708 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.708 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.708 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:17.708 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.887424+0000", 2024-04-19T01:04:17.708 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4545999999999998e-05 2024-04-19T01:04:17.708 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.708 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.708 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:17.708 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.895177+0000", 2024-04-19T01:04:17.708 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.007752643 2024-04-19T01:04:17.708 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.708 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.708 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:17.709 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.895268+0000", 2024-04-19T01:04:17.709 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.1706000000000006e-05 2024-04-19T01:04:17.709 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:17.709 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:17.709 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:17.709 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.709 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.709 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.ds2 44/31 ECSubWrite(tid=45, reqid=osd.12.0:45, at_version=44'20, trim_to=0'0, roll_forward_to=43'18))", 2024-04-19T01:04:17.709 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.888380+0000", 2024-04-19T01:04:17.709 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 2.7585661990000001, 2024-04-19T01:04:17.709 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0076145199999999996, 2024-04-19T01:04:17.709 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:17.709 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:17.709 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:17.709 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:17.710 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.710 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:17.710 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.888380+0000", 2024-04-19T01:04:17.710 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:17.710 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.710 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.710 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:17.710 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.888380+0000", 2024-04-19T01:04:17.710 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:17.710 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.710 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.710 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:17.710 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.888381+0000", 2024-04-19T01:04:17.710 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.23e-07 2024-04-19T01:04:17.711 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.711 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.711 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:17.711 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.888477+0000", 2024-04-19T01:04:17.711 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.6095999999999996e-05 2024-04-19T01:04:17.711 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.711 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.711 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:17.711 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.888511+0000", 2024-04-19T01:04:17.711 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.4184999999999999e-05 2024-04-19T01:04:17.711 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.711 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.711 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:17.711 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.888537+0000", 2024-04-19T01:04:17.712 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5202e-05 2024-04-19T01:04:17.712 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.712 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.712 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:17.712 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.888786+0000", 2024-04-19T01:04:17.712 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00024980899999999998 2024-04-19T01:04:17.712 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.712 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.712 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:17.712 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.888812+0000", 2024-04-19T01:04:17.712 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5629000000000001e-05 2024-04-19T01:04:17.712 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.712 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.712 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:17.713 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.895898+0000", 2024-04-19T01:04:17.713 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0070860430000000002 2024-04-19T01:04:17.713 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.713 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.713 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:17.713 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.895995+0000", 2024-04-19T01:04:17.713 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.6732999999999997e-05 2024-04-19T01:04:17.713 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:17.713 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:17.713 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:17.713 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.713 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.713 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.cs4 44/31 ECSubWrite(tid=52, reqid=osd.0.0:52, at_version=44'25, trim_to=0'0, roll_forward_to=43'23))", 2024-04-19T01:04:17.713 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.889007+0000", 2024-04-19T01:04:17.713 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 2.7579393510000001, 2024-04-19T01:04:17.714 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0074716929999999997, 2024-04-19T01:04:17.714 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:17.714 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:17.714 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:17.714 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:17.714 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.714 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:17.714 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.889007+0000", 2024-04-19T01:04:17.714 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:17.714 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.714 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.714 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:17.714 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.889007+0000", 2024-04-19T01:04:17.714 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:17.715 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.715 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.715 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:17.715 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.889008+0000", 2024-04-19T01:04:17.715 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.6000000000000003e-07 2024-04-19T01:04:17.715 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.715 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.715 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:17.715 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.889062+0000", 2024-04-19T01:04:17.715 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.4112e-05 2024-04-19T01:04:17.715 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.715 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.715 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:17.715 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.889103+0000", 2024-04-19T01:04:17.716 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.1134000000000003e-05 2024-04-19T01:04:17.716 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.716 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.716 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:17.716 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.889125+0000", 2024-04-19T01:04:17.716 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2232999999999999e-05 2024-04-19T01:04:17.716 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.716 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.716 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:17.716 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.889497+0000", 2024-04-19T01:04:17.716 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00037114599999999999 2024-04-19T01:04:17.716 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.716 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.716 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:17.716 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.889522+0000", 2024-04-19T01:04:17.717 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5352000000000001e-05 2024-04-19T01:04:17.717 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.717 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.717 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:17.717 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.896386+0000", 2024-04-19T01:04:17.717 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0068645499999999996 2024-04-19T01:04:17.717 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.717 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.717 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:17.717 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.896479+0000", 2024-04-19T01:04:17.717 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.2505999999999998e-05 2024-04-19T01:04:17.717 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:17.717 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:17.717 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:17.718 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.718 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.718 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.ds2 44/31 ECSubWrite(tid=46, reqid=osd.12.0:46, at_version=44'22, trim_to=0'0, roll_forward_to=43'18))", 2024-04-19T01:04:17.718 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.889639+0000", 2024-04-19T01:04:17.718 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 2.7573071220000003, 2024-04-19T01:04:17.718 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0069399529999999996, 2024-04-19T01:04:17.718 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:17.718 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:17.718 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:17.718 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:17.718 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.718 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:17.718 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.889639+0000", 2024-04-19T01:04:17.718 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:17.719 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.719 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.719 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:17.719 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.889639+0000", 2024-04-19T01:04:17.719 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:17.719 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.719 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.719 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:17.719 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.889641+0000", 2024-04-19T01:04:17.719 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7069999999999999e-06 2024-04-19T01:04:17.719 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.719 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.719 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:17.719 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.889694+0000", 2024-04-19T01:04:17.720 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.3260000000000002e-05 2024-04-19T01:04:17.720 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.720 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.720 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:17.720 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.889721+0000", 2024-04-19T01:04:17.720 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6991999999999998e-05 2024-04-19T01:04:17.720 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.720 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.720 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:17.720 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.889744+0000", 2024-04-19T01:04:17.720 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2606e-05 2024-04-19T01:04:17.720 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.720 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.720 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:17.720 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.889938+0000", 2024-04-19T01:04:17.721 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000194305 2024-04-19T01:04:17.721 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.721 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.721 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:17.721 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.889964+0000", 2024-04-19T01:04:17.721 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5819000000000001e-05 2024-04-19T01:04:17.721 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.721 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.721 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:17.721 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.896499+0000", 2024-04-19T01:04:17.721 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0065345910000000002 2024-04-19T01:04:17.721 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.721 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.721 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:17.722 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.896579+0000", 2024-04-19T01:04:17.722 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.0673000000000005e-05 2024-04-19T01:04:17.722 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:17.722 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:17.722 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:17.722 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.722 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.722 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 44/31 ECSubWrite(tid=84, reqid=client.4604.0:281, at_version=44'51, trim_to=0'0, roll_forward_to=43'48))", 2024-04-19T01:04:17.722 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.906686+0000", 2024-04-19T01:04:17.722 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 2.7402608740000001, 2024-04-19T01:04:17.722 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.018013254999999999, 2024-04-19T01:04:17.722 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:17.722 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:17.722 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:17.723 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:17.723 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.723 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:17.723 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.906686+0000", 2024-04-19T01:04:17.723 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:17.723 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.723 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.723 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:17.723 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.906686+0000", 2024-04-19T01:04:17.723 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:17.723 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.723 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.723 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:17.724 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.906688+0000", 2024-04-19T01:04:17.724 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1569999999999998e-06 2024-04-19T01:04:17.724 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.724 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.724 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:17.724 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.906789+0000", 2024-04-19T01:04:17.724 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000101488 2024-04-19T01:04:17.724 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.724 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.724 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:17.724 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.906831+0000", 2024-04-19T01:04:17.724 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.1488e-05 2024-04-19T01:04:17.724 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.724 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.724 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:17.725 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.906855+0000", 2024-04-19T01:04:17.725 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4269000000000001e-05 2024-04-19T01:04:17.725 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.725 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.725 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:17.725 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.913161+0000", 2024-04-19T01:04:17.725 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0063058070000000001 2024-04-19T01:04:17.725 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.725 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.725 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:17.725 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.913193+0000", 2024-04-19T01:04:17.725 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.2007e-05 2024-04-19T01:04:17.725 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.725 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.725 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:17.726 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.924624+0000", 2024-04-19T01:04:17.726 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.011430879999999999 2024-04-19T01:04:17.726 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.726 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.726 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:17.726 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.924699+0000", 2024-04-19T01:04:17.726 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.5159000000000003e-05 2024-04-19T01:04:17.726 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:17.726 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:17.726 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:17.726 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.726 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.726 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.1s4 44/31 ECSubWrite(tid=16, reqid=client.4604.0:289, at_version=0'0, trim_to=0'0, roll_forward_to=44'8))", 2024-04-19T01:04:17.726 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.915358+0000", 2024-04-19T01:04:17.727 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 2.7315889200000001, 2024-04-19T01:04:17.727 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0069017230000000002, 2024-04-19T01:04:17.727 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:17.727 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:17.727 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:17.727 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:17.727 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.727 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:17.727 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.915358+0000", 2024-04-19T01:04:17.727 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:17.727 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.727 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.727 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:17.727 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.915358+0000", 2024-04-19T01:04:17.728 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:17.728 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.728 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.728 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:17.728 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.915360+0000", 2024-04-19T01:04:17.728 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2000000000000001e-06 2024-04-19T01:04:17.728 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.728 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.728 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:17.728 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.915389+0000", 2024-04-19T01:04:17.728 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.8926999999999999e-05 2024-04-19T01:04:17.728 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.728 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.728 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:17.728 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.915432+0000", 2024-04-19T01:04:17.729 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.3475e-05 2024-04-19T01:04:17.729 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.729 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.729 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:17.729 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.915457+0000", 2024-04-19T01:04:17.729 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5075999999999998e-05 2024-04-19T01:04:17.729 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.729 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.729 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:17.729 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.915684+0000", 2024-04-19T01:04:17.729 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00022654399999999999 2024-04-19T01:04:17.729 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.729 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.729 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:17.729 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.915709+0000", 2024-04-19T01:04:17.730 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5068000000000001e-05 2024-04-19T01:04:17.730 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.730 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.730 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:17.730 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.922070+0000", 2024-04-19T01:04:17.730 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0063615130000000001 2024-04-19T01:04:17.730 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.730 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.730 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:17.730 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.922259+0000", 2024-04-19T01:04:17.730 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00018892 2024-04-19T01:04:17.730 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:17.730 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:17.730 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:17.731 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.731 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.731 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 44/31 ECSubWrite(tid=85, reqid=client.4604.0:282, at_version=44'52, trim_to=0'0, roll_forward_to=43'48))", 2024-04-19T01:04:17.731 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.916938+0000", 2024-04-19T01:04:17.731 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 2.730008599, 2024-04-19T01:04:17.731 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0078204020000000006, 2024-04-19T01:04:17.731 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:17.731 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:17.731 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:17.731 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:17.731 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.731 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:17.731 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.916938+0000", 2024-04-19T01:04:17.731 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:17.731 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.732 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.732 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:17.732 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.916938+0000", 2024-04-19T01:04:17.732 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:17.732 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.732 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.732 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:17.732 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.916940+0000", 2024-04-19T01:04:17.732 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0159999999999998e-06 2024-04-19T01:04:17.732 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.732 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.732 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:17.732 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.916992+0000", 2024-04-19T01:04:17.732 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.2494999999999998e-05 2024-04-19T01:04:17.733 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.733 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.733 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:17.733 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.917033+0000", 2024-04-19T01:04:17.733 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.0908000000000002e-05 2024-04-19T01:04:17.733 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.733 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.733 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:17.733 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.917057+0000", 2024-04-19T01:04:17.733 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4042000000000001e-05 2024-04-19T01:04:17.733 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.733 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.733 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:17.733 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.922174+0000", 2024-04-19T01:04:17.733 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.005116308 2024-04-19T01:04:17.734 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.734 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.734 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:17.734 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.922203+0000", 2024-04-19T01:04:17.734 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.9615e-05 2024-04-19T01:04:17.734 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.734 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.734 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:17.734 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.924709+0000", 2024-04-19T01:04:17.734 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0025060299999999998 2024-04-19T01:04:17.734 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.734 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.734 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:17.734 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.924758+0000", 2024-04-19T01:04:17.734 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.8987999999999999e-05 2024-04-19T01:04:17.735 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:17.735 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:17.735 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:17.735 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.735 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.735 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.0s1 44/31 ECSubWrite(tid=25, reqid=client.4604.0:284, at_version=44'22, trim_to=0'0, roll_forward_to=44'20))", 2024-04-19T01:04:17.735 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.922536+0000", 2024-04-19T01:04:17.735 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 2.7244105960000002, 2024-04-19T01:04:17.735 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.029007871000000001, 2024-04-19T01:04:17.735 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:17.735 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:17.735 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:17.735 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:17.735 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.735 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:17.736 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.922536+0000", 2024-04-19T01:04:17.736 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:17.736 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.736 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.736 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:17.736 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.922536+0000", 2024-04-19T01:04:17.736 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:17.736 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.736 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.736 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:17.736 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.922538+0000", 2024-04-19T01:04:17.736 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0829999999999998e-06 2024-04-19T01:04:17.736 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.736 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.736 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:17.737 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.923282+0000", 2024-04-19T01:04:17.737 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000743554 2024-04-19T01:04:17.737 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.737 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.737 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:17.737 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.923311+0000", 2024-04-19T01:04:17.737 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.9147999999999999e-05 2024-04-19T01:04:17.737 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.737 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.737 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:17.737 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.923337+0000", 2024-04-19T01:04:17.737 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6648999999999998e-05 2024-04-19T01:04:17.737 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.737 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.737 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:17.738 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.923466+0000", 2024-04-19T01:04:17.738 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00012821100000000001 2024-04-19T01:04:17.738 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.738 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.738 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:17.738 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.923478+0000", 2024-04-19T01:04:17.738 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.2291e-05 2024-04-19T01:04:17.738 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.738 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.738 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:17.738 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.951479+0000", 2024-04-19T01:04:17.738 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.02800066 2024-04-19T01:04:17.738 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:17.738 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:17.738 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:17.739 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.951544+0000", 2024-04-19T01:04:17.739 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.5275000000000001e-05 2024-04-19T01:04:17.739 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:17.739 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:17.739 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:17.739 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:17.739 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:17.739 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:17.740 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2024-04-19T01:04:17.741 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-19T01:04:17.742 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2024-04-19T01:04:17.742 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-04-19T01:04:17.743 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-19T01:04:17.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:04:17.675+0000 7faf86cf5640 0 -- 172.21.15.33:0/4215500959 >> [v2:172.21.15.155:6807/741132602,v1:172.21.15.155:6810/741132602] conn(0x7faf8022af00 msgr2=0x7faf8022d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-04-19T01:04:17.744 INFO:tasks.rados.rados.0.smithi155.stdout:121: finishing write tid 1 to smithi15528762-6 2024-04-19T01:04:17.744 INFO:tasks.rados.rados.0.smithi155.stdout:121: finishing write tid 2 to smithi15528762-6 2024-04-19T01:04:17.744 INFO:tasks.rados.rados.0.smithi155.stdout:121: finishing write tid 3 to smithi15528762-6 2024-04-19T01:04:17.744 INFO:tasks.rados.rados.0.smithi155.stdout:121: finishing write tid 4 to smithi15528762-6 2024-04-19T01:04:17.744 INFO:tasks.rados.rados.0.smithi155.stdout:121: finishing write tid 5 to smithi15528762-6 2024-04-19T01:04:17.744 INFO:tasks.rados.rados.0.smithi155.stdout:121: oid 6 updating version 0 to 17 2024-04-19T01:04:17.744 INFO:tasks.rados.rados.0.smithi155.stdout:121: oid 6 version 17 is already newer than 16 2024-04-19T01:04:17.744 INFO:tasks.rados.rados.0.smithi155.stdout:121: oid 6 updating version 17 to 18 2024-04-19T01:04:17.744 INFO:tasks.rados.rados.0.smithi155.stdout:121: oid 6 updating version 18 to 19 2024-04-19T01:04:17.745 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 6 v 19 (ObjNum 74 snap 8 seq_num 74) dirty exists 2024-04-19T01:04:17.745 INFO:tasks.rados.rados.0.smithi155.stdout:121: left oid 6 (ObjNum 74 snap 8 seq_num 74) 2024-04-19T01:04:17.745 INFO:tasks.rados.rados.0.smithi155.stdout:121: done (2 left) 2024-04-19T01:04:17.745 INFO:tasks.rados.rados.0.smithi155.stdout:122: done (1 left) 2024-04-19T01:04:17.745 INFO:tasks.rados.rados.0.smithi155.stdout:125: read oid 4 snap -1 2024-04-19T01:04:17.745 INFO:tasks.rados.rados.0.smithi155.stdout:125: expect deleted 2024-04-19T01:04:17.745 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 2 current snap is 8 2024-04-19T01:04:17.745 INFO:tasks.rados.rados.0.smithi155.stdout:126: seq_num 76 ranges {196608=507904,704512=507904,1212416=262144} 2024-04-19T01:04:17.745 INFO:tasks.rados.rados.0.smithi155.stdout:126: writing smithi15528762-2 from 196608 to 704512 tid 1 2024-04-19T01:04:17.745 INFO:tasks.rados.rados.0.smithi155.stdout:126: writing smithi15528762-2 from 704512 to 1212416 tid 2 2024-04-19T01:04:17.745 INFO:tasks.rados.rados.0.smithi155.stdout:126: writing smithi15528762-2 from 1212416 to 1474560 tid 3 2024-04-19T01:04:17.745 INFO:tasks.rados.rados.0.smithi155.stdout:127: delete oid 28 current snap is 8 2024-04-19T01:04:17.745 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 32 v 67 (ObjNum 129 snap 0 seq_num 1053130296) dirty exists 2024-04-19T01:04:17.745 INFO:tasks.rados.rados.0.smithi155.stdout:124: done (3 left) 2024-04-19T01:04:17.746 INFO:tasks.rados.rados.0.smithi155.stdout:125: done (2 left) 2024-04-19T01:04:17.746 INFO:tasks.rados.rados.0.smithi155.stdout:127: done (1 left) 2024-04-19T01:04:17.746 INFO:tasks.rados.rados.0.smithi155.stdout:128: setattr oid 48 current snap is 8 2024-04-19T01:04:17.746 INFO:tasks.rados.rados.0.smithi155.stdout:129: setattr oid 8 current snap is 8 2024-04-19T01:04:17.746 INFO:tasks.rados.rados.0.smithi155.stdout:130: rollback oid 36 current snap is 8 2024-04-19T01:04:17.746 INFO:tasks.rados.rados.0.smithi155.stdout:rollback oid 36 to 6 2024-04-19T01:04:17.746 INFO:tasks.rados.rados.0.smithi155.stdout:131: snap_remove snap 7 2024-04-19T01:04:17.746 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:04:17.741+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:04:17.842 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:17.840+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:17.873 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:17.873 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [], 2024-04-19T01:04:17.873 INFO:teuthology.orchestra.run.smithi110.stdout: "num_ops": 0 2024-04-19T01:04:17.873 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:17.887 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-04-19T01:04:17.902 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2024-04-19T01:04:17.902 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-19T01:04:17.943 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:04:17.941+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:04:18.009 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:18.009 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [], 2024-04-19T01:04:18.009 INFO:teuthology.orchestra.run.smithi110.stdout: "complaint_time": 30, 2024-04-19T01:04:18.009 INFO:teuthology.orchestra.run.smithi110.stdout: "num_blocked_ops": 0 2024-04-19T01:04:18.009 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:18.023 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-04-19T01:04:18.043 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:04:18.045+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:04:18.143 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:18.143 INFO:teuthology.orchestra.run.smithi110.stdout: "size": 20, 2024-04-19T01:04:18.143 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 600, 2024-04-19T01:04:18.143 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [ 2024-04-19T01:04:18.143 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.143 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:181 3.9s0 3:93d266e5:::smithi15528762-34:head [append 0~524288] snapc 6=[6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-04-19T01:04:18.143 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:09.817970+0000", 2024-04-19T01:04:18.143 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 8.3241303040000005, 2024-04-19T01:04:18.143 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.030101993, 2024-04-19T01:04:18.143 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:18.144 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:18.144 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:18.144 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:18.144 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:18.144 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:18.144 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 181 2024-04-19T01:04:18.144 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.144 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:18.144 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.144 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:18.144 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.817970+0000", 2024-04-19T01:04:18.144 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:18.144 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.144 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.145 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:18.145 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.817970+0000", 2024-04-19T01:04:18.145 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:18.145 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.145 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.145 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:18.145 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.817974+0000", 2024-04-19T01:04:18.145 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.546e-06 2024-04-19T01:04:18.145 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.145 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.145 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:18.145 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.819093+0000", 2024-04-19T01:04:18.145 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001119054 2024-04-19T01:04:18.145 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.146 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.146 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:18.146 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.819129+0000", 2024-04-19T01:04:18.146 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.6613999999999999e-05 2024-04-19T01:04:18.146 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.146 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.146 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:18.146 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.819155+0000", 2024-04-19T01:04:18.146 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6089e-05 2024-04-19T01:04:18.146 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.146 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.146 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:18.146 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.819284+0000", 2024-04-19T01:04:18.146 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000128085 2024-04-19T01:04:18.147 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.147 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.147 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:18.147 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.819408+0000", 2024-04-19T01:04:18.147 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000124058 2024-04-19T01:04:18.147 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.147 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.147 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:18.147 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.820674+0000", 2024-04-19T01:04:18.147 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0012667819999999999 2024-04-19T01:04:18.147 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.147 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.147 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:18.147 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.836303+0000", 2024-04-19T01:04:18.147 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.015628607999999999 2024-04-19T01:04:18.148 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.148 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.148 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:18.148 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.847942+0000", 2024-04-19T01:04:18.148 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.011638895999999999 2024-04-19T01:04:18.148 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.148 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.148 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:18.148 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.848072+0000", 2024-04-19T01:04:18.148 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00013026100000000001 2024-04-19T01:04:18.148 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:18.148 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:18.148 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:18.148 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.149 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.149 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:187 3.9s0 3:98647342:::smithi15528762-43:head [append 0~540672] snapc 6=[6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-04-19T01:04:18.149 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:09.887536+0000", 2024-04-19T01:04:18.149 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 8.2545641609999993, 2024-04-19T01:04:18.149 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.028193009000000002, 2024-04-19T01:04:18.149 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:18.149 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:18.149 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:18.149 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:18.149 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:18.149 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:18.149 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 187 2024-04-19T01:04:18.149 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.149 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:18.150 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.150 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:18.151 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.887536+0000", 2024-04-19T01:04:18.151 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:18.151 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.151 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.151 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:18.151 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.887536+0000", 2024-04-19T01:04:18.151 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:18.151 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.151 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.151 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:18.151 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.887539+0000", 2024-04-19T01:04:18.151 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3039999999999999e-06 2024-04-19T01:04:18.152 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.152 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.152 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:18.152 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.887941+0000", 2024-04-19T01:04:18.152 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000402791 2024-04-19T01:04:18.152 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.152 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.152 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:18.152 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.887964+0000", 2024-04-19T01:04:18.152 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2164999999999999e-05 2024-04-19T01:04:18.152 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.153 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.153 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:18.153 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.887978+0000", 2024-04-19T01:04:18.153 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.4728e-05 2024-04-19T01:04:18.153 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.153 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.153 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:18.153 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.888095+0000", 2024-04-19T01:04:18.153 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00011656 2024-04-19T01:04:18.153 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.153 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.153 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:18.153 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.888225+0000", 2024-04-19T01:04:18.154 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00012985899999999999 2024-04-19T01:04:18.154 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.154 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.154 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:18.154 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.889442+0000", 2024-04-19T01:04:18.154 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001217711 2024-04-19T01:04:18.154 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.154 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.154 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:18.154 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.900551+0000", 2024-04-19T01:04:18.154 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.011108916 2024-04-19T01:04:18.154 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.154 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.154 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:18.155 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.915641+0000", 2024-04-19T01:04:18.155 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.015089562000000001 2024-04-19T01:04:18.155 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.155 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.155 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:18.155 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.915729+0000", 2024-04-19T01:04:18.155 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.8412999999999995e-05 2024-04-19T01:04:18.155 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:18.155 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:18.155 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:18.155 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.155 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.155 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:235 3.9s0 3:93d266e5:::smithi15528762-34:head [append 0~688128] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:18.155 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.857683+0000", 2024-04-19T01:04:18.156 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 4.2844169980000002, 2024-04-19T01:04:18.156 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.044416514999999997, 2024-04-19T01:04:18.156 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:18.156 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:18.156 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:18.156 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:18.156 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:18.156 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:18.156 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 235 2024-04-19T01:04:18.156 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.156 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:18.156 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.156 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:18.156 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.857683+0000", 2024-04-19T01:04:18.157 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:18.157 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.157 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.157 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:18.157 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.857683+0000", 2024-04-19T01:04:18.157 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:18.157 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.157 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.157 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:18.157 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.857687+0000", 2024-04-19T01:04:18.157 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.1130000000000001e-06 2024-04-19T01:04:18.157 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.157 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.157 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:18.158 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.859118+0000", 2024-04-19T01:04:18.158 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001431114 2024-04-19T01:04:18.158 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.158 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.158 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:18.158 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.859150+0000", 2024-04-19T01:04:18.158 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.2589000000000003e-05 2024-04-19T01:04:18.158 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.158 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.158 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:18.158 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.860377+0000", 2024-04-19T01:04:18.158 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001226878 2024-04-19T01:04:18.158 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.159 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.159 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:18.159 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.866898+0000", 2024-04-19T01:04:18.159 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0065209250000000003 2024-04-19T01:04:18.159 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.159 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.159 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:18.159 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.867163+0000", 2024-04-19T01:04:18.159 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000264901 2024-04-19T01:04:18.159 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.159 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.159 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:18.159 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.869905+0000", 2024-04-19T01:04:18.159 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0027423550000000001 2024-04-19T01:04:18.160 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.160 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.160 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:18.160 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.897422+0000", 2024-04-19T01:04:18.160 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.027516445 2024-04-19T01:04:18.160 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.160 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.160 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:18.160 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.901877+0000", 2024-04-19T01:04:18.160 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.004454965 2024-04-19T01:04:18.160 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.160 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.160 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:18.160 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.902100+0000", 2024-04-19T01:04:18.160 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00022322999999999999 2024-04-19T01:04:18.161 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:18.161 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:18.161 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:18.161 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.161 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.161 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:236 3.9s0 3:93d266e5:::smithi15528762-34:head [append 0~376832] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:18.161 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.860599+0000", 2024-04-19T01:04:18.161 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 4.2815009780000004, 2024-04-19T01:04:18.161 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.042094899999999998, 2024-04-19T01:04:18.161 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:18.161 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:18.161 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:18.161 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:18.161 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:18.162 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:18.162 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 236 2024-04-19T01:04:18.162 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.162 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:18.162 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.162 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:18.162 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.860599+0000", 2024-04-19T01:04:18.162 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:18.162 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.162 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.162 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:18.162 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.860599+0000", 2024-04-19T01:04:18.162 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:18.162 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.163 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.163 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:18.163 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.860602+0000", 2024-04-19T01:04:18.163 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4899999999999999e-06 2024-04-19T01:04:18.163 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.163 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.163 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:18.163 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861320+0000", 2024-04-19T01:04:18.163 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00071774399999999998 2024-04-19T01:04:18.163 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.163 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.163 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:18.163 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861368+0000", 2024-04-19T01:04:18.163 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.8016999999999998e-05 2024-04-19T01:04:18.164 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.164 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.164 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:18.164 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861389+0000", 2024-04-19T01:04:18.164 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1175999999999999e-05 2024-04-19T01:04:18.164 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.164 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.164 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:18.164 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.888682+0000", 2024-04-19T01:04:18.164 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.027292739 2024-04-19T01:04:18.164 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.164 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.164 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:18.164 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.888867+0000", 2024-04-19T01:04:18.165 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00018496499999999999 2024-04-19T01:04:18.165 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.165 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.165 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:18.165 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.890430+0000", 2024-04-19T01:04:18.165 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001563069 2024-04-19T01:04:18.165 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.165 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.165 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:18.165 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.898524+0000", 2024-04-19T01:04:18.165 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0080939800000000006 2024-04-19T01:04:18.165 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.165 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.166 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:18.166 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.902524+0000", 2024-04-19T01:04:18.166 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0039999930000000003 2024-04-19T01:04:18.166 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.166 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.166 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:18.166 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.902694+0000", 2024-04-19T01:04:18.166 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00017072699999999999 2024-04-19T01:04:18.166 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:18.166 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:18.166 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:18.166 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.167 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.167 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:237 3.9s0 3:93d266e5:::smithi15528762-34:head [setxattr _header (56) in=7b] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:18.167 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.861460+0000", 2024-04-19T01:04:18.167 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 4.2806404210000002, 2024-04-19T01:04:18.167 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.042252307000000003, 2024-04-19T01:04:18.167 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:18.167 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:18.167 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:18.167 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:18.167 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:18.167 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:18.167 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 237 2024-04-19T01:04:18.167 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.168 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:18.168 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.168 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:18.168 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861460+0000", 2024-04-19T01:04:18.168 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:18.168 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.168 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.168 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:18.168 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861460+0000", 2024-04-19T01:04:18.168 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:18.168 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.169 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.169 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:18.169 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861462+0000", 2024-04-19T01:04:18.169 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9929999999999998e-06 2024-04-19T01:04:18.169 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.169 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.169 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:18.170 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861472+0000", 2024-04-19T01:04:18.170 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.9070000000000002e-06 2024-04-19T01:04:18.170 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.170 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.170 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:18.170 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861493+0000", 2024-04-19T01:04:18.170 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1386000000000002e-05 2024-04-19T01:04:18.170 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.171 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.171 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:18.171 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861510+0000", 2024-04-19T01:04:18.171 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7198999999999999e-05 2024-04-19T01:04:18.171 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.171 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.171 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:18.171 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.899576+0000", 2024-04-19T01:04:18.171 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.038065486000000003 2024-04-19T01:04:18.171 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.171 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.171 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:18.171 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.899753+0000", 2024-04-19T01:04:18.171 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000176522 2024-04-19T01:04:18.172 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.172 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.172 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:18.172 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.900579+0000", 2024-04-19T01:04:18.172 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00082676899999999998 2024-04-19T01:04:18.172 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.172 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.172 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:18.172 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.902843+0000", 2024-04-19T01:04:18.172 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0022635329999999999 2024-04-19T01:04:18.172 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.172 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.172 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:18.173 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.903504+0000", 2024-04-19T01:04:18.173 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00066085099999999995 2024-04-19T01:04:18.173 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.173 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.173 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:18.173 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.903712+0000", 2024-04-19T01:04:18.173 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00020866099999999999 2024-04-19T01:04:18.173 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:18.173 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:18.173 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:18.173 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.174 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.174 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:238 3.9s0 3:93d266e5:::smithi15528762-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:18.174 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.861534+0000", 2024-04-19T01:04:18.174 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 4.2805662340000001, 2024-04-19T01:04:18.174 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.046352598000000002, 2024-04-19T01:04:18.174 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:18.174 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:18.174 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:04:18.174 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:18.174 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:18.174 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:18.174 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 238 2024-04-19T01:04:18.174 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.175 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:18.175 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.175 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:18.175 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861534+0000", 2024-04-19T01:04:18.175 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:18.175 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.175 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.175 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:18.175 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861534+0000", 2024-04-19T01:04:18.175 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:18.175 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.175 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.175 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:18.176 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861536+0000", 2024-04-19T01:04:18.176 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.607e-06 2024-04-19T01:04:18.176 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.176 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.176 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:18.176 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861542+0000", 2024-04-19T01:04:18.176 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.1689999999999997e-06 2024-04-19T01:04:18.176 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.176 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.176 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:18.176 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861562+0000", 2024-04-19T01:04:18.177 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0043000000000001e-05 2024-04-19T01:04:18.177 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.177 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.177 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:18.177 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861882+0000", 2024-04-19T01:04:18.177 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000320398 2024-04-19T01:04:18.177 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.177 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.177 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:18.177 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.901266+0000", 2024-04-19T01:04:18.177 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.039383373999999999 2024-04-19T01:04:18.177 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.177 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.177 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting for rw locks", 2024-04-19T01:04:18.178 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.901474+0000", 2024-04-19T01:04:18.178 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00020846999999999999 2024-04-19T01:04:18.178 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.178 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.178 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:18.178 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.904551+0000", 2024-04-19T01:04:18.178 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0030766690000000002 2024-04-19T01:04:18.178 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.178 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.178 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:18.178 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.904806+0000", 2024-04-19T01:04:18.178 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00025481899999999999 2024-04-19T01:04:18.178 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.179 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.179 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:18.179 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.907887+0000", 2024-04-19T01:04:18.179 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0030810490000000002 2024-04-19T01:04:18.179 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:18.179 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:18.179 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:18.179 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.179 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.179 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 43/31 ECSubWriteReply(tid=67, last_complete=43'39, committed=1, applied=1))", 2024-04-19T01:04:18.179 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.867911+0000", 2024-04-19T01:04:18.179 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 4.2741892650000004, 2024-04-19T01:04:18.179 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.029356884, 2024-04-19T01:04:18.180 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:18.180 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:18.180 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:18.180 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:18.180 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.180 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:18.180 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.867911+0000", 2024-04-19T01:04:18.180 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:18.180 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.180 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.180 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:18.180 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.867911+0000", 2024-04-19T01:04:18.180 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:18.181 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.181 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.181 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:18.181 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.867912+0000", 2024-04-19T01:04:18.181 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.9300000000000006e-07 2024-04-19T01:04:18.181 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.181 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.182 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:18.182 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.867919+0000", 2024-04-19T01:04:18.182 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.2799999999999998e-06 2024-04-19T01:04:18.182 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.182 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.182 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:18.182 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.867940+0000", 2024-04-19T01:04:18.183 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0576000000000001e-05 2024-04-19T01:04:18.183 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.183 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.183 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:18.183 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.867956+0000", 2024-04-19T01:04:18.183 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5971999999999999e-05 2024-04-19T01:04:18.183 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.183 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.183 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:18.183 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.897221+0000", 2024-04-19T01:04:18.183 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.029264935999999998 2024-04-19T01:04:18.183 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.183 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.183 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:18.184 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.897268+0000", 2024-04-19T01:04:18.184 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.7126999999999998e-05 2024-04-19T01:04:18.184 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:18.184 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:18.184 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:18.184 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.185 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.185 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 43/31 ECSubWriteReply(tid=67, last_complete=43'39, committed=1, applied=1))", 2024-04-19T01:04:18.185 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.868181+0000", 2024-04-19T01:04:18.185 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 4.2739196269999997, 2024-04-19T01:04:18.185 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.029159768999999999, 2024-04-19T01:04:18.185 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:18.186 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:18.186 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:18.186 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:18.186 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.186 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:18.186 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.868181+0000", 2024-04-19T01:04:18.186 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:18.186 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.186 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.186 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:18.186 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.868181+0000", 2024-04-19T01:04:18.186 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:18.186 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.186 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.187 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:18.187 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.868182+0000", 2024-04-19T01:04:18.187 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5889999999999999e-06 2024-04-19T01:04:18.187 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.187 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.187 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:18.187 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.868193+0000", 2024-04-19T01:04:18.187 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.0403e-05 2024-04-19T01:04:18.187 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.187 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.187 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:18.187 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.868218+0000", 2024-04-19T01:04:18.187 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5012999999999999e-05 2024-04-19T01:04:18.187 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.188 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.188 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:18.188 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.868236+0000", 2024-04-19T01:04:18.188 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7708999999999999e-05 2024-04-19T01:04:18.188 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.188 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.188 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:18.188 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.897308+0000", 2024-04-19T01:04:18.189 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.029072594 2024-04-19T01:04:18.189 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.189 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.189 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:18.189 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.897341+0000", 2024-04-19T01:04:18.189 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.2461e-05 2024-04-19T01:04:18.189 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:18.189 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:18.189 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:18.190 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.190 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.190 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 43/31 ECSubWriteReply(tid=69, last_complete=43'41, committed=1, applied=1))", 2024-04-19T01:04:18.190 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.868883+0000", 2024-04-19T01:04:18.190 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 4.2732176060000002, 2024-04-19T01:04:18.190 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.028541337999999999, 2024-04-19T01:04:18.190 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:18.190 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:18.190 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:18.190 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:18.190 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.190 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:18.190 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.868883+0000", 2024-04-19T01:04:18.191 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:18.191 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.191 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.191 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:18.191 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.868883+0000", 2024-04-19T01:04:18.191 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:18.191 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.191 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.191 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:18.191 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.868885+0000", 2024-04-19T01:04:18.191 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7129999999999999e-06 2024-04-19T01:04:18.191 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.191 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.192 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:18.192 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.868895+0000", 2024-04-19T01:04:18.192 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.0426e-05 2024-04-19T01:04:18.192 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.192 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.192 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:18.192 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.868924+0000", 2024-04-19T01:04:18.192 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.9026e-05 2024-04-19T01:04:18.192 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.192 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.192 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:18.192 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.868947+0000", 2024-04-19T01:04:18.192 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2974999999999999e-05 2024-04-19T01:04:18.192 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.193 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.193 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:18.193 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.897394+0000", 2024-04-19T01:04:18.193 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.028446756 2024-04-19T01:04:18.193 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.193 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.193 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:18.193 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.897424+0000", 2024-04-19T01:04:18.193 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.0442000000000001e-05 2024-04-19T01:04:18.193 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:18.193 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:18.193 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:18.193 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.193 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.194 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 43/31 ECSubWriteReply(tid=67, last_complete=43'39, committed=1, applied=1))", 2024-04-19T01:04:18.194 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.869244+0000", 2024-04-19T01:04:18.194 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 4.2728565559999998, 2024-04-19T01:04:18.194 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.028256531000000001, 2024-04-19T01:04:18.194 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:18.194 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:18.194 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:18.194 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:18.194 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.194 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:18.194 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.869244+0000", 2024-04-19T01:04:18.195 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:18.195 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.195 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.195 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:18.195 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.869244+0000", 2024-04-19T01:04:18.195 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:18.195 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.195 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.195 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:18.195 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.869245+0000", 2024-04-19T01:04:18.195 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.3030000000000001e-06 2024-04-19T01:04:18.196 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.196 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.196 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:18.196 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.869255+0000", 2024-04-19T01:04:18.196 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.5929999999999995e-06 2024-04-19T01:04:18.196 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.196 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.196 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:18.196 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.869281+0000", 2024-04-19T01:04:18.196 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5865000000000001e-05 2024-04-19T01:04:18.196 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.196 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.196 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:18.196 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.869298+0000", 2024-04-19T01:04:18.196 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6906e-05 2024-04-19T01:04:18.197 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.197 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.197 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:18.197 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.897474+0000", 2024-04-19T01:04:18.197 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.028176632 2024-04-19T01:04:18.197 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.197 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.197 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:18.197 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.897500+0000", 2024-04-19T01:04:18.197 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6231999999999999e-05 2024-04-19T01:04:18.197 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:18.197 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:18.197 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:18.197 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.198 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.198 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 43/31 ECSubWriteReply(tid=67, last_complete=43'39, committed=1, applied=1))", 2024-04-19T01:04:18.198 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.869383+0000", 2024-04-19T01:04:18.198 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 4.2727171989999997, 2024-04-19T01:04:18.198 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.028212906999999999, 2024-04-19T01:04:18.198 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:18.198 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:18.198 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:18.198 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:18.198 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.198 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:18.198 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.869383+0000", 2024-04-19T01:04:18.199 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:18.199 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.199 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.199 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:18.199 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.869383+0000", 2024-04-19T01:04:18.199 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:18.199 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.199 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.199 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:18.199 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.869385+0000", 2024-04-19T01:04:18.199 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.3030000000000001e-06 2024-04-19T01:04:18.199 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.199 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.200 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:18.200 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.869393+0000", 2024-04-19T01:04:18.200 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.7159999999999992e-06 2024-04-19T01:04:18.200 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.200 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.200 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:18.200 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.869416+0000", 2024-04-19T01:04:18.200 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2566e-05 2024-04-19T01:04:18.200 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.200 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.200 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:18.200 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.869432+0000", 2024-04-19T01:04:18.200 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5991999999999999e-05 2024-04-19T01:04:18.200 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.200 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.201 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:18.201 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.897569+0000", 2024-04-19T01:04:18.201 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.028137348 2024-04-19T01:04:18.201 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.201 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.201 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:18.201 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.897596+0000", 2024-04-19T01:04:18.201 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6982e-05 2024-04-19T01:04:18.201 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:18.201 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:18.201 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:18.201 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.201 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.201 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs1 43/31 ECSubWrite(tid=47, reqid=client.4604.0:263, at_version=43'8, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:04:18.202 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.994106+0000", 2024-04-19T01:04:18.202 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 4.1479943820000003, 2024-04-19T01:04:18.202 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.031552210999999997, 2024-04-19T01:04:18.202 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:18.202 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:18.202 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:18.202 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:18.202 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.202 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:18.202 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.994106+0000", 2024-04-19T01:04:18.202 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:18.202 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.202 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.202 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:18.203 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.994106+0000", 2024-04-19T01:04:18.203 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:18.203 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.203 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.203 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:18.203 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.994107+0000", 2024-04-19T01:04:18.203 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.8000000000000004e-07 2024-04-19T01:04:18.203 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.203 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.203 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:18.203 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995101+0000", 2024-04-19T01:04:18.203 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00099370800000000005 2024-04-19T01:04:18.203 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.203 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.203 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:18.204 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995142+0000", 2024-04-19T01:04:18.204 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.0982000000000002e-05 2024-04-19T01:04:18.204 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.204 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.204 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:18.204 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995166+0000", 2024-04-19T01:04:18.204 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4485e-05 2024-04-19T01:04:18.204 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.204 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.204 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:18.204 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995315+0000", 2024-04-19T01:04:18.204 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00014939 2024-04-19T01:04:18.204 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.204 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.205 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:18.205 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995332+0000", 2024-04-19T01:04:18.205 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6469000000000001e-05 2024-04-19T01:04:18.205 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.205 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.205 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:18.205 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.025589+0000", 2024-04-19T01:04:18.205 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.03025707 2024-04-19T01:04:18.205 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.205 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.205 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:18.205 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.025658+0000", 2024-04-19T01:04:18.205 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.9227000000000007e-05 2024-04-19T01:04:18.205 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:18.205 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:18.206 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:18.206 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.206 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.206 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:267 3.9s0 3:93d266e5:::smithi15528762-34:head [append 0~737280] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:18.206 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.013235+0000", 2024-04-19T01:04:18.206 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 4.1288651129999998, 2024-04-19T01:04:18.206 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.043578804999999998, 2024-04-19T01:04:18.206 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:18.206 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:18.206 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:18.206 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:18.206 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:18.206 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:18.207 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 267 2024-04-19T01:04:18.207 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.207 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:18.207 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.207 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:18.207 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.013235+0000", 2024-04-19T01:04:18.207 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:18.207 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.207 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.207 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:18.207 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.013235+0000", 2024-04-19T01:04:18.207 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:18.207 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.207 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.208 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:18.208 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.013239+0000", 2024-04-19T01:04:18.208 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.6200000000000001e-06 2024-04-19T01:04:18.208 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.208 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.208 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:18.208 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.014549+0000", 2024-04-19T01:04:18.208 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001310075 2024-04-19T01:04:18.208 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.208 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.208 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:18.208 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.014585+0000", 2024-04-19T01:04:18.208 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.5494999999999998e-05 2024-04-19T01:04:18.208 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.209 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.209 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:18.209 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.014607+0000", 2024-04-19T01:04:18.209 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2371999999999999e-05 2024-04-19T01:04:18.209 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.209 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.209 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:18.209 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.014744+0000", 2024-04-19T01:04:18.209 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000136988 2024-04-19T01:04:18.209 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.209 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.209 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:18.209 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.015097+0000", 2024-04-19T01:04:18.209 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00035269400000000002 2024-04-19T01:04:18.209 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.210 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.210 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:18.210 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.020447+0000", 2024-04-19T01:04:18.210 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0053503739999999998 2024-04-19T01:04:18.210 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.210 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.210 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:18.210 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.047175+0000", 2024-04-19T01:04:18.210 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.026728372 2024-04-19T01:04:18.210 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.210 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.211 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:18.211 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.056595+0000", 2024-04-19T01:04:18.211 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0094197320000000001 2024-04-19T01:04:18.211 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.211 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.211 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:18.211 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.056814+0000", 2024-04-19T01:04:18.211 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00021908299999999999 2024-04-19T01:04:18.211 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:18.211 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:18.211 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:18.211 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.211 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.211 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:268 3.9s0 3:93d266e5:::smithi15528762-34:head [append 0~376832] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:18.212 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.015660+0000", 2024-04-19T01:04:18.212 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 4.1264400779999999, 2024-04-19T01:04:18.212 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.041602957000000003, 2024-04-19T01:04:18.212 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:18.212 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:18.212 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:18.212 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:18.212 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:18.212 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:18.212 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 268 2024-04-19T01:04:18.212 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.212 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:18.212 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.212 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:18.213 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.015660+0000", 2024-04-19T01:04:18.213 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:18.213 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.213 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.213 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:18.213 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.015660+0000", 2024-04-19T01:04:18.213 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:18.213 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.213 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.213 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:18.213 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.015663+0000", 2024-04-19T01:04:18.214 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.0299999999999998e-06 2024-04-19T01:04:18.214 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.214 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.214 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:18.214 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016328+0000", 2024-04-19T01:04:18.214 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00066432000000000004 2024-04-19T01:04:18.214 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.214 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.214 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:18.214 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016362+0000", 2024-04-19T01:04:18.215 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.4709000000000003e-05 2024-04-19T01:04:18.215 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.215 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.215 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:18.215 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016384+0000", 2024-04-19T01:04:18.215 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1172999999999999e-05 2024-04-19T01:04:18.215 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.215 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.215 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:18.215 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.038612+0000", 2024-04-19T01:04:18.215 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.022227925999999999 2024-04-19T01:04:18.215 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.215 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.215 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:18.215 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.038788+0000", 2024-04-19T01:04:18.216 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000176095 2024-04-19T01:04:18.216 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.216 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.216 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:18.216 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.040120+0000", 2024-04-19T01:04:18.216 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001331917 2024-04-19T01:04:18.216 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.216 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.216 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:18.216 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.047665+0000", 2024-04-19T01:04:18.216 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0075450789999999997 2024-04-19T01:04:18.216 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.216 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.216 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:18.217 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.057085+0000", 2024-04-19T01:04:18.217 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0094201930000000003 2024-04-19T01:04:18.217 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.217 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.217 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:18.217 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.057263+0000", 2024-04-19T01:04:18.217 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00017851499999999999 2024-04-19T01:04:18.217 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:18.217 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:18.217 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:18.217 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.217 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.217 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:269 3.9s0 3:93d266e5:::smithi15528762-34:head [setxattr _header (56) in=7b] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:18.217 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.016434+0000", 2024-04-19T01:04:18.218 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 4.1256661159999997, 2024-04-19T01:04:18.218 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.041381567000000001, 2024-04-19T01:04:18.218 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:18.218 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:18.218 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:18.218 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:18.218 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:18.218 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:18.218 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 269 2024-04-19T01:04:18.218 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.218 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:18.218 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.218 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:18.218 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016434+0000", 2024-04-19T01:04:18.219 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:18.219 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.219 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.219 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:18.219 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016434+0000", 2024-04-19T01:04:18.219 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:18.219 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.219 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.219 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:18.219 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016436+0000", 2024-04-19T01:04:18.219 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5939999999999999e-06 2024-04-19T01:04:18.219 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.219 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.219 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:18.220 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016445+0000", 2024-04-19T01:04:18.220 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.9889999999999998e-06 2024-04-19T01:04:18.220 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.220 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.220 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:18.220 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016465+0000", 2024-04-19T01:04:18.220 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0403000000000001e-05 2024-04-19T01:04:18.220 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.220 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.220 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:18.221 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016481+0000", 2024-04-19T01:04:18.221 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5772000000000001e-05 2024-04-19T01:04:18.221 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.221 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.221 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:18.221 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.047733+0000", 2024-04-19T01:04:18.221 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.031251689999999999 2024-04-19T01:04:18.221 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.221 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.221 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:18.221 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.047885+0000", 2024-04-19T01:04:18.221 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015215300000000001 2024-04-19T01:04:18.221 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.221 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.222 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:18.222 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.048690+0000", 2024-04-19T01:04:18.222 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00080468100000000002 2024-04-19T01:04:18.222 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.222 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.222 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:18.222 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.049873+0000", 2024-04-19T01:04:18.222 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0011833200000000001 2024-04-19T01:04:18.222 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.222 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.222 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:18.222 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.057577+0000", 2024-04-19T01:04:18.222 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0077039860000000003 2024-04-19T01:04:18.222 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.223 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.223 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:18.223 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.057816+0000", 2024-04-19T01:04:18.223 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000238979 2024-04-19T01:04:18.223 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:18.223 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:18.223 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:18.223 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.223 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.223 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:270 3.9s0 3:93d266e5:::smithi15528762-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:18.223 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.016503+0000", 2024-04-19T01:04:18.223 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 4.1255969490000002, 2024-04-19T01:04:18.223 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.044494194000000001, 2024-04-19T01:04:18.223 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:18.223 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:18.224 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:04:18.224 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:18.224 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:18.224 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:18.224 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 270 2024-04-19T01:04:18.224 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.224 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:18.224 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.224 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:18.224 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016503+0000", 2024-04-19T01:04:18.224 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:18.224 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.224 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.224 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:18.225 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016503+0000", 2024-04-19T01:04:18.225 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:18.225 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.225 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.225 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:18.225 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016505+0000", 2024-04-19T01:04:18.225 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.2729999999999999e-06 2024-04-19T01:04:18.225 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.225 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.225 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:18.225 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016511+0000", 2024-04-19T01:04:18.225 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.7570000000000001e-06 2024-04-19T01:04:18.225 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.225 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.225 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:18.226 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016530+0000", 2024-04-19T01:04:18.226 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9565999999999999e-05 2024-04-19T01:04:18.226 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.226 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.226 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:18.226 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016545+0000", 2024-04-19T01:04:18.226 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.4963000000000001e-05 2024-04-19T01:04:18.226 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.226 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.226 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:18.226 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.049233+0000", 2024-04-19T01:04:18.226 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.032687815000000002 2024-04-19T01:04:18.226 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.226 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.227 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting for rw locks", 2024-04-19T01:04:18.227 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.049365+0000", 2024-04-19T01:04:18.227 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00013258399999999999 2024-04-19T01:04:18.227 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.227 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.227 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:18.227 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.058814+0000", 2024-04-19T01:04:18.227 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0094481080000000002 2024-04-19T01:04:18.227 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.227 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.227 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:18.227 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.059010+0000", 2024-04-19T01:04:18.227 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00019622699999999999 2024-04-19T01:04:18.227 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.228 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.228 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:18.228 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.060998+0000", 2024-04-19T01:04:18.228 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0019879009999999998 2024-04-19T01:04:18.228 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:18.228 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:18.228 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:18.228 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.228 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.228 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:280 3.9s0 3:98647342:::smithi15528762-43:head [append 0~753664] snapc 7=[7,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-19T01:04:18.228 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.869239+0000", 2024-04-19T01:04:18.228 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 3.2728610800000002, 2024-04-19T01:04:18.228 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.062785076999999995, 2024-04-19T01:04:18.228 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:18.228 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:18.229 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:18.229 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:18.229 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:18.229 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:18.229 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 280 2024-04-19T01:04:18.229 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.229 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:18.229 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.229 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:18.229 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.869239+0000", 2024-04-19T01:04:18.229 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:18.229 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.229 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.229 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:18.230 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.869239+0000", 2024-04-19T01:04:18.230 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:18.230 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.230 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.230 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:18.230 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.869243+0000", 2024-04-19T01:04:18.230 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.444e-06 2024-04-19T01:04:18.230 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.230 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.230 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:18.230 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.870594+0000", 2024-04-19T01:04:18.230 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001351662 2024-04-19T01:04:18.230 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.230 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.231 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:18.231 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.870629+0000", 2024-04-19T01:04:18.231 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.4329000000000003e-05 2024-04-19T01:04:18.231 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.231 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.231 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:18.231 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.870647+0000", 2024-04-19T01:04:18.231 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7768999999999999e-05 2024-04-19T01:04:18.231 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.231 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.231 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:18.231 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.870806+0000", 2024-04-19T01:04:18.231 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015962300000000001 2024-04-19T01:04:18.231 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.232 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.232 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "op must wait for map", 2024-04-19T01:04:18.232 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.870819+0000", 2024-04-19T01:04:18.232 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.3266e-05 2024-04-19T01:04:18.232 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.232 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.232 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:18.232 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875741+0000", 2024-04-19T01:04:18.232 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0049211599999999999 2024-04-19T01:04:18.232 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.232 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.232 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:18.232 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.876018+0000", 2024-04-19T01:04:18.232 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000276944 2024-04-19T01:04:18.232 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.233 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.233 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:18.233 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.880141+0000", 2024-04-19T01:04:18.233 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0041231030000000004 2024-04-19T01:04:18.233 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.233 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.233 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:18.233 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.915254+0000", 2024-04-19T01:04:18.233 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.035113610000000003 2024-04-19T01:04:18.233 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.233 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.233 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:18.233 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.931834+0000", 2024-04-19T01:04:18.233 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.016580212 2024-04-19T01:04:18.233 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.234 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.234 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:18.234 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.932024+0000", 2024-04-19T01:04:18.234 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00018995500000000001 2024-04-19T01:04:18.234 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:18.234 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:18.234 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:18.234 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.234 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.234 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:281 3.9s0 3:98647342:::smithi15528762-43:head [append 0~376832] snapc 7=[7,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-19T01:04:18.234 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.872157+0000", 2024-04-19T01:04:18.234 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 3.2699434570000001, 2024-04-19T01:04:18.234 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.061221235999999998, 2024-04-19T01:04:18.235 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:18.235 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:18.235 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:18.235 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:18.235 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:18.235 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:18.235 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 281 2024-04-19T01:04:18.235 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.235 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:18.235 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.235 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:18.235 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872157+0000", 2024-04-19T01:04:18.235 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:18.236 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.236 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.236 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:18.236 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872157+0000", 2024-04-19T01:04:18.236 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:18.236 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.236 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.236 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:18.236 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872159+0000", 2024-04-19T01:04:18.236 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5129999999999999e-06 2024-04-19T01:04:18.236 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.236 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.236 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:18.236 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872435+0000", 2024-04-19T01:04:18.236 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00027559500000000001 2024-04-19T01:04:18.237 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.237 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.237 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:18.237 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872470+0000", 2024-04-19T01:04:18.237 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.5170999999999999e-05 2024-04-19T01:04:18.237 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.237 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.237 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:18.237 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872488+0000", 2024-04-19T01:04:18.237 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.8136000000000001e-05 2024-04-19T01:04:18.238 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.238 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.238 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:18.238 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872639+0000", 2024-04-19T01:04:18.238 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015092 2024-04-19T01:04:18.238 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.238 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.238 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:04:18.238 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872658+0000", 2024-04-19T01:04:18.238 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.8882999999999999e-05 2024-04-19T01:04:18.238 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.239 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.239 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:18.239 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.903660+0000", 2024-04-19T01:04:18.239 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.031001846999999999 2024-04-19T01:04:18.239 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.239 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.239 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:18.239 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.904268+0000", 2024-04-19T01:04:18.239 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00060793999999999996 2024-04-19T01:04:18.239 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.239 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.239 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:18.239 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.906536+0000", 2024-04-19T01:04:18.239 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0022680180000000001 2024-04-19T01:04:18.239 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.240 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.240 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:18.240 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.918698+0000", 2024-04-19T01:04:18.240 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.012161852000000001 2024-04-19T01:04:18.240 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.240 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.240 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:18.240 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.933228+0000", 2024-04-19T01:04:18.240 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.014529781 2024-04-19T01:04:18.240 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.240 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.240 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:18.240 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.933378+0000", 2024-04-19T01:04:18.240 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015058000000000001 2024-04-19T01:04:18.241 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:18.241 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:18.241 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:18.241 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.241 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.241 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:282 3.9s0 3:98647342:::smithi15528762-43:head [setxattr _header (56) in=7b] snapc 7=[7,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-19T01:04:18.241 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.872536+0000", 2024-04-19T01:04:18.241 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 3.2695639540000001, 2024-04-19T01:04:18.241 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.062122945999999998, 2024-04-19T01:04:18.241 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:18.241 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:18.241 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:18.241 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:18.242 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:18.242 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:18.242 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 282 2024-04-19T01:04:18.242 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.242 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:18.242 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.242 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:18.242 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872536+0000", 2024-04-19T01:04:18.242 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:18.242 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.242 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.242 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:18.242 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872536+0000", 2024-04-19T01:04:18.242 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:18.243 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.243 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.243 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:18.243 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872538+0000", 2024-04-19T01:04:18.243 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.423e-06 2024-04-19T01:04:18.243 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.243 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.243 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:18.243 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872548+0000", 2024-04-19T01:04:18.243 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.8430000000000005e-06 2024-04-19T01:04:18.243 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.243 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.243 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:18.243 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872569+0000", 2024-04-19T01:04:18.244 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1336e-05 2024-04-19T01:04:18.244 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.244 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.244 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:18.244 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872585+0000", 2024-04-19T01:04:18.244 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6288999999999999e-05 2024-04-19T01:04:18.244 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.244 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.244 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:18.244 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872783+0000", 2024-04-19T01:04:18.244 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000197488 2024-04-19T01:04:18.244 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.244 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.244 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:04:18.244 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872804+0000", 2024-04-19T01:04:18.245 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0839000000000002e-05 2024-04-19T01:04:18.245 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.245 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.245 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:18.245 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.915520+0000", 2024-04-19T01:04:18.245 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.042715887000000001 2024-04-19T01:04:18.245 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.245 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.245 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:18.245 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.915852+0000", 2024-04-19T01:04:18.245 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00033199899999999998 2024-04-19T01:04:18.245 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.245 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.245 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:18.246 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.916938+0000", 2024-04-19T01:04:18.246 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0010863940000000001 2024-04-19T01:04:18.246 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.246 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.246 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:18.246 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.918701+0000", 2024-04-19T01:04:18.246 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001763351 2024-04-19T01:04:18.246 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.246 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.246 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:18.246 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.934283+0000", 2024-04-19T01:04:18.246 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.015581526 2024-04-19T01:04:18.246 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.246 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.247 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:18.247 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.934659+0000", 2024-04-19T01:04:18.247 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00037657100000000002 2024-04-19T01:04:18.247 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:18.247 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:18.247 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:18.247 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.247 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.247 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:283 3.9s0 3:98647342:::smithi15528762-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-04-19T01:04:18.247 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.872609+0000", 2024-04-19T01:04:18.247 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 3.2694911040000001, 2024-04-19T01:04:18.247 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.067525222999999995, 2024-04-19T01:04:18.247 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:18.247 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:18.247 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:04:18.248 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:18.248 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:18.248 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:18.248 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 283 2024-04-19T01:04:18.248 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.248 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:18.248 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.248 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:18.248 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872609+0000", 2024-04-19T01:04:18.248 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:18.248 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.248 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.248 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:18.248 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872609+0000", 2024-04-19T01:04:18.249 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:18.249 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.249 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.249 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:18.249 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872611+0000", 2024-04-19T01:04:18.249 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.367e-06 2024-04-19T01:04:18.249 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.249 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.249 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:18.249 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872617+0000", 2024-04-19T01:04:18.249 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.2400000000000004e-06 2024-04-19T01:04:18.249 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.249 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.249 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:18.250 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872636+0000", 2024-04-19T01:04:18.250 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.8941999999999999e-05 2024-04-19T01:04:18.250 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.250 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.250 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:18.250 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872652+0000", 2024-04-19T01:04:18.250 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5651999999999999e-05 2024-04-19T01:04:18.250 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.250 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.250 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:18.250 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872858+0000", 2024-04-19T01:04:18.250 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00020631100000000001 2024-04-19T01:04:18.250 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.250 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.251 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:04:18.251 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872867+0000", 2024-04-19T01:04:18.251 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.5400000000000001e-06 2024-04-19T01:04:18.251 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.251 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.251 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:18.251 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.917636+0000", 2024-04-19T01:04:18.251 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.044768372000000001 2024-04-19T01:04:18.251 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.251 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.251 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting for rw locks", 2024-04-19T01:04:18.251 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.917899+0000", 2024-04-19T01:04:18.251 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00026336799999999999 2024-04-19T01:04:18.251 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.252 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.252 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:18.252 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.935711+0000", 2024-04-19T01:04:18.252 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.017811862000000001 2024-04-19T01:04:18.252 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.252 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.252 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:18.252 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.935937+0000", 2024-04-19T01:04:18.252 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00022568800000000001 2024-04-19T01:04:18.252 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:18.252 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:18.252 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:18.252 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.940135+0000", 2024-04-19T01:04:18.252 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0041978809999999997 2024-04-19T01:04:18.253 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:18.253 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:18.253 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:18.253 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:18.253 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:18.253 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:18.254 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-19T01:04:18.255 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:04:18.255+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:04:18.356 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:18.355+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:18.387 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:04:18.387 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-04-19T01:04:18.387 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-04-19T01:04:18.387 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:04:18.402 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-19T01:04:18.457 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:04:18.457+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:04:18.503 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.12 to 1 (10000) 2024-04-19T01:04:18.509 INFO:tasks.rados.rados.0.smithi155.stdout:131: done (4 left) 2024-04-19T01:04:18.509 INFO:tasks.rados.rados.0.smithi155.stdout:132: rollback oid 37 current snap is 8 2024-04-19T01:04:18.509 INFO:tasks.rados.rados.0.smithi155.stdout:rollback oid 37 to 5 2024-04-19T01:04:18.509 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 48 v 30 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-04-19T01:04:18.509 INFO:tasks.rados.rados.0.smithi155.stdout:130: finishing rollback tid 0 to smithi15528762-36 2024-04-19T01:04:18.509 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 8 v 12 (ObjNum 65 snap 0 seq_num 1025906128) dirty exists 2024-04-19T01:04:18.509 INFO:tasks.rados.rados.0.smithi155.stdout:130: finishing rollback tid 1 to smithi15528762-36 2024-04-19T01:04:18.510 INFO:tasks.rados.rados.0.smithi155.stdout:130: finishing rollback tid 2 to smithi15528762-36 2024-04-19T01:04:18.510 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 36 v 35 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-04-19T01:04:18.510 INFO:tasks.rados.rados.0.smithi155.stdout:126: finishing write tid 1 to smithi15528762-2 2024-04-19T01:04:18.510 INFO:tasks.rados.rados.0.smithi155.stdout:126: finishing write tid 2 to smithi15528762-2 2024-04-19T01:04:18.510 INFO:tasks.rados.rados.0.smithi155.stdout:126: finishing write tid 3 to smithi15528762-2 2024-04-19T01:04:18.510 INFO:tasks.rados.rados.0.smithi155.stdout:126: finishing write tid 4 to smithi15528762-2 2024-04-19T01:04:18.510 INFO:tasks.rados.rados.0.smithi155.stdout:126: finishing write tid 5 to smithi15528762-2 2024-04-19T01:04:18.510 INFO:tasks.rados.rados.0.smithi155.stdout:126: oid 2 updating version 0 to 34 2024-04-19T01:04:18.510 INFO:tasks.rados.rados.0.smithi155.stdout:126: oid 2 version 34 is already newer than 33 2024-04-19T01:04:18.510 INFO:tasks.rados.rados.0.smithi155.stdout:126: oid 2 updating version 34 to 35 2024-04-19T01:04:18.510 INFO:tasks.rados.rados.0.smithi155.stdout:126: oid 2 version 35 is already newer than 32 2024-04-19T01:04:18.510 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 2 v 35 (ObjNum 76 snap 8 seq_num 76) dirty exists 2024-04-19T01:04:18.510 INFO:tasks.rados.rados.0.smithi155.stdout:126: left oid 2 (ObjNum 76 snap 8 seq_num 76) 2024-04-19T01:04:18.511 INFO:tasks.rados.rados.0.smithi155.stdout:126: done (4 left) 2024-04-19T01:04:18.511 INFO:tasks.rados.rados.0.smithi155.stdout:128: done (3 left) 2024-04-19T01:04:18.511 INFO:tasks.rados.rados.0.smithi155.stdout:129: done (2 left) 2024-04-19T01:04:18.511 INFO:tasks.rados.rados.0.smithi155.stdout:130: done (1 left) 2024-04-19T01:04:18.511 INFO:tasks.rados.rados.0.smithi155.stdout:133: copy_from oid 23 from oid 16 current snap is 8 2024-04-19T01:04:18.511 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 6 current snap is 8 2024-04-19T01:04:18.511 INFO:tasks.rados.rados.0.smithi155.stdout:134: seq_num 78 ranges {1343488=557056,1900544=671744,2572288=98304} 2024-04-19T01:04:18.532 INFO:tasks.rados.rados.0.smithi155.stdout:134: writing smithi15528762-6 from 1343488 to 1900544 tid 1 2024-04-19T01:04:18.540 INFO:tasks.rados.rados.0.smithi155.stdout:134: writing smithi15528762-6 from 1900544 to 2572288 tid 2 2024-04-19T01:04:18.540 INFO:tasks.rados.rados.0.smithi155.stdout:134: writing smithi15528762-6 from 2572288 to 2670592 tid 3 2024-04-19T01:04:18.540 INFO:tasks.rados.rados.0.smithi155.stdout:132: finishing rollback tid 1 to smithi15528762-37 2024-04-19T01:04:18.540 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 37 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-04-19T01:04:18.540 INFO:tasks.rados.rados.0.smithi155.stdout:132: done (2 left) 2024-04-19T01:04:18.540 INFO:tasks.rados.rados.0.smithi155.stdout:135: rollback oid 43 current snap is 8 2024-04-19T01:04:18.540 INFO:tasks.rados.rados.0.smithi155.stdout:rollback oid 43 to 5 2024-04-19T01:04:18.540 INFO:tasks.rados.rados.0.smithi155.stdout:136: setattr oid 33 current snap is 8 2024-04-19T01:04:18.540 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 13 1 2024-04-19T01:04:18.541 INFO:tasks.rados.rados.0.smithi155.stdout:137: copy_from oid 47 from oid 2 current snap is 8 2024-04-19T01:04:18.541 INFO:tasks.rados.rados.0.smithi155.stdout:138: rollback oid 27 current snap is 8 2024-04-19T01:04:18.541 INFO:tasks.rados.rados.0.smithi155.stdout:rollback oid 27 to 5 2024-04-19T01:04:18.541 INFO:tasks.rados.rados.0.smithi155.stdout:139: snap_remove snap 6 2024-04-19T01:04:18.542 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:04:18.542 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-04-19T01:04:18.542 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-04-19T01:04:18.542 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-04-19T01:04:18.542 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:04:18.558 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:04:18.560+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:04:18.561 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-19T01:04:18.659 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:18.660+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:18.730 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:04:18.730 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-04-19T01:04:18.731 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-04-19T01:04:18.731 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-04-19T01:04:18.731 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.731 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.3s2 38/31 ECSubWrite(tid=13, reqid=client.4604.0:173, at_version=38'9, trim_to=0'0, roll_forward_to=34'7))", 2024-04-19T01:04:18.731 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:08.824914+0000", 2024-04-19T01:04:18.731 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 9.9023255920000004, 2024-04-19T01:04:18.731 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015651700000000001, 2024-04-19T01:04:18.731 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:18.731 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:18.732 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:18.732 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:18.732 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.732 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:18.732 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:08.824914+0000", 2024-04-19T01:04:18.732 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:18.733 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.733 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.733 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:18.733 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:08.824914+0000", 2024-04-19T01:04:18.733 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:18.733 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.733 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.733 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:18.733 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:08.824915+0000", 2024-04-19T01:04:18.733 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.046e-06 2024-04-19T01:04:18.733 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.733 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.734 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:18.734 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:08.825330+0000", 2024-04-19T01:04:18.734 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00041453399999999999 2024-04-19T01:04:18.734 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.734 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.734 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:18.734 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:08.825359+0000", 2024-04-19T01:04:18.734 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9159e-05 2024-04-19T01:04:18.734 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.735 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.735 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:18.735 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:08.825377+0000", 2024-04-19T01:04:18.735 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8145999999999999e-05 2024-04-19T01:04:18.735 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.735 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.735 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:18.735 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:08.825501+0000", 2024-04-19T01:04:18.735 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012435899999999999 2024-04-19T01:04:18.735 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.735 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.735 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:18.735 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:08.825519+0000", 2024-04-19T01:04:18.735 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7909000000000001e-05 2024-04-19T01:04:18.736 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.736 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.736 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:18.736 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:08.840461+0000", 2024-04-19T01:04:18.736 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014941538000000001 2024-04-19T01:04:18.736 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.736 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.736 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:18.736 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:08.840566+0000", 2024-04-19T01:04:18.736 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000105009 2024-04-19T01:04:18.736 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:18.736 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:18.736 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:18.736 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.736 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.737 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 43/31 ECSubWrite(tid=43, reqid=client.4604.0:251, at_version=43'29, trim_to=0'0, roll_forward_to=43'27))", 2024-04-19T01:04:18.737 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.986035+0000", 2024-04-19T01:04:18.737 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.7412047429999999, 2024-04-19T01:04:18.737 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024180535999999999, 2024-04-19T01:04:18.737 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:18.737 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:18.737 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:18.737 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:18.737 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.738 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:18.738 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986035+0000", 2024-04-19T01:04:18.738 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:18.738 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.738 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.738 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:18.738 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986035+0000", 2024-04-19T01:04:18.738 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:18.738 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.738 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.738 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:18.738 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986058+0000", 2024-04-19T01:04:18.739 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2903000000000001e-05 2024-04-19T01:04:18.739 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.739 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.739 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:18.739 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986841+0000", 2024-04-19T01:04:18.739 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00078285599999999996 2024-04-19T01:04:18.739 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.739 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.739 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:18.739 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986887+0000", 2024-04-19T01:04:18.739 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6202000000000003e-05 2024-04-19T01:04:18.739 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.740 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.740 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:18.740 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986913+0000", 2024-04-19T01:04:18.740 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6162000000000001e-05 2024-04-19T01:04:18.740 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.740 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.740 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:18.740 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987069+0000", 2024-04-19T01:04:18.740 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015625099999999999 2024-04-19T01:04:18.740 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.740 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.740 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:18.741 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987099+0000", 2024-04-19T01:04:18.741 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9578999999999999e-05 2024-04-19T01:04:18.741 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.741 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.741 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:18.741 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.010110+0000", 2024-04-19T01:04:18.741 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023011357999999999 2024-04-19T01:04:18.741 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.741 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.741 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:18.741 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.010215+0000", 2024-04-19T01:04:18.741 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000105225 2024-04-19T01:04:18.742 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:18.742 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:18.742 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:18.742 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.742 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.742 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds5 43/31 ECSubWrite(tid=38, reqid=client.4604.0:253, at_version=43'14, trim_to=0'0, roll_forward_to=43'12))", 2024-04-19T01:04:18.742 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.987245+0000", 2024-04-19T01:04:18.742 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.7399947210000004, 2024-04-19T01:04:18.742 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022838027, 2024-04-19T01:04:18.742 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:18.742 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:18.742 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:18.743 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:18.743 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.743 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:18.743 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987245+0000", 2024-04-19T01:04:18.743 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:18.743 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.743 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.743 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:18.743 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987245+0000", 2024-04-19T01:04:18.743 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:18.743 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.743 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.743 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:18.743 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987247+0000", 2024-04-19T01:04:18.744 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2800000000000002e-06 2024-04-19T01:04:18.744 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.744 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.744 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:18.744 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988142+0000", 2024-04-19T01:04:18.744 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00089449199999999999 2024-04-19T01:04:18.744 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.744 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.744 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:18.744 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988180+0000", 2024-04-19T01:04:18.744 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8615000000000002e-05 2024-04-19T01:04:18.744 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.744 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.745 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:18.745 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988206+0000", 2024-04-19T01:04:18.745 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6023000000000001e-05 2024-04-19T01:04:18.745 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.745 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.745 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:18.745 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988349+0000", 2024-04-19T01:04:18.745 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014252199999999999 2024-04-19T01:04:18.745 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.745 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.745 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:18.745 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988365+0000", 2024-04-19T01:04:18.745 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5855e-05 2024-04-19T01:04:18.745 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.746 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.746 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:18.746 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.010029+0000", 2024-04-19T01:04:18.746 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.021664454999999999 2024-04-19T01:04:18.746 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.746 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.746 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:18.746 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.010083+0000", 2024-04-19T01:04:18.746 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3785000000000001e-05 2024-04-19T01:04:18.746 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:18.746 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:18.746 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:18.746 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.747 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.747 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.fs6 43/31 ECSubWrite(tid=47, reqid=client.4604.0:263, at_version=43'8, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:04:18.747 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.994092+0000", 2024-04-19T01:04:18.747 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.7331477360000003, 2024-04-19T01:04:18.747 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.027264745999999999, 2024-04-19T01:04:18.747 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:18.747 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:18.747 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:18.747 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:18.747 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.747 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:18.747 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.994092+0000", 2024-04-19T01:04:18.747 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:18.748 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.748 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.748 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:18.748 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.994092+0000", 2024-04-19T01:04:18.748 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:18.748 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.748 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.748 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:18.748 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.994094+0000", 2024-04-19T01:04:18.748 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1399999999999998e-06 2024-04-19T01:04:18.748 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.748 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.748 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:18.748 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.994786+0000", 2024-04-19T01:04:18.748 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00069247999999999998 2024-04-19T01:04:18.749 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.749 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.749 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:18.749 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.994827+0000", 2024-04-19T01:04:18.749 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0105000000000003e-05 2024-04-19T01:04:18.749 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.749 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.749 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:18.749 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.994853+0000", 2024-04-19T01:04:18.749 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6259e-05 2024-04-19T01:04:18.749 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.749 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.749 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:18.749 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.995043+0000", 2024-04-19T01:04:18.750 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0001906 2024-04-19T01:04:18.750 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.750 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.750 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:18.750 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.995069+0000", 2024-04-19T01:04:18.750 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5633e-05 2024-04-19T01:04:18.750 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.750 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.750 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:18.750 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.021288+0000", 2024-04-19T01:04:18.750 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.026218865000000001 2024-04-19T01:04:18.750 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.750 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.751 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:18.751 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.021357+0000", 2024-04-19T01:04:18.751 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8664000000000005e-05 2024-04-19T01:04:18.751 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:18.751 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:18.751 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:18.751 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.751 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.751 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds5 43/31 ECSubWrite(tid=39, reqid=client.4604.0:254, at_version=43'15, trim_to=0'0, roll_forward_to=43'12))", 2024-04-19T01:04:18.751 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.997374+0000", 2024-04-19T01:04:18.751 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.7298658859999998, 2024-04-19T01:04:18.751 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013084780000000001, 2024-04-19T01:04:18.751 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:18.751 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:18.752 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:18.752 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:18.752 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.752 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:18.752 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.997374+0000", 2024-04-19T01:04:18.752 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:18.752 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.752 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.752 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:18.752 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.997374+0000", 2024-04-19T01:04:18.752 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:18.752 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.752 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.752 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:18.753 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.997376+0000", 2024-04-19T01:04:18.753 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9e-06 2024-04-19T01:04:18.753 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.753 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.753 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:18.753 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.997615+0000", 2024-04-19T01:04:18.753 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00023961100000000001 2024-04-19T01:04:18.753 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.753 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.753 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:18.753 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.997664+0000", 2024-04-19T01:04:18.753 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8476000000000001e-05 2024-04-19T01:04:18.753 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.754 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.754 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:18.754 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.997689+0000", 2024-04-19T01:04:18.754 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5109e-05 2024-04-19T01:04:18.754 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.754 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.754 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:18.754 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.001350+0000", 2024-04-19T01:04:18.754 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0036615749999999998 2024-04-19T01:04:18.754 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.754 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.754 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:18.754 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.001375+0000", 2024-04-19T01:04:18.755 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5032999999999999e-05 2024-04-19T01:04:18.755 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.755 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.755 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:18.755 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.010366+0000", 2024-04-19T01:04:18.755 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0089907159999999993 2024-04-19T01:04:18.755 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.755 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.755 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:18.755 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.010458+0000", 2024-04-19T01:04:18.755 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2360000000000003e-05 2024-04-19T01:04:18.755 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:18.755 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:18.756 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:18.756 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.756 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.756 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.fs6 43/31 ECSubWrite(tid=48, reqid=client.4604.0:264, at_version=43'9, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:04:18.756 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.006377+0000", 2024-04-19T01:04:18.756 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.7208625199999998, 2024-04-19T01:04:18.756 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015258295, 2024-04-19T01:04:18.756 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:18.756 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:18.756 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:18.756 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:18.756 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.756 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:18.756 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.006377+0000", 2024-04-19T01:04:18.757 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:18.757 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.757 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.757 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:18.757 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.006377+0000", 2024-04-19T01:04:18.757 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:18.757 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.757 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.757 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:18.757 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.006379+0000", 2024-04-19T01:04:18.757 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.114e-06 2024-04-19T01:04:18.757 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.757 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.758 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:18.758 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.006643+0000", 2024-04-19T01:04:18.758 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000263854 2024-04-19T01:04:18.758 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.758 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.758 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:18.758 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.006677+0000", 2024-04-19T01:04:18.758 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4388999999999999e-05 2024-04-19T01:04:18.758 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.758 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.758 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:18.758 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.006700+0000", 2024-04-19T01:04:18.758 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2849000000000001e-05 2024-04-19T01:04:18.758 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.758 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.759 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:18.759 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.011877+0000", 2024-04-19T01:04:18.759 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0051769679999999997 2024-04-19T01:04:18.759 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.759 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.759 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:18.759 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.011902+0000", 2024-04-19T01:04:18.759 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4926000000000002e-05 2024-04-19T01:04:18.759 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.759 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.759 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:18.759 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.021480+0000", 2024-04-19T01:04:18.759 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0095781359999999992 2024-04-19T01:04:18.759 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.760 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.760 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:18.760 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.021635+0000", 2024-04-19T01:04:18.760 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015505900000000001 2024-04-19T01:04:18.760 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:18.760 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:18.760 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:18.760 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.760 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.760 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.fs6 43/31 ECSubWrite(tid=49, reqid=client.4604.0:265, at_version=43'10, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:04:18.760 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.006754+0000", 2024-04-19T01:04:18.760 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.7204859250000002, 2024-04-19T01:04:18.760 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015562184, 2024-04-19T01:04:18.760 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:18.760 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:18.761 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:18.761 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:18.761 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.761 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:18.761 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.006754+0000", 2024-04-19T01:04:18.761 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:18.761 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.761 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.761 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:18.761 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.006754+0000", 2024-04-19T01:04:18.761 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:18.761 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.761 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.761 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:18.762 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.006755+0000", 2024-04-19T01:04:18.762 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7000000000000003e-07 2024-04-19T01:04:18.762 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.762 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.762 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:18.762 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.006813+0000", 2024-04-19T01:04:18.762 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8227999999999999e-05 2024-04-19T01:04:18.762 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.762 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.762 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:18.762 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.006836+0000", 2024-04-19T01:04:18.762 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2949e-05 2024-04-19T01:04:18.762 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.763 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.763 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:18.763 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.006853+0000", 2024-04-19T01:04:18.763 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7133e-05 2024-04-19T01:04:18.763 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.763 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.763 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:18.763 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.020855+0000", 2024-04-19T01:04:18.763 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014002199999999999 2024-04-19T01:04:18.763 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.763 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.763 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:18.763 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.020875+0000", 2024-04-19T01:04:18.763 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0273999999999999e-05 2024-04-19T01:04:18.763 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.764 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.764 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:18.764 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.022241+0000", 2024-04-19T01:04:18.764 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001365556 2024-04-19T01:04:18.764 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.764 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.764 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:18.764 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.022316+0000", 2024-04-19T01:04:18.764 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4874000000000002e-05 2024-04-19T01:04:18.764 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:18.764 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:18.764 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:18.764 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.764 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.765 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 44/31 ECSubWrite(tid=49, reqid=osd.8.0:49, at_version=44'33, trim_to=0'0, roll_forward_to=43'29))", 2024-04-19T01:04:18.765 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.876274+0000", 2024-04-19T01:04:18.765 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 3.850965242, 2024-04-19T01:04:18.765 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011032062, 2024-04-19T01:04:18.765 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:18.765 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:18.765 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:18.765 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:18.765 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.765 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:18.765 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876274+0000", 2024-04-19T01:04:18.765 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:18.765 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.765 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.766 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:18.766 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876274+0000", 2024-04-19T01:04:18.766 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:18.766 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.766 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.766 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:18.766 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876276+0000", 2024-04-19T01:04:18.766 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6330000000000001e-06 2024-04-19T01:04:18.766 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.766 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.766 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:18.766 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876327+0000", 2024-04-19T01:04:18.766 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0631999999999999e-05 2024-04-19T01:04:18.767 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.767 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.767 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:18.767 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876356+0000", 2024-04-19T01:04:18.767 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9442e-05 2024-04-19T01:04:18.767 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.767 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.767 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:18.767 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876377+0000", 2024-04-19T01:04:18.767 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1288999999999999e-05 2024-04-19T01:04:18.767 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.767 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.767 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:18.767 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.878543+0000", 2024-04-19T01:04:18.767 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0021657360000000001 2024-04-19T01:04:18.768 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.768 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.768 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:18.768 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.878570+0000", 2024-04-19T01:04:18.768 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7455999999999999e-05 2024-04-19T01:04:18.768 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.768 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.768 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:18.768 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.887209+0000", 2024-04-19T01:04:18.768 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0086386540000000008 2024-04-19T01:04:18.768 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.768 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.768 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:18.768 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.887306+0000", 2024-04-19T01:04:18.768 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7219999999999994e-05 2024-04-19T01:04:18.769 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:18.769 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:18.769 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:18.769 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.769 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.769 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.es4 47/31 ECSubWrite(tid=104, reqid=client.4604.0:312, at_version=47'18, trim_to=0'0, roll_forward_to=45'16))", 2024-04-19T01:04:18.769 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.582390+0000", 2024-04-19T01:04:18.769 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 1.144849955, 2024-04-19T01:04:18.769 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022795626999999999, 2024-04-19T01:04:18.769 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:18.769 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:18.769 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:18.769 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:18.769 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.769 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:18.770 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.582390+0000", 2024-04-19T01:04:18.770 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:18.770 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.770 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.770 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:18.770 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.582390+0000", 2024-04-19T01:04:18.770 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:18.770 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.770 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.770 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:18.770 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.582390+0000", 2024-04-19T01:04:18.770 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8299999999999996e-07 2024-04-19T01:04:18.770 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.770 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.771 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:18.771 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.582798+0000", 2024-04-19T01:04:18.771 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00040746499999999998 2024-04-19T01:04:18.771 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.771 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.771 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:18.771 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.582829+0000", 2024-04-19T01:04:18.771 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0988999999999998e-05 2024-04-19T01:04:18.771 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.771 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.771 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:18.771 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.582846+0000", 2024-04-19T01:04:18.771 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7252e-05 2024-04-19T01:04:18.771 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.771 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.772 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:18.772 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.582989+0000", 2024-04-19T01:04:18.772 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014342000000000001 2024-04-19T01:04:18.772 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.772 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.772 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:18.772 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.583006+0000", 2024-04-19T01:04:18.772 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6671999999999999e-05 2024-04-19T01:04:18.772 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.772 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.772 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:18.772 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.605104+0000", 2024-04-19T01:04:18.772 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022098222000000001 2024-04-19T01:04:18.772 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.772 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.773 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:18.773 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.605185+0000", 2024-04-19T01:04:18.773 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0823999999999997e-05 2024-04-19T01:04:18.773 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:18.773 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:18.773 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:18.773 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.773 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.773 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.es4 47/31 ECSubWrite(tid=105, reqid=client.4604.0:313, at_version=47'19, trim_to=0'0, roll_forward_to=45'16))", 2024-04-19T01:04:18.773 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.591244+0000", 2024-04-19T01:04:18.774 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 1.1359952099999999, 2024-04-19T01:04:18.774 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013988917, 2024-04-19T01:04:18.774 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:18.774 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:18.774 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:18.774 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:18.774 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.774 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:18.774 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.591244+0000", 2024-04-19T01:04:18.774 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:18.774 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.774 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.774 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:18.774 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.591244+0000", 2024-04-19T01:04:18.775 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:18.775 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.775 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.775 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:18.775 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.591246+0000", 2024-04-19T01:04:18.775 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.73e-06 2024-04-19T01:04:18.775 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.775 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.775 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:18.775 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.591388+0000", 2024-04-19T01:04:18.775 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000141602 2024-04-19T01:04:18.775 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.775 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.775 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:18.776 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.591413+0000", 2024-04-19T01:04:18.776 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5133000000000001e-05 2024-04-19T01:04:18.776 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.776 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.776 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:18.776 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.591428+0000", 2024-04-19T01:04:18.776 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5142e-05 2024-04-19T01:04:18.776 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.776 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.776 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:18.776 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.592121+0000", 2024-04-19T01:04:18.776 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00069288499999999996 2024-04-19T01:04:18.776 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.776 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.777 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:18.777 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.592149+0000", 2024-04-19T01:04:18.777 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8382e-05 2024-04-19T01:04:18.777 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.777 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.777 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:18.777 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.605197+0000", 2024-04-19T01:04:18.777 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013047621000000001 2024-04-19T01:04:18.777 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.777 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.777 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:18.777 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.605233+0000", 2024-04-19T01:04:18.777 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6421999999999998e-05 2024-04-19T01:04:18.777 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:18.778 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:18.778 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:18.778 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.778 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.778 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.fs6 47/31 ECSubWrite(tid=61, reqid=client.4604.0:317, at_version=47'16, trim_to=0'0, roll_forward_to=45'14))", 2024-04-19T01:04:18.778 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.605368+0000", 2024-04-19T01:04:18.778 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 1.12187185, 2024-04-19T01:04:18.778 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018611628000000002, 2024-04-19T01:04:18.778 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:18.778 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:18.778 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:18.778 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:18.778 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.779 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:18.779 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.605368+0000", 2024-04-19T01:04:18.779 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:18.779 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.779 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.779 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:18.779 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.605368+0000", 2024-04-19T01:04:18.779 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:18.779 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.779 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.779 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:18.779 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.605368+0000", 2024-04-19T01:04:18.779 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3700000000000001e-07 2024-04-19T01:04:18.779 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.780 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.780 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:18.780 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.605723+0000", 2024-04-19T01:04:18.780 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035445499999999999 2024-04-19T01:04:18.780 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.780 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.780 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:18.780 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.605744+0000", 2024-04-19T01:04:18.780 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1229e-05 2024-04-19T01:04:18.780 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.780 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.780 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:18.780 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.605759+0000", 2024-04-19T01:04:18.780 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5e-05 2024-04-19T01:04:18.780 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.781 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.781 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:18.781 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.605888+0000", 2024-04-19T01:04:18.781 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012950200000000001 2024-04-19T01:04:18.781 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.781 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.781 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:18.781 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.605906+0000", 2024-04-19T01:04:18.781 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7462e-05 2024-04-19T01:04:18.781 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.781 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.781 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:18.781 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.623858+0000", 2024-04-19T01:04:18.782 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017952630000000001 2024-04-19T01:04:18.782 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.782 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.782 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:18.782 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.623979+0000", 2024-04-19T01:04:18.782 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000120913 2024-04-19T01:04:18.782 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:18.782 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:18.782 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:18.782 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.782 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.782 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.fs6 47/31 ECSubWrite(tid=62, reqid=client.4604.0:318, at_version=47'17, trim_to=0'0, roll_forward_to=45'14))", 2024-04-19T01:04:18.782 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.612635+0000", 2024-04-19T01:04:18.783 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 1.11460406, 2024-04-19T01:04:18.783 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014639107, 2024-04-19T01:04:18.783 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:18.783 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:18.783 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:18.783 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:18.783 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.783 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:18.783 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.612635+0000", 2024-04-19T01:04:18.783 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:18.783 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.783 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.783 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:18.783 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.612635+0000", 2024-04-19T01:04:18.784 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:18.784 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.784 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.784 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:18.784 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.612637+0000", 2024-04-19T01:04:18.784 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6199999999999999e-06 2024-04-19T01:04:18.784 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.784 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.784 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:18.784 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.613041+0000", 2024-04-19T01:04:18.784 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00040351699999999999 2024-04-19T01:04:18.784 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.784 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.784 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:18.785 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.613066+0000", 2024-04-19T01:04:18.785 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5675999999999999e-05 2024-04-19T01:04:18.785 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.785 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.785 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:18.785 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.613081+0000", 2024-04-19T01:04:18.785 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4929999999999999e-05 2024-04-19T01:04:18.785 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.785 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.785 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:18.785 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.613751+0000", 2024-04-19T01:04:18.785 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00066930199999999998 2024-04-19T01:04:18.785 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.785 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.785 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:18.786 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.613764+0000", 2024-04-19T01:04:18.786 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3454e-05 2024-04-19T01:04:18.786 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.786 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.786 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:18.786 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.627156+0000", 2024-04-19T01:04:18.786 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013391568 2024-04-19T01:04:18.786 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.786 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.786 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:18.786 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.627275+0000", 2024-04-19T01:04:18.786 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011904 2024-04-19T01:04:18.787 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:18.787 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:18.787 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:18.787 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.787 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.787 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds5 47/31 ECSubWrite(tid=61, reqid=client.4604.0:330, at_version=47'32, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:04:18.787 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.677579+0000", 2024-04-19T01:04:18.787 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 1.049660896, 2024-04-19T01:04:18.787 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.033702862, 2024-04-19T01:04:18.787 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:18.787 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:18.787 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:18.787 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:18.787 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.787 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:18.788 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.677579+0000", 2024-04-19T01:04:18.788 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:18.788 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.788 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.788 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:18.788 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.677579+0000", 2024-04-19T01:04:18.788 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:18.788 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.788 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.788 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:18.788 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.677581+0000", 2024-04-19T01:04:18.788 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2800000000000002e-06 2024-04-19T01:04:18.788 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.788 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.789 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:18.789 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678062+0000", 2024-04-19T01:04:18.789 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00048067900000000002 2024-04-19T01:04:18.789 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.789 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.789 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:18.789 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678096+0000", 2024-04-19T01:04:18.790 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4381999999999997e-05 2024-04-19T01:04:18.790 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.790 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.790 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:18.790 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678117+0000", 2024-04-19T01:04:18.790 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0679999999999999e-05 2024-04-19T01:04:18.790 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.790 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.790 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:18.790 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678249+0000", 2024-04-19T01:04:18.790 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013211899999999999 2024-04-19T01:04:18.790 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.791 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.791 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:18.791 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678270+0000", 2024-04-19T01:04:18.791 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1409000000000001e-05 2024-04-19T01:04:18.791 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.791 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.791 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:18.791 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711228+0000", 2024-04-19T01:04:18.791 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.032957575000000003 2024-04-19T01:04:18.791 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.791 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.791 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:18.791 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711281+0000", 2024-04-19T01:04:18.791 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3737999999999999e-05 2024-04-19T01:04:18.791 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:18.792 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:18.792 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:18.792 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.792 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.792 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds5 47/31 ECSubWrite(tid=62, reqid=client.4604.0:331, at_version=47'33, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:04:18.792 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.682545+0000", 2024-04-19T01:04:18.792 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 1.0446948549999999, 2024-04-19T01:04:18.792 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.028771769999999999, 2024-04-19T01:04:18.792 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:18.792 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:18.792 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:18.792 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:18.793 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.793 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:18.793 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682545+0000", 2024-04-19T01:04:18.793 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:18.793 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.793 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.793 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:18.793 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682545+0000", 2024-04-19T01:04:18.793 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:18.793 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.793 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.793 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:18.793 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682546+0000", 2024-04-19T01:04:18.793 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.567e-06 2024-04-19T01:04:18.794 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.794 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.794 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:18.794 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682965+0000", 2024-04-19T01:04:18.794 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00041881399999999999 2024-04-19T01:04:18.794 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.794 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.794 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:18.794 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682999+0000", 2024-04-19T01:04:18.794 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3738999999999997e-05 2024-04-19T01:04:18.794 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.794 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.794 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:18.795 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.683021+0000", 2024-04-19T01:04:18.795 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2592e-05 2024-04-19T01:04:18.795 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.795 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.795 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:18.795 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.691707+0000", 2024-04-19T01:04:18.795 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0086853629999999998 2024-04-19T01:04:18.795 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.795 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.795 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:18.795 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.691732+0000", 2024-04-19T01:04:18.795 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5366000000000001e-05 2024-04-19T01:04:18.795 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.796 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.796 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:18.796 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711288+0000", 2024-04-19T01:04:18.796 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019555922999999999 2024-04-19T01:04:18.796 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.796 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.796 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:18.796 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711316+0000", 2024-04-19T01:04:18.796 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8405999999999999e-05 2024-04-19T01:04:18.796 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:18.796 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:18.796 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:18.796 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.797 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.797 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds5 47/31 ECSubWrite(tid=63, reqid=client.4604.0:332, at_version=47'34, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:04:18.797 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.688708+0000", 2024-04-19T01:04:18.797 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 1.0385319580000001, 2024-04-19T01:04:18.797 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.02327926, 2024-04-19T01:04:18.797 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:18.797 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:18.797 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:18.797 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:18.797 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.798 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:18.798 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.688708+0000", 2024-04-19T01:04:18.798 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:18.798 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.798 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.798 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:18.798 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.688708+0000", 2024-04-19T01:04:18.798 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:18.798 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.798 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.798 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:18.798 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.688709+0000", 2024-04-19T01:04:18.798 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5e-06 2024-04-19T01:04:18.799 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.799 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.799 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:18.799 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.688989+0000", 2024-04-19T01:04:18.799 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00028006800000000002 2024-04-19T01:04:18.799 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.799 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.799 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:18.799 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.689021+0000", 2024-04-19T01:04:18.799 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2026000000000001e-05 2024-04-19T01:04:18.799 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.799 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.800 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:18.800 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.689047+0000", 2024-04-19T01:04:18.800 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5809999999999999e-05 2024-04-19T01:04:18.800 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.800 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.800 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:18.800 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.704765+0000", 2024-04-19T01:04:18.800 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015717913 2024-04-19T01:04:18.800 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.800 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.800 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:18.800 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.704793+0000", 2024-04-19T01:04:18.801 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8143000000000001e-05 2024-04-19T01:04:18.801 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.801 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.801 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:18.801 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711907+0000", 2024-04-19T01:04:18.801 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071141879999999996 2024-04-19T01:04:18.801 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.801 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.801 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:18.801 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711987+0000", 2024-04-19T01:04:18.801 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9611999999999996e-05 2024-04-19T01:04:18.801 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:18.801 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:18.801 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:18.802 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.802 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.802 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds5 47/31 ECSubWrite(tid=64, reqid=client.4604.0:333, at_version=47'35, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:04:18.802 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.692104+0000", 2024-04-19T01:04:18.802 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 1.0351356979999999, 2024-04-19T01:04:18.802 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020504654000000001, 2024-04-19T01:04:18.802 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:18.802 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:18.802 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:18.802 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:18.802 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.802 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:18.802 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692104+0000", 2024-04-19T01:04:18.802 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:18.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.803 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.803 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:18.803 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692104+0000", 2024-04-19T01:04:18.803 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:18.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.803 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.803 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:18.803 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692105+0000", 2024-04-19T01:04:18.803 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.64e-06 2024-04-19T01:04:18.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.803 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.803 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:18.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692137+0000", 2024-04-19T01:04:18.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1631999999999998e-05 2024-04-19T01:04:18.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:18.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692165+0000", 2024-04-19T01:04:18.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7526e-05 2024-04-19T01:04:18.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:18.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692180+0000", 2024-04-19T01:04:18.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5146e-05 2024-04-19T01:04:18.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:18.805 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711392+0000", 2024-04-19T01:04:18.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019211967 2024-04-19T01:04:18.805 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.805 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:18.805 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711407+0000", 2024-04-19T01:04:18.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4866e-05 2024-04-19T01:04:18.805 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.805 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:18.805 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.712524+0000", 2024-04-19T01:04:18.806 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011173159999999999 2024-04-19T01:04:18.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:18.806 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.712608+0000", 2024-04-19T01:04:18.806 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4561000000000006e-05 2024-04-19T01:04:18.806 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:18.806 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:18.806 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:18.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.806 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds5 48/31 ECSubWrite(tid=67, reqid=osd.12.0:67, at_version=48'37, trim_to=0'0, roll_forward_to=47'35))", 2024-04-19T01:04:18.807 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:18.512882+0000", 2024-04-19T01:04:18.807 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.21435716599999999, 2024-04-19T01:04:18.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010685573, 2024-04-19T01:04:18.807 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:18.807 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:18.807 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:18.807 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:18.807 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.807 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:18.807 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.512882+0000", 2024-04-19T01:04:18.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:18.807 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.807 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.808 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:18.808 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.512882+0000", 2024-04-19T01:04:18.808 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:18.808 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.808 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.808 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:18.808 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.512883+0000", 2024-04-19T01:04:18.808 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8400000000000004e-07 2024-04-19T01:04:18.808 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.808 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.808 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:18.808 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.512946+0000", 2024-04-19T01:04:18.808 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2787999999999996e-05 2024-04-19T01:04:18.808 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.808 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.809 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:18.809 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.512980+0000", 2024-04-19T01:04:18.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4591999999999997e-05 2024-04-19T01:04:18.809 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.809 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.809 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:18.809 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.513011+0000", 2024-04-19T01:04:18.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0182e-05 2024-04-19T01:04:18.809 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.809 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.809 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:18.809 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.513775+0000", 2024-04-19T01:04:18.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00076438700000000005 2024-04-19T01:04:18.809 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.810 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.810 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:18.810 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.513817+0000", 2024-04-19T01:04:18.810 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1835000000000002e-05 2024-04-19T01:04:18.810 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.810 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.810 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:18.810 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.523393+0000", 2024-04-19T01:04:18.810 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0095763640000000004 2024-04-19T01:04:18.810 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.810 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.810 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:18.810 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.523568+0000", 2024-04-19T01:04:18.811 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017474100000000001 2024-04-19T01:04:18.811 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:18.811 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:18.811 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:18.811 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.811 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.811 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.3s2 48/31 ECSubWrite(tid=43, reqid=osd.15.0:43, at_version=48'32, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:04:18.811 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:18.513772+0000", 2024-04-19T01:04:18.811 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.21346786800000001, 2024-04-19T01:04:18.811 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0095513420000000009, 2024-04-19T01:04:18.811 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:18.811 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:18.811 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:18.811 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:18.811 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.812 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:18.812 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.513772+0000", 2024-04-19T01:04:18.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:18.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.812 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:18.812 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.513772+0000", 2024-04-19T01:04:18.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:18.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.812 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:18.812 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.513772+0000", 2024-04-19T01:04:18.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7700000000000001e-07 2024-04-19T01:04:18.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:18.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.513793+0000", 2024-04-19T01:04:18.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1107999999999998e-05 2024-04-19T01:04:18.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:18.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.513810+0000", 2024-04-19T01:04:18.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7363e-05 2024-04-19T01:04:18.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:18.814 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.513819+0000", 2024-04-19T01:04:18.814 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6419999999999996e-06 2024-04-19T01:04:18.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:18.814 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.513965+0000", 2024-04-19T01:04:18.814 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014563600000000001 2024-04-19T01:04:18.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:18.814 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.513987+0000", 2024-04-19T01:04:18.814 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2342000000000001e-05 2024-04-19T01:04:18.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.815 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.815 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:18.815 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.523199+0000", 2024-04-19T01:04:18.815 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0092117520000000001 2024-04-19T01:04:18.815 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.815 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.815 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:18.815 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.523323+0000", 2024-04-19T01:04:18.815 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000124222 2024-04-19T01:04:18.815 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:18.815 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:18.815 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:18.815 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.815 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.816 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds5 48/31 ECSubWrite(tid=68, reqid=osd.12.0:68, at_version=48'39, trim_to=0'0, roll_forward_to=47'35))", 2024-04-19T01:04:18.816 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:18.514474+0000", 2024-04-19T01:04:18.816 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.212765498, 2024-04-19T01:04:18.816 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010441201000000001, 2024-04-19T01:04:18.816 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:18.816 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:18.816 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:18.816 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:18.816 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.816 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:18.816 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.514474+0000", 2024-04-19T01:04:18.816 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:18.816 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.817 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:18.817 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.514474+0000", 2024-04-19T01:04:18.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:18.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.817 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:18.817 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.514476+0000", 2024-04-19T01:04:18.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.64e-06 2024-04-19T01:04:18.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.817 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:18.817 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.514529+0000", 2024-04-19T01:04:18.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3612000000000001e-05 2024-04-19T01:04:18.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:18.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.514559+0000", 2024-04-19T01:04:18.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9496e-05 2024-04-19T01:04:18.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:18.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.514581+0000", 2024-04-19T01:04:18.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2042e-05 2024-04-19T01:04:18.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.819 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:18.819 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.518726+0000", 2024-04-19T01:04:18.819 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0041449369999999996 2024-04-19T01:04:18.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.819 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:18.819 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.518778+0000", 2024-04-19T01:04:18.819 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2525000000000003e-05 2024-04-19T01:04:18.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.820 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:18.820 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.524757+0000", 2024-04-19T01:04:18.820 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0059790160000000002 2024-04-19T01:04:18.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.820 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:18.820 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.524915+0000", 2024-04-19T01:04:18.820 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015793299999999999 2024-04-19T01:04:18.820 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:18.820 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:18.820 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:18.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.821 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.fs6 48/31 ECSubWrite(tid=76, reqid=client.4604.0:345, at_version=48'22, trim_to=0'0, roll_forward_to=47'19))", 2024-04-19T01:04:18.821 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:18.538520+0000", 2024-04-19T01:04:18.821 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.18871924400000001, 2024-04-19T01:04:18.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019548551000000001, 2024-04-19T01:04:18.821 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:18.821 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:18.821 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:18.821 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:18.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.821 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:18.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.538520+0000", 2024-04-19T01:04:18.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:18.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:18.822 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.538520+0000", 2024-04-19T01:04:18.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:18.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:18.822 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.538521+0000", 2024-04-19T01:04:18.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.001e-06 2024-04-19T01:04:18.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:18.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.538786+0000", 2024-04-19T01:04:18.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026452700000000002 2024-04-19T01:04:18.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:18.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.538803+0000", 2024-04-19T01:04:18.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7428999999999999e-05 2024-04-19T01:04:18.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.824 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:18.824 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.538813+0000", 2024-04-19T01:04:18.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7040000000000006e-06 2024-04-19T01:04:18.824 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.824 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.824 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:18.824 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.538943+0000", 2024-04-19T01:04:18.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013024400000000001 2024-04-19T01:04:18.825 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.825 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.825 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:18.825 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.538970+0000", 2024-04-19T01:04:18.825 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7279000000000001e-05 2024-04-19T01:04:18.825 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.825 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.825 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:18.825 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.557980+0000", 2024-04-19T01:04:18.825 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019009169999999999 2024-04-19T01:04:18.825 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:18.825 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:18.825 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:18.826 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.558069+0000", 2024-04-19T01:04:18.826 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9197000000000007e-05 2024-04-19T01:04:18.826 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:18.826 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:18.826 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:18.826 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:18.826 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:18.826 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:04:18.828 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-19T01:04:18.828 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:04:18.828+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:04:18.929 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:04:18.929+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:04:18.997 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:04:18.997 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-04-19T01:04:18.997 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-04-19T01:04:18.997 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:04:19.011 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-19T01:04:19.054 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:19.056+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:19.130 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:04:19.127+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:04:19.132 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:04:19.132 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-04-19T01:04:19.132 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-04-19T01:04:19.132 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-04-19T01:04:19.132 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:04:19.146 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-19T01:04:19.231 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:04:19.227+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:04:19.313 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:04:19.313 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-04-19T01:04:19.313 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-04-19T01:04:19.313 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-04-19T01:04:19.313 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.313 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:240 3.1s0 3:82ff4f48:::smithi15528762-41:head [read 0~688128,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:19.313 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.861180+0000", 2024-04-19T01:04:19.314 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.4533566239999995, 2024-04-19T01:04:19.314 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.026860730999999999, 2024-04-19T01:04:19.314 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.314 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.314 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-04-19T01:04:19.314 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:04:19.314 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:04:19.314 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:19.314 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 240 2024-04-19T01:04:19.314 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.314 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.314 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.314 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.315 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861180+0000", 2024-04-19T01:04:19.315 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.315 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.315 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.315 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.315 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861180+0000", 2024-04-19T01:04:19.315 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.315 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.315 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.315 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.315 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861182+0000", 2024-04-19T01:04:19.315 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.654e-06 2024-04-19T01:04:19.315 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.316 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.316 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.316 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861212+0000", 2024-04-19T01:04:19.316 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0511999999999999e-05 2024-04-19T01:04:19.316 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.316 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.316 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.316 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861234+0000", 2024-04-19T01:04:19.316 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2135999999999999e-05 2024-04-19T01:04:19.316 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.316 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.316 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.316 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861250+0000", 2024-04-19T01:04:19.316 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6141999999999999e-05 2024-04-19T01:04:19.317 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.317 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.317 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.317 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.865146+0000", 2024-04-19T01:04:19.317 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0038953350000000002 2024-04-19T01:04:19.317 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.317 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.317 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:04:19.317 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.865375+0000", 2024-04-19T01:04:19.317 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00022880900000000001 2024-04-19T01:04:19.317 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.317 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.317 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:19.318 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.888041+0000", 2024-04-19T01:04:19.318 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022666143 2024-04-19T01:04:19.318 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.318 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.318 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.318 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.318 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.318 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:241 3.1s0 3:82ff4f48:::smithi15528762-41:head [read 0~688128,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:19.318 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.861274+0000", 2024-04-19T01:04:19.318 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.453262777, 2024-04-19T01:04:19.318 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.027990521000000001, 2024-04-19T01:04:19.318 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.318 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.319 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-04-19T01:04:19.319 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:04:19.319 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:04:19.319 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:19.319 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 241 2024-04-19T01:04:19.319 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.319 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.319 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.319 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.319 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861274+0000", 2024-04-19T01:04:19.320 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.320 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.320 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.320 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.320 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861274+0000", 2024-04-19T01:04:19.320 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.320 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.320 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.320 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.320 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861275+0000", 2024-04-19T01:04:19.321 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3030000000000001e-06 2024-04-19T01:04:19.321 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.321 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.321 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.321 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861282+0000", 2024-04-19T01:04:19.321 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8569999999999997e-06 2024-04-19T01:04:19.321 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.321 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.322 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.322 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861300+0000", 2024-04-19T01:04:19.322 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7898999999999999e-05 2024-04-19T01:04:19.322 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.322 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.322 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.322 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861314+0000", 2024-04-19T01:04:19.322 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.364e-05 2024-04-19T01:04:19.322 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.322 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.322 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.322 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.868359+0000", 2024-04-19T01:04:19.322 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0070459019999999997 2024-04-19T01:04:19.322 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.323 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.323 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:04:19.323 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.868594+0000", 2024-04-19T01:04:19.323 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000234942 2024-04-19T01:04:19.323 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.323 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.323 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:19.323 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.889264+0000", 2024-04-19T01:04:19.323 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020669977999999999 2024-04-19T01:04:19.323 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.323 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.323 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.323 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.324 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.324 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s6 43/31 ECSubWrite(tid=67, reqid=osd.13.0:67, at_version=43'39, trim_to=0'0, roll_forward_to=41'37))", 2024-04-19T01:04:19.324 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.867247+0000", 2024-04-19T01:04:19.324 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.4472900209999997, 2024-04-19T01:04:19.324 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019643094999999999, 2024-04-19T01:04:19.324 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.324 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.324 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:19.324 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.324 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.324 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.324 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.867247+0000", 2024-04-19T01:04:19.324 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.324 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.324 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.325 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.325 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.867247+0000", 2024-04-19T01:04:19.325 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.325 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.325 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.325 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.325 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.867247+0000", 2024-04-19T01:04:19.325 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7299999999999996e-07 2024-04-19T01:04:19.325 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.325 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.325 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.325 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.867324+0000", 2024-04-19T01:04:19.325 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6981e-05 2024-04-19T01:04:19.325 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.326 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.326 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.326 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.867357+0000", 2024-04-19T01:04:19.326 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3269000000000003e-05 2024-04-19T01:04:19.326 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.326 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.326 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.326 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.867380+0000", 2024-04-19T01:04:19.326 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2272999999999998e-05 2024-04-19T01:04:19.326 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.326 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.326 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.327 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.870949+0000", 2024-04-19T01:04:19.327 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0035693259999999998 2024-04-19T01:04:19.327 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.327 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.327 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:19.327 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.870973+0000", 2024-04-19T01:04:19.327 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3859e-05 2024-04-19T01:04:19.327 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.327 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.327 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:19.327 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.886787+0000", 2024-04-19T01:04:19.327 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0158142 2024-04-19T01:04:19.327 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.328 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.328 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:19.328 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.886890+0000", 2024-04-19T01:04:19.328 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000102614 2024-04-19T01:04:19.328 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.328 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.328 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.328 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.328 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.328 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s6 43/31 ECSubWrite(tid=69, reqid=osd.13.0:69, at_version=43'41, trim_to=0'0, roll_forward_to=41'37))", 2024-04-19T01:04:19.328 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.867419+0000", 2024-04-19T01:04:19.328 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.4471180500000003, 2024-04-19T01:04:19.329 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019827727999999999, 2024-04-19T01:04:19.329 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.329 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.329 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:19.329 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.329 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.329 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.329 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.867419+0000", 2024-04-19T01:04:19.330 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.330 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.330 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.330 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.330 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.867419+0000", 2024-04-19T01:04:19.330 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.330 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.330 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.330 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.330 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.867420+0000", 2024-04-19T01:04:19.330 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.04e-06 2024-04-19T01:04:19.330 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.330 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.330 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.331 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.867470+0000", 2024-04-19T01:04:19.331 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9914999999999999e-05 2024-04-19T01:04:19.331 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.331 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.331 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.331 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.867498+0000", 2024-04-19T01:04:19.332 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8969e-05 2024-04-19T01:04:19.332 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.332 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.332 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.332 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.867517+0000", 2024-04-19T01:04:19.332 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8675999999999999e-05 2024-04-19T01:04:19.332 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.332 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.332 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.332 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.871964+0000", 2024-04-19T01:04:19.332 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0044469260000000004 2024-04-19T01:04:19.332 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.332 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.333 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:19.333 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.871987+0000", 2024-04-19T01:04:19.333 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2521999999999999e-05 2024-04-19T01:04:19.333 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.333 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.333 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:19.333 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.887168+0000", 2024-04-19T01:04:19.333 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015181019 2024-04-19T01:04:19.333 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.333 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.334 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:19.334 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.887246+0000", 2024-04-19T01:04:19.334 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8660999999999998e-05 2024-04-19T01:04:19.334 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.334 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.334 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.334 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.334 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.334 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpReadReply(3.1s0 43/31 ECSubReadReply(tid=13, attrs_read=0))", 2024-04-19T01:04:19.335 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.868861+0000", 2024-04-19T01:04:19.335 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.445675617, 2024-04-19T01:04:19.335 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018039893000000001, 2024-04-19T01:04:19.335 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.335 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.335 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:19.335 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.335 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.335 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.335 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.868861+0000", 2024-04-19T01:04:19.335 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.335 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.335 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.336 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.336 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.868861+0000", 2024-04-19T01:04:19.336 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.336 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.336 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.336 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.336 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.868862+0000", 2024-04-19T01:04:19.336 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3969999999999999e-06 2024-04-19T01:04:19.336 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.336 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.336 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.337 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.869204+0000", 2024-04-19T01:04:19.337 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000341961 2024-04-19T01:04:19.337 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.337 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.337 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.337 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.869235+0000", 2024-04-19T01:04:19.337 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0932000000000001e-05 2024-04-19T01:04:19.337 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.337 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.337 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.337 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.869261+0000", 2024-04-19T01:04:19.337 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5352999999999999e-05 2024-04-19T01:04:19.337 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.337 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.337 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.338 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.873546+0000", 2024-04-19T01:04:19.338 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0042855109999999997 2024-04-19T01:04:19.338 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.338 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.338 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:19.338 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.886901+0000", 2024-04-19T01:04:19.338 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013354738999999999 2024-04-19T01:04:19.338 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.338 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.338 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.338 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.338 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.338 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s6 43/31 ECSubWrite(tid=70, reqid=client.4604.0:235, at_version=43'43, trim_to=0'0, roll_forward_to=41'37))", 2024-04-19T01:04:19.338 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.870558+0000", 2024-04-19T01:04:19.339 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.4439783229999996, 2024-04-19T01:04:19.339 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017980526, 2024-04-19T01:04:19.339 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.339 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.339 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:19.339 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.339 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.339 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.340 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.870558+0000", 2024-04-19T01:04:19.340 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.340 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.340 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.340 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.340 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.870558+0000", 2024-04-19T01:04:19.340 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.340 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.340 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.340 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.340 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.870560+0000", 2024-04-19T01:04:19.340 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.074e-06 2024-04-19T01:04:19.340 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.340 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.341 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.341 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.871178+0000", 2024-04-19T01:04:19.341 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00061785200000000001 2024-04-19T01:04:19.341 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.341 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.341 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.341 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.871213+0000", 2024-04-19T01:04:19.341 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4941999999999999e-05 2024-04-19T01:04:19.341 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.341 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.342 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.342 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.871240+0000", 2024-04-19T01:04:19.342 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6806000000000001e-05 2024-04-19T01:04:19.342 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.342 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.342 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.342 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.873533+0000", 2024-04-19T01:04:19.345 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022929500000000002 2024-04-19T01:04:19.345 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.345 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.345 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:19.345 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.873555+0000", 2024-04-19T01:04:19.345 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1852e-05 2024-04-19T01:04:19.345 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.346 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.346 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:19.346 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.888450+0000", 2024-04-19T01:04:19.346 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014895293 2024-04-19T01:04:19.346 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.346 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.346 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:19.346 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.888539+0000", 2024-04-19T01:04:19.346 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8757000000000004e-05 2024-04-19T01:04:19.346 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.346 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.346 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.346 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.346 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.347 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpReadReply(3.1s0 43/31 ECSubReadReply(tid=13, attrs_read=0))", 2024-04-19T01:04:19.347 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.870658+0000", 2024-04-19T01:04:19.347 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.4438785220000003, 2024-04-19T01:04:19.347 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.01638684, 2024-04-19T01:04:19.347 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.347 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.347 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:19.347 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.347 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.347 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.347 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.870658+0000", 2024-04-19T01:04:19.347 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.347 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.347 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.348 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.348 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.870658+0000", 2024-04-19T01:04:19.348 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.348 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.348 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.348 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.348 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.870660+0000", 2024-04-19T01:04:19.348 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6199999999999999e-06 2024-04-19T01:04:19.348 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.348 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.348 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.348 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.871988+0000", 2024-04-19T01:04:19.348 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0013283870000000001 2024-04-19T01:04:19.348 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.348 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.348 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.349 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.872028+0000", 2024-04-19T01:04:19.349 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9925000000000001e-05 2024-04-19T01:04:19.349 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.349 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.349 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.349 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.872048+0000", 2024-04-19T01:04:19.349 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0366000000000001e-05 2024-04-19T01:04:19.350 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.350 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.350 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.350 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.886965+0000", 2024-04-19T01:04:19.350 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014916448000000001 2024-04-19T01:04:19.350 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.351 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.351 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:19.351 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.887045+0000", 2024-04-19T01:04:19.351 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0093999999999996e-05 2024-04-19T01:04:19.351 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.351 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.351 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.351 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.351 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.351 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpReadReply(3.1s0 43/31 ECSubReadReply(tid=13, attrs_read=0))", 2024-04-19T01:04:19.351 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.871264+0000", 2024-04-19T01:04:19.351 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.4432725929999997, 2024-04-19T01:04:19.351 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016812595999999999, 2024-04-19T01:04:19.351 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.352 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.352 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:19.352 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.352 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.352 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.352 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.871264+0000", 2024-04-19T01:04:19.352 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.352 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.352 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.352 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.352 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.871264+0000", 2024-04-19T01:04:19.352 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.352 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.352 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.353 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.353 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.871264+0000", 2024-04-19T01:04:19.353 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.353 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.353 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.353 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.353 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.871264+0000", 2024-04-19T01:04:19.353 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.353 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.353 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.353 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.353 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.871312+0000", 2024-04-19T01:04:19.353 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8111000000000001e-05 2024-04-19T01:04:19.353 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.353 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.354 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.354 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.871331+0000", 2024-04-19T01:04:19.354 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9162999999999999e-05 2024-04-19T01:04:19.354 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.354 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.354 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.354 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.887241+0000", 2024-04-19T01:04:19.354 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015909553999999999 2024-04-19T01:04:19.354 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.354 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.354 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:19.354 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.888077+0000", 2024-04-19T01:04:19.354 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00083576799999999997 2024-04-19T01:04:19.354 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.355 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.355 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.355 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.355 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.355 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpReadReply(3.1s0 43/31 ECSubReadReply(tid=14, attrs_read=0))", 2024-04-19T01:04:19.355 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.871358+0000", 2024-04-19T01:04:19.355 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.4431790969999998, 2024-04-19T01:04:19.355 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016856771, 2024-04-19T01:04:19.355 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.355 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.355 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:19.355 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.355 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.355 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.356 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.871358+0000", 2024-04-19T01:04:19.356 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.356 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.356 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.356 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.356 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.871358+0000", 2024-04-19T01:04:19.356 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.356 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.356 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.356 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.357 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.871359+0000", 2024-04-19T01:04:19.357 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5099999999999999e-06 2024-04-19T01:04:19.357 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.357 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.357 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.357 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.871840+0000", 2024-04-19T01:04:19.357 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00048063000000000001 2024-04-19T01:04:19.357 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.357 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.357 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.357 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.871863+0000", 2024-04-19T01:04:19.357 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3186000000000001e-05 2024-04-19T01:04:19.357 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.357 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.358 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.358 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.871879+0000", 2024-04-19T01:04:19.358 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6646e-05 2024-04-19T01:04:19.358 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.358 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.358 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.358 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.888153+0000", 2024-04-19T01:04:19.358 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016273721000000001 2024-04-19T01:04:19.358 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.358 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.358 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:19.358 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.888214+0000", 2024-04-19T01:04:19.358 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1078e-05 2024-04-19T01:04:19.358 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.359 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.359 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.359 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.359 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.359 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s1 43/31 ECSubWrite(tid=43, reqid=client.4604.0:251, at_version=43'29, trim_to=0'0, roll_forward_to=43'27))", 2024-04-19T01:04:19.359 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.985924+0000", 2024-04-19T01:04:19.359 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.3286123239999998, 2024-04-19T01:04:19.359 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.029591623000000001, 2024-04-19T01:04:19.359 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.359 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.359 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:19.359 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.359 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.359 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.360 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.985924+0000", 2024-04-19T01:04:19.360 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.360 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.360 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.360 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.360 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.985924+0000", 2024-04-19T01:04:19.360 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.360 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.360 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.360 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.360 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.985927+0000", 2024-04-19T01:04:19.360 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3460000000000001e-06 2024-04-19T01:04:19.360 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.360 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.361 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.361 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986452+0000", 2024-04-19T01:04:19.361 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00052555199999999999 2024-04-19T01:04:19.361 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.361 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.361 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.361 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986541+0000", 2024-04-19T01:04:19.361 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8352999999999999e-05 2024-04-19T01:04:19.361 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.361 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.361 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.361 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986597+0000", 2024-04-19T01:04:19.361 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6144999999999999e-05 2024-04-19T01:04:19.361 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.362 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.362 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.362 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986776+0000", 2024-04-19T01:04:19.362 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017914999999999999 2024-04-19T01:04:19.362 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.362 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.362 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:19.362 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986811+0000", 2024-04-19T01:04:19.362 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5233e-05 2024-04-19T01:04:19.362 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.362 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.362 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:19.362 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.015444+0000", 2024-04-19T01:04:19.362 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.028632676999999999 2024-04-19T01:04:19.363 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.363 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.363 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:19.363 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.015516+0000", 2024-04-19T01:04:19.363 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2167000000000002e-05 2024-04-19T01:04:19.363 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.363 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.363 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.363 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.363 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.363 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds6 43/31 ECSubWrite(tid=38, reqid=client.4604.0:253, at_version=43'14, trim_to=0'0, roll_forward_to=43'12))", 2024-04-19T01:04:19.363 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.987449+0000", 2024-04-19T01:04:19.363 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.327087444, 2024-04-19T01:04:19.363 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.027803888999999998, 2024-04-19T01:04:19.363 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.364 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.364 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:19.364 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.364 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.364 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.364 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987449+0000", 2024-04-19T01:04:19.364 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.364 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.364 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.364 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.364 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987449+0000", 2024-04-19T01:04:19.364 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.364 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.364 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.365 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.365 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987451+0000", 2024-04-19T01:04:19.365 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2929999999999999e-06 2024-04-19T01:04:19.365 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.365 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.365 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.365 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988097+0000", 2024-04-19T01:04:19.365 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00064595099999999997 2024-04-19T01:04:19.365 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.365 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.365 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.365 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988140+0000", 2024-04-19T01:04:19.365 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2809000000000003e-05 2024-04-19T01:04:19.365 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.365 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.366 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.366 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988168+0000", 2024-04-19T01:04:19.366 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7498999999999999e-05 2024-04-19T01:04:19.366 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.366 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.366 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.366 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988319+0000", 2024-04-19T01:04:19.366 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000151621 2024-04-19T01:04:19.366 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.366 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.366 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:19.366 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988345+0000", 2024-04-19T01:04:19.366 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5846e-05 2024-04-19T01:04:19.366 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.366 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.367 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:19.367 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.015183+0000", 2024-04-19T01:04:19.367 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.026837928 2024-04-19T01:04:19.367 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.367 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.367 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:19.367 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.015253+0000", 2024-04-19T01:04:19.367 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9942000000000002e-05 2024-04-19T01:04:19.367 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.367 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.367 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.367 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.367 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.367 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:289 3.1s0 3:8ea800b0:::smithi15528762-8:head [stat,delete] snapc 7=[7,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-19T01:04:19.368 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.872650+0000", 2024-04-19T01:04:19.368 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.441886556, 2024-04-19T01:04:19.368 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.043399042999999998, 2024-04-19T01:04:19.368 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.368 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.368 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:19.368 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:04:19.368 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:04:19.368 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:19.368 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 289 2024-04-19T01:04:19.368 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.369 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.369 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.369 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.369 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.872650+0000", 2024-04-19T01:04:19.369 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.369 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.369 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.369 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.369 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.872650+0000", 2024-04-19T01:04:19.369 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.369 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.369 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.369 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.369 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.872653+0000", 2024-04-19T01:04:19.369 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.23e-06 2024-04-19T01:04:19.370 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.370 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.370 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.370 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.872669+0000", 2024-04-19T01:04:19.370 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5293000000000001e-05 2024-04-19T01:04:19.370 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.370 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.370 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.370 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.872701+0000", 2024-04-19T01:04:19.370 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2391999999999998e-05 2024-04-19T01:04:19.370 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.370 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.370 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.370 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.872721+0000", 2024-04-19T01:04:19.371 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0400000000000001e-05 2024-04-19T01:04:19.371 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.371 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.371 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.371 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.872889+0000", 2024-04-19T01:04:19.371 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016723000000000001 2024-04-19T01:04:19.371 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.371 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.371 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:04:19.371 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.873172+0000", 2024-04-19T01:04:19.371 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000283338 2024-04-19T01:04:19.371 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.371 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.371 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:19.371 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.875406+0000", 2024-04-19T01:04:19.372 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022335839999999998 2024-04-19T01:04:19.372 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.372 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.372 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:19.372 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.914580+0000", 2024-04-19T01:04:19.372 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.039174253999999999 2024-04-19T01:04:19.372 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.372 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.372 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-04-19T01:04:19.372 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.914900+0000", 2024-04-19T01:04:19.372 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000320663 2024-04-19T01:04:19.372 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.373 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.373 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:19.373 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.916049+0000", 2024-04-19T01:04:19.373 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001148659 2024-04-19T01:04:19.373 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.373 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.373 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.373 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.373 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.373 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s1 44/31 ECSubWrite(tid=49, reqid=osd.8.0:49, at_version=44'33, trim_to=0'0, roll_forward_to=43'29))", 2024-04-19T01:04:19.373 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.876436+0000", 2024-04-19T01:04:19.373 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.4381008729999998, 2024-04-19T01:04:19.373 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024659179, 2024-04-19T01:04:19.374 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.374 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.374 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:19.374 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.374 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.374 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.374 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876436+0000", 2024-04-19T01:04:19.374 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.374 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.374 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.374 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.374 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876436+0000", 2024-04-19T01:04:19.374 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.375 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.375 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.375 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.375 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876437+0000", 2024-04-19T01:04:19.375 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2860000000000001e-06 2024-04-19T01:04:19.375 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.375 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.375 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.375 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876485+0000", 2024-04-19T01:04:19.375 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8294999999999998e-05 2024-04-19T01:04:19.375 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.375 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.375 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.375 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876512+0000", 2024-04-19T01:04:19.376 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6363000000000002e-05 2024-04-19T01:04:19.376 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.376 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.376 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.376 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876531+0000", 2024-04-19T01:04:19.376 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8902999999999998e-05 2024-04-19T01:04:19.376 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.376 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.376 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.376 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.877917+0000", 2024-04-19T01:04:19.377 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0013865870000000001 2024-04-19T01:04:19.377 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.377 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.377 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:19.377 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.877944+0000", 2024-04-19T01:04:19.377 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6553000000000001e-05 2024-04-19T01:04:19.377 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.377 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.377 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:19.377 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.901009+0000", 2024-04-19T01:04:19.377 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023064772000000001 2024-04-19T01:04:19.377 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.377 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.378 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:19.378 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.901095+0000", 2024-04-19T01:04:19.378 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6420000000000003e-05 2024-04-19T01:04:19.378 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.378 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.378 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.378 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.378 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.378 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s6 44/31 ECSubWrite(tid=82, reqid=client.4604.0:280, at_version=44'50, trim_to=0'0, roll_forward_to=43'48))", 2024-04-19T01:04:19.378 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.882093+0000", 2024-04-19T01:04:19.378 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.4324432619999996, 2024-04-19T01:04:19.378 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.042938993000000002, 2024-04-19T01:04:19.378 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.378 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.379 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:19.379 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.379 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.379 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.379 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882093+0000", 2024-04-19T01:04:19.379 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.379 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.379 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.379 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.379 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882093+0000", 2024-04-19T01:04:19.379 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.379 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.379 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.379 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.380 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882094+0000", 2024-04-19T01:04:19.380 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0130000000000001e-06 2024-04-19T01:04:19.380 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.380 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.380 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.380 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882590+0000", 2024-04-19T01:04:19.380 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00049558999999999996 2024-04-19T01:04:19.380 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.380 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.380 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.380 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882631+0000", 2024-04-19T01:04:19.380 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1399000000000001e-05 2024-04-19T01:04:19.380 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.380 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.380 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.381 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882654+0000", 2024-04-19T01:04:19.381 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2535999999999999e-05 2024-04-19T01:04:19.381 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.381 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.381 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.381 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882871+0000", 2024-04-19T01:04:19.381 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021730999999999999 2024-04-19T01:04:19.381 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.381 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.381 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:19.381 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882896+0000", 2024-04-19T01:04:19.381 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5111999999999999e-05 2024-04-19T01:04:19.381 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.381 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.381 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:19.382 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.924961+0000", 2024-04-19T01:04:19.382 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.042065022000000001 2024-04-19T01:04:19.382 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.382 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.382 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:19.382 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.925032+0000", 2024-04-19T01:04:19.382 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1011000000000002e-05 2024-04-19T01:04:19.382 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.382 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.382 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.382 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.382 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.382 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s6 44/31 ECSubWrite(tid=84, reqid=client.4604.0:281, at_version=44'51, trim_to=0'0, roll_forward_to=43'48))", 2024-04-19T01:04:19.382 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.907467+0000", 2024-04-19T01:04:19.382 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.4070697320000001, 2024-04-19T01:04:19.383 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018626453000000001, 2024-04-19T01:04:19.383 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.383 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.383 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:19.383 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.383 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.383 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.383 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.907467+0000", 2024-04-19T01:04:19.383 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.383 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.383 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.383 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.383 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.907467+0000", 2024-04-19T01:04:19.384 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.384 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.384 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.384 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.384 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.907468+0000", 2024-04-19T01:04:19.384 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3230000000000001e-06 2024-04-19T01:04:19.385 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.385 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.385 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.385 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.907725+0000", 2024-04-19T01:04:19.385 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00025646499999999999 2024-04-19T01:04:19.385 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.385 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.385 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.385 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.907754+0000", 2024-04-19T01:04:19.385 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9799e-05 2024-04-19T01:04:19.386 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.386 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.386 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.386 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.907776+0000", 2024-04-19T01:04:19.386 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1849999999999999e-05 2024-04-19T01:04:19.386 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.386 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.386 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.386 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.914523+0000", 2024-04-19T01:04:19.386 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0067470459999999996 2024-04-19T01:04:19.386 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.386 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.386 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:19.386 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.914550+0000", 2024-04-19T01:04:19.387 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6259e-05 2024-04-19T01:04:19.387 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.387 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.387 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:19.387 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.926057+0000", 2024-04-19T01:04:19.387 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011507224999999999 2024-04-19T01:04:19.387 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.387 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.387 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:19.387 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.926093+0000", 2024-04-19T01:04:19.387 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6486000000000003e-05 2024-04-19T01:04:19.387 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.387 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.387 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.387 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.388 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.388 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.0s6 44/31 ECSubWrite(tid=25, reqid=client.4604.0:284, at_version=44'22, trim_to=0'0, roll_forward_to=44'20))", 2024-04-19T01:04:19.388 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.921980+0000", 2024-04-19T01:04:19.388 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.392556989, 2024-04-19T01:04:19.388 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.036689795999999997, 2024-04-19T01:04:19.388 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.388 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.388 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:19.388 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.388 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.388 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.388 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.921980+0000", 2024-04-19T01:04:19.388 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.388 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.388 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.389 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.389 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.921980+0000", 2024-04-19T01:04:19.389 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.389 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.389 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.389 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.389 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.921981+0000", 2024-04-19T01:04:19.389 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.378e-06 2024-04-19T01:04:19.389 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.389 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.389 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.389 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.923355+0000", 2024-04-19T01:04:19.389 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0013742820000000001 2024-04-19T01:04:19.389 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.389 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.390 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.390 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.923387+0000", 2024-04-19T01:04:19.390 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1702999999999998e-05 2024-04-19T01:04:19.390 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.390 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.390 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.391 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.923405+0000", 2024-04-19T01:04:19.391 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8071999999999999e-05 2024-04-19T01:04:19.391 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.391 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.391 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.391 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.923699+0000", 2024-04-19T01:04:19.391 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029384700000000002 2024-04-19T01:04:19.391 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.391 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.391 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:19.391 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.923727+0000", 2024-04-19T01:04:19.391 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8085999999999998e-05 2024-04-19T01:04:19.391 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.391 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.392 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:19.392 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.958618+0000", 2024-04-19T01:04:19.392 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.034891349000000002 2024-04-19T01:04:19.392 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.392 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.392 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:19.392 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.958669+0000", 2024-04-19T01:04:19.392 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1078999999999997e-05 2024-04-19T01:04:19.392 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.392 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.392 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.392 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.392 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.392 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds6 47/31 ECSubWrite(tid=61, reqid=client.4604.0:330, at_version=47'32, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:04:19.392 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.677697+0000", 2024-04-19T01:04:19.393 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 1.636840045, 2024-04-19T01:04:19.393 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.034267463999999997, 2024-04-19T01:04:19.393 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.393 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.393 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:19.393 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.393 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.393 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.393 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.677697+0000", 2024-04-19T01:04:19.393 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.393 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.393 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.393 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.677697+0000", 2024-04-19T01:04:19.394 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.394 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.394 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.394 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.394 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.677699+0000", 2024-04-19T01:04:19.394 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3929999999999998e-06 2024-04-19T01:04:19.394 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.394 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.394 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.394 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678182+0000", 2024-04-19T01:04:19.394 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00048326199999999998 2024-04-19T01:04:19.394 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.394 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.394 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.395 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678227+0000", 2024-04-19T01:04:19.395 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4301999999999997e-05 2024-04-19T01:04:19.395 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.395 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.395 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.395 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678254+0000", 2024-04-19T01:04:19.395 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7396e-05 2024-04-19T01:04:19.395 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.395 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.395 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.395 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678409+0000", 2024-04-19T01:04:19.395 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000154759 2024-04-19T01:04:19.395 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.395 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.396 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:19.396 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678438+0000", 2024-04-19T01:04:19.396 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8926000000000001e-05 2024-04-19T01:04:19.396 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.396 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.396 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:19.396 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711880+0000", 2024-04-19T01:04:19.396 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.033442209000000001 2024-04-19T01:04:19.396 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.396 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.396 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:19.396 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711964+0000", 2024-04-19T01:04:19.396 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4216999999999997e-05 2024-04-19T01:04:19.396 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.396 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.397 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.397 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.397 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.397 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds6 47/31 ECSubWrite(tid=62, reqid=client.4604.0:331, at_version=47'33, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:04:19.397 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.682679+0000", 2024-04-19T01:04:19.397 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 1.6318576550000001, 2024-04-19T01:04:19.397 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.029354201999999999, 2024-04-19T01:04:19.397 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.397 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.397 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:19.397 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.397 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.397 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.397 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682679+0000", 2024-04-19T01:04:19.397 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.398 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.398 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.398 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.398 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682679+0000", 2024-04-19T01:04:19.398 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.398 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.398 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.398 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.398 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682681+0000", 2024-04-19T01:04:19.398 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7799999999999999e-06 2024-04-19T01:04:19.398 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.398 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.398 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.398 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682945+0000", 2024-04-19T01:04:19.399 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000263875 2024-04-19T01:04:19.399 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.399 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.399 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.399 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682975+0000", 2024-04-19T01:04:19.399 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0216e-05 2024-04-19T01:04:19.399 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.399 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.399 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.399 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682990+0000", 2024-04-19T01:04:19.399 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5333e-05 2024-04-19T01:04:19.399 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.399 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.399 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.399 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692463+0000", 2024-04-19T01:04:19.400 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0094724660000000006 2024-04-19T01:04:19.400 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.400 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.400 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:19.400 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692479+0000", 2024-04-19T01:04:19.400 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5892e-05 2024-04-19T01:04:19.400 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.400 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.400 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:19.400 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711976+0000", 2024-04-19T01:04:19.400 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019497012000000001 2024-04-19T01:04:19.400 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.400 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.400 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:19.400 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.712033+0000", 2024-04-19T01:04:19.401 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7627999999999998e-05 2024-04-19T01:04:19.401 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.401 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.401 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.401 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.401 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.401 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds6 47/31 ECSubWrite(tid=63, reqid=client.4604.0:332, at_version=47'34, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:04:19.401 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.688944+0000", 2024-04-19T01:04:19.401 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 1.6255927940000001, 2024-04-19T01:04:19.401 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023924708999999999, 2024-04-19T01:04:19.401 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.401 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.401 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:19.401 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.401 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.402 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.402 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.688944+0000", 2024-04-19T01:04:19.402 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.402 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.402 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.402 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.402 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.688944+0000", 2024-04-19T01:04:19.402 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.402 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.402 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.402 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.402 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.688945+0000", 2024-04-19T01:04:19.402 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5540000000000001e-06 2024-04-19T01:04:19.403 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.403 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.403 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.403 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.689042+0000", 2024-04-19T01:04:19.403 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6903000000000004e-05 2024-04-19T01:04:19.403 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.403 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.403 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.403 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.689071+0000", 2024-04-19T01:04:19.403 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8249e-05 2024-04-19T01:04:19.403 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.404 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.404 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.404 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.689092+0000", 2024-04-19T01:04:19.404 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1685999999999999e-05 2024-04-19T01:04:19.404 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.404 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.404 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.404 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.705134+0000", 2024-04-19T01:04:19.404 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016041454 2024-04-19T01:04:19.404 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.404 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.404 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:19.404 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.705155+0000", 2024-04-19T01:04:19.404 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.177e-05 2024-04-19T01:04:19.405 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.405 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.405 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:19.405 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.712801+0000", 2024-04-19T01:04:19.405 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0076452489999999998 2024-04-19T01:04:19.405 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.405 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.405 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:19.405 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.712869+0000", 2024-04-19T01:04:19.405 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7843999999999996e-05 2024-04-19T01:04:19.405 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.405 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.405 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.405 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.405 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.406 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds6 47/31 ECSubWrite(tid=64, reqid=client.4604.0:333, at_version=47'35, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:04:19.406 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.692117+0000", 2024-04-19T01:04:19.406 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 1.6224199609999999, 2024-04-19T01:04:19.406 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020827234, 2024-04-19T01:04:19.406 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.406 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.406 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:19.406 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.406 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.406 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.406 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692117+0000", 2024-04-19T01:04:19.406 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.406 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.406 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.407 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.407 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692117+0000", 2024-04-19T01:04:19.407 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.407 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.407 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.407 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.407 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692118+0000", 2024-04-19T01:04:19.407 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7489999999999999e-06 2024-04-19T01:04:19.407 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.407 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.407 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.407 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692163+0000", 2024-04-19T01:04:19.407 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4342999999999999e-05 2024-04-19T01:04:19.407 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.407 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.408 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.408 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692204+0000", 2024-04-19T01:04:19.408 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1588999999999998e-05 2024-04-19T01:04:19.408 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.408 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.408 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.408 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692227+0000", 2024-04-19T01:04:19.408 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3059000000000001e-05 2024-04-19T01:04:19.408 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.408 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.408 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.408 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711560+0000", 2024-04-19T01:04:19.408 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019332623 2024-04-19T01:04:19.408 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.409 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.409 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:19.409 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711575+0000", 2024-04-19T01:04:19.409 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5218999999999999e-05 2024-04-19T01:04:19.409 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.409 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.409 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:19.409 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.712879+0000", 2024-04-19T01:04:19.409 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001303397 2024-04-19T01:04:19.409 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.409 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.409 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:19.409 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.712944+0000", 2024-04-19T01:04:19.409 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5254999999999998e-05 2024-04-19T01:04:19.409 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.409 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.410 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.410 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.410 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.410 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:04:19.411 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-19T01:04:19.412 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:04:19.413+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:04:19.512 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.13 to 1 (10000) 2024-04-19T01:04:19.512 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:04:19.511+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:04:19.518 INFO:tasks.rados.rados.0.smithi155.stdout:139: done (6 left) 2024-04-19T01:04:19.518 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 45 current snap is 8 2024-04-19T01:04:19.518 INFO:tasks.rados.rados.0.smithi155.stdout:133: finishing copy_from racing read to smithi15528762-23 2024-04-19T01:04:19.518 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 33 v 20 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-04-19T01:04:19.518 INFO:tasks.rados.rados.0.smithi155.stdout:133: finishing copy_from to smithi15528762-23 2024-04-19T01:04:19.518 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 23 v 14 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-04-19T01:04:19.518 INFO:tasks.rados.rados.0.smithi155.stdout:138: finishing rollback tid 0 to smithi15528762-27 2024-04-19T01:04:19.519 INFO:tasks.rados.rados.0.smithi155.stdout:135: finishing rollback tid 0 to smithi15528762-43 2024-04-19T01:04:19.519 INFO:tasks.rados.rados.0.smithi155.stdout:137: finishing copy_from racing read to smithi15528762-47 2024-04-19T01:04:19.519 INFO:tasks.rados.rados.0.smithi155.stdout:135: finishing rollback tid 1 to smithi15528762-43 2024-04-19T01:04:19.519 INFO:tasks.rados.rados.0.smithi155.stdout:135: finishing rollback tid 2 to smithi15528762-43 2024-04-19T01:04:19.519 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 43 v 73 (ObjNum 61 snap 5 seq_num 61) dirty exists 2024-04-19T01:04:19.519 INFO:tasks.rados.rados.0.smithi155.stdout:138: finishing rollback tid 1 to smithi15528762-27 2024-04-19T01:04:19.519 INFO:tasks.rados.rados.0.smithi155.stdout:138: finishing rollback tid 2 to smithi15528762-27 2024-04-19T01:04:19.519 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 27 v 26 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-04-19T01:04:19.519 INFO:tasks.rados.rados.0.smithi155.stdout:134: finishing write tid 1 to smithi15528762-6 2024-04-19T01:04:19.519 INFO:tasks.rados.rados.0.smithi155.stdout:134: finishing write tid 2 to smithi15528762-6 2024-04-19T01:04:19.520 INFO:tasks.rados.rados.0.smithi155.stdout:134: finishing write tid 3 to smithi15528762-6 2024-04-19T01:04:19.520 INFO:tasks.rados.rados.0.smithi155.stdout:134: finishing write tid 4 to smithi15528762-6 2024-04-19T01:04:19.520 INFO:tasks.rados.rados.0.smithi155.stdout:134: finishing write tid 5 to smithi15528762-6 2024-04-19T01:04:19.520 INFO:tasks.rados.rados.0.smithi155.stdout:134: oid 6 updating version 0 to 23 2024-04-19T01:04:19.520 INFO:tasks.rados.rados.0.smithi155.stdout:134: oid 6 updating version 23 to 24 2024-04-19T01:04:19.520 INFO:tasks.rados.rados.0.smithi155.stdout:134: oid 6 updating version 24 to 25 2024-04-19T01:04:19.520 INFO:tasks.rados.rados.0.smithi155.stdout:134: oid 6 version 25 is already newer than 22 2024-04-19T01:04:19.520 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 6 v 25 (ObjNum 78 snap 8 seq_num 78) dirty exists 2024-04-19T01:04:19.520 INFO:tasks.rados.rados.0.smithi155.stdout:134: left oid 6 (ObjNum 78 snap 8 seq_num 78) 2024-04-19T01:04:19.520 INFO:tasks.rados.rados.0.smithi155.stdout:137: finishing copy_from to smithi15528762-47 2024-04-19T01:04:19.521 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 47 v 35 (ObjNum 76 snap 8 seq_num 76) dirty exists 2024-04-19T01:04:19.521 INFO:tasks.rados.rados.0.smithi155.stdout:140: seq_num 79 ranges {0=573440,573440=425984,999424=344064} 2024-04-19T01:04:19.521 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-19T01:04:19.528 INFO:tasks.rados.rados.0.smithi155.stdout:140: writing smithi15528762-45 from 0 to 573440 tid 1 2024-04-19T01:04:19.537 INFO:tasks.rados.rados.0.smithi155.stdout:140: writing smithi15528762-45 from 573440 to 999424 tid 2 2024-04-19T01:04:19.544 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 14 1 2024-04-19T01:04:19.545 INFO:tasks.rados.rados.0.smithi155.stdout:140: writing smithi15528762-45 from 999424 to 1343488 tid 3 2024-04-19T01:04:19.545 INFO:tasks.rados.rados.0.smithi155.stdout:133: done (6 left) 2024-04-19T01:04:19.545 INFO:tasks.rados.rados.0.smithi155.stdout:134: done (5 left) 2024-04-19T01:04:19.545 INFO:tasks.rados.rados.0.smithi155.stdout:135: done (4 left) 2024-04-19T01:04:19.545 INFO:tasks.rados.rados.0.smithi155.stdout:136: done (3 left) 2024-04-19T01:04:19.545 INFO:tasks.rados.rados.0.smithi155.stdout:137: done (2 left) 2024-04-19T01:04:19.545 INFO:tasks.rados.rados.0.smithi155.stdout:138: done (1 left) 2024-04-19T01:04:19.545 INFO:tasks.rados.rados.0.smithi155.stdout:141: read oid 26 snap -1 2024-04-19T01:04:19.545 INFO:tasks.rados.rados.0.smithi155.stdout:141: expect (ObjNum 55 snap 5 seq_num 55) 2024-04-19T01:04:19.545 INFO:tasks.rados.rados.0.smithi155.stdout:142: snap_create 2024-04-19T01:04:19.549 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:04:19.549 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-04-19T01:04:19.549 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.549 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s5 49/31 ECSubWrite(tid=124, reqid=osd.13.0:124, at_version=49'75, trim_to=0'0, roll_forward_to=48'73))", 2024-04-19T01:04:19.549 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:19.537588+0000", 2024-04-19T01:04:19.550 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.013594189, 2024-04-19T01:04:19.550 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013618073, 2024-04-19T01:04:19.550 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.550 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.550 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:19.550 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.550 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.550 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.550 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.537588+0000", 2024-04-19T01:04:19.550 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.550 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.550 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.550 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.550 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.537588+0000", 2024-04-19T01:04:19.551 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.551 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.551 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.551 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.551 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.537589+0000", 2024-04-19T01:04:19.551 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5400000000000001e-06 2024-04-19T01:04:19.551 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.551 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.551 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.551 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.537638+0000", 2024-04-19T01:04:19.551 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9067999999999998e-05 2024-04-19T01:04:19.553 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.553 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.553 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.553 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.537666+0000", 2024-04-19T01:04:19.553 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7793e-05 2024-04-19T01:04:19.553 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.553 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.553 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.553 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.537686+0000", 2024-04-19T01:04:19.553 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9983000000000002e-05 2024-04-19T01:04:19.553 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.553 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.554 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.554 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.537845+0000", 2024-04-19T01:04:19.554 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000158925 2024-04-19T01:04:19.554 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.554 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.554 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:19.554 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.537881+0000", 2024-04-19T01:04:19.554 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5695999999999998e-05 2024-04-19T01:04:19.554 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.554 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.554 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:19.554 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.551146+0000", 2024-04-19T01:04:19.554 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013265371999999999 2024-04-19T01:04:19.554 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.555 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.555 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.555 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.555 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.555 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s1 49/31 ECSubWrite(tid=203, reqid=client.4604.0:360, at_version=49'25, trim_to=0'0, roll_forward_to=49'24))", 2024-04-19T01:04:19.555 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:19.539012+0000", 2024-04-19T01:04:19.555 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.012169407, 2024-04-19T01:04:19.555 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012283407999999999, 2024-04-19T01:04:19.555 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.555 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.555 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:19.555 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.555 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.556 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.556 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.539012+0000", 2024-04-19T01:04:19.556 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.556 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.556 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.556 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.556 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.539012+0000", 2024-04-19T01:04:19.556 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.556 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.556 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.556 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.556 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.539014+0000", 2024-04-19T01:04:19.556 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.81e-06 2024-04-19T01:04:19.556 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.557 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.557 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.557 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.539479+0000", 2024-04-19T01:04:19.557 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00046503700000000001 2024-04-19T01:04:19.557 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.557 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.557 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.557 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.539519+0000", 2024-04-19T01:04:19.557 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9965000000000001e-05 2024-04-19T01:04:19.557 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.557 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.557 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.557 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.539540+0000", 2024-04-19T01:04:19.557 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0733e-05 2024-04-19T01:04:19.558 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.558 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.558 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.558 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.539713+0000", 2024-04-19T01:04:19.558 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017261800000000001 2024-04-19T01:04:19.558 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.558 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.558 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:19.558 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.539739+0000", 2024-04-19T01:04:19.558 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6769999999999999e-05 2024-04-19T01:04:19.558 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.558 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.559 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.559 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.559 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.559 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.0s4 49/31 ECSubWrite(tid=39, reqid=osd.1.0:39, at_version=49'35, trim_to=0'0, roll_forward_to=48'33))", 2024-04-19T01:04:19.559 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:19.541020+0000", 2024-04-19T01:04:19.559 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.010161870999999999, 2024-04-19T01:04:19.559 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010342875, 2024-04-19T01:04:19.559 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.559 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.559 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:19.559 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.559 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.559 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.560 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.541020+0000", 2024-04-19T01:04:19.560 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.560 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.560 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.560 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.560 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.541020+0000", 2024-04-19T01:04:19.560 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.560 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.560 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.560 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.560 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.541021+0000", 2024-04-19T01:04:19.560 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5000000000000001e-07 2024-04-19T01:04:19.560 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.561 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.561 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.561 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.541069+0000", 2024-04-19T01:04:19.561 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8288000000000003e-05 2024-04-19T01:04:19.561 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.561 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.561 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.561 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.541093+0000", 2024-04-19T01:04:19.561 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4263000000000001e-05 2024-04-19T01:04:19.562 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.562 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.562 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.562 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.541111+0000", 2024-04-19T01:04:19.562 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7535999999999999e-05 2024-04-19T01:04:19.562 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.562 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.562 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.562 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.541259+0000", 2024-04-19T01:04:19.562 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014775300000000001 2024-04-19T01:04:19.562 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.562 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.562 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:19.562 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.541285+0000", 2024-04-19T01:04:19.563 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6064999999999999e-05 2024-04-19T01:04:19.563 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.563 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.563 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.563 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.563 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.563 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s5 49/31 ECSubWrite(tid=126, reqid=osd.13.0:126, at_version=49'77, trim_to=0'0, roll_forward_to=48'73))", 2024-04-19T01:04:19.563 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:19.542547+0000", 2024-04-19T01:04:19.563 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.008634915, 2024-04-19T01:04:19.563 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.008897505, 2024-04-19T01:04:19.563 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.563 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.563 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:19.563 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.564 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.564 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.564 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.542547+0000", 2024-04-19T01:04:19.564 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.564 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.564 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.564 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.564 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.542547+0000", 2024-04-19T01:04:19.564 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.564 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.564 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.564 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.564 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.542549+0000", 2024-04-19T01:04:19.564 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8169999999999999e-06 2024-04-19T01:04:19.564 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.565 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.565 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.565 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.542597+0000", 2024-04-19T01:04:19.565 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8677999999999997e-05 2024-04-19T01:04:19.565 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.565 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.565 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.565 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.542629+0000", 2024-04-19T01:04:19.565 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1639e-05 2024-04-19T01:04:19.565 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.565 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.565 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.565 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.542650+0000", 2024-04-19T01:04:19.565 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0766000000000001e-05 2024-04-19T01:04:19.566 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.566 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.566 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.566 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.542833+0000", 2024-04-19T01:04:19.566 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018286800000000001 2024-04-19T01:04:19.566 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.566 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.566 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:19.566 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.542858+0000", 2024-04-19T01:04:19.566 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5656e-05 2024-04-19T01:04:19.566 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.567 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.567 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.567 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.567 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-04-19T01:04:19.567 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 4 2024-04-19T01:04:19.567 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:04:19.568 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-19T01:04:19.572 INFO:tasks.rados.rados.0.smithi155.stdout:140: finishing write tid 1 to smithi15528762-45 2024-04-19T01:04:19.579 INFO:tasks.rados.rados.0.smithi155.stdout:141: expect (ObjNum 55 snap 5 seq_num 55) 2024-04-19T01:04:19.633 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:04:19.631+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:04:19.660 INFO:tasks.rados.rados.0.smithi155.stdout:140: finishing write tid 2 to smithi15528762-45 2024-04-19T01:04:19.661 INFO:tasks.rados.rados.0.smithi155.stdout:140: finishing write tid 3 to smithi15528762-45 2024-04-19T01:04:19.661 INFO:tasks.rados.rados.0.smithi155.stdout:140: finishing write tid 4 to smithi15528762-45 2024-04-19T01:04:19.661 INFO:tasks.rados.rados.0.smithi155.stdout:140: finishing write tid 5 to smithi15528762-45 2024-04-19T01:04:19.661 INFO:tasks.rados.rados.0.smithi155.stdout:140: oid 45 updating version 0 to 27 2024-04-19T01:04:19.661 INFO:tasks.rados.rados.0.smithi155.stdout:140: oid 45 version 27 is already newer than 25 2024-04-19T01:04:19.661 INFO:tasks.rados.rados.0.smithi155.stdout:140: oid 45 updating version 27 to 28 2024-04-19T01:04:19.661 INFO:tasks.rados.rados.0.smithi155.stdout:140: oid 45 version 28 is already newer than 26 2024-04-19T01:04:19.661 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 45 v 28 (ObjNum 79 snap 8 seq_num 79) dirty exists 2024-04-19T01:04:19.661 INFO:tasks.rados.rados.0.smithi155.stdout:140: left oid 45 (ObjNum 79 snap 8 seq_num 79) 2024-04-19T01:04:19.689 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:04:19.689 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-04-19T01:04:19.689 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-04-19T01:04:19.689 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-04-19T01:04:19.689 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:04:19.705 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-19T01:04:19.750 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:04:19.752+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:04:19.816 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:04:19.815+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:04:19.832 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:04:19.832 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-04-19T01:04:19.832 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-04-19T01:04:19.832 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-04-19T01:04:19.833 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.833 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s5 39/31 ECSubWrite(tid=32, reqid=client.4604.0:181, at_version=39'16, trim_to=0'0, roll_forward_to=38'14))", 2024-04-19T01:04:19.833 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:09.821218+0000", 2024-04-19T01:04:19.833 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.012059644000001, 2024-04-19T01:04:19.833 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024070148999999999, 2024-04-19T01:04:19.833 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.833 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.833 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:19.833 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.833 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.833 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.834 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.821218+0000", 2024-04-19T01:04:19.834 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.834 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.834 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.834 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.821218+0000", 2024-04-19T01:04:19.834 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.834 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.834 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.834 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.821218+0000", 2024-04-19T01:04:19.835 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6699999999999999e-07 2024-04-19T01:04:19.835 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.835 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.835 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.835 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.821605+0000", 2024-04-19T01:04:19.835 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00038695599999999999 2024-04-19T01:04:19.835 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.835 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.835 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.836 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.821641+0000", 2024-04-19T01:04:19.836 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5998000000000003e-05 2024-04-19T01:04:19.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.836 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.836 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.821671+0000", 2024-04-19T01:04:19.836 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9859e-05 2024-04-19T01:04:19.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.837 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.837 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.821894+0000", 2024-04-19T01:04:19.837 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000223365 2024-04-19T01:04:19.837 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.838 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.838 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:19.838 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.821923+0000", 2024-04-19T01:04:19.838 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8198999999999999e-05 2024-04-19T01:04:19.838 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.838 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.838 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:19.838 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.845179+0000", 2024-04-19T01:04:19.838 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023255983000000001 2024-04-19T01:04:19.838 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.838 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.838 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:19.839 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.845288+0000", 2024-04-19T01:04:19.839 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000109422 2024-04-19T01:04:19.839 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.839 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.839 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.839 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.839 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.839 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s5 39/31 ECSubWrite(tid=33, reqid=client.4604.0:182, at_version=39'17, trim_to=0'0, roll_forward_to=38'14))", 2024-04-19T01:04:19.839 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:09.828409+0000", 2024-04-19T01:04:19.839 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.004868991, 2024-04-19T01:04:19.839 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017438192000000002, 2024-04-19T01:04:19.840 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.840 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.840 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:19.840 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.840 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.841 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.841 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.828409+0000", 2024-04-19T01:04:19.842 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.842 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.842 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.842 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.842 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.828409+0000", 2024-04-19T01:04:19.842 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.842 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.842 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.842 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.842 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.828410+0000", 2024-04-19T01:04:19.843 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.593e-06 2024-04-19T01:04:19.843 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.843 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.843 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.843 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.828583+0000", 2024-04-19T01:04:19.843 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017271699999999999 2024-04-19T01:04:19.843 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.843 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.843 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.844 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.828612+0000", 2024-04-19T01:04:19.844 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9505e-05 2024-04-19T01:04:19.844 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.844 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.844 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.845 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.828630+0000", 2024-04-19T01:04:19.845 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7856999999999999e-05 2024-04-19T01:04:19.845 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.845 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.845 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.845 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.838007+0000", 2024-04-19T01:04:19.845 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0093766490000000008 2024-04-19T01:04:19.845 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.845 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.845 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:19.845 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.838022+0000", 2024-04-19T01:04:19.845 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4579999999999999e-05 2024-04-19T01:04:19.845 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.845 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.845 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:19.846 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.845742+0000", 2024-04-19T01:04:19.846 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0077209649999999998 2024-04-19T01:04:19.846 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.846 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.846 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:19.846 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.845847+0000", 2024-04-19T01:04:19.846 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000104326 2024-04-19T01:04:19.846 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.846 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.847 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.847 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.847 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.847 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s5 43/31 ECSubWrite(tid=70, reqid=client.4604.0:235, at_version=43'43, trim_to=0'0, roll_forward_to=41'37))", 2024-04-19T01:04:19.847 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.873887+0000", 2024-04-19T01:04:19.847 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.9593909639999998, 2024-04-19T01:04:19.847 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.026150459000000001, 2024-04-19T01:04:19.847 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.847 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.847 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:19.847 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.847 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.847 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.847 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.873887+0000", 2024-04-19T01:04:19.848 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.848 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.848 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.848 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.848 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.873887+0000", 2024-04-19T01:04:19.848 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.848 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.848 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.848 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.848 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.873889+0000", 2024-04-19T01:04:19.848 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3099999999999999e-06 2024-04-19T01:04:19.848 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.848 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.848 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.849 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.874386+0000", 2024-04-19T01:04:19.849 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00049737900000000005 2024-04-19T01:04:19.849 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.849 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.849 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.849 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.874427+0000", 2024-04-19T01:04:19.849 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0349000000000003e-05 2024-04-19T01:04:19.849 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.849 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.849 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.849 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.874452+0000", 2024-04-19T01:04:19.849 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4888999999999998e-05 2024-04-19T01:04:19.849 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.849 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.849 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.850 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.874622+0000", 2024-04-19T01:04:19.850 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017063399999999999 2024-04-19T01:04:19.850 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.850 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.850 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:19.850 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.874652+0000", 2024-04-19T01:04:19.850 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9329e-05 2024-04-19T01:04:19.850 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.850 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.850 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:19.850 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.899980+0000", 2024-04-19T01:04:19.850 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025328053999999999 2024-04-19T01:04:19.850 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.850 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.851 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:19.851 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.900037+0000", 2024-04-19T01:04:19.851 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7515000000000001e-05 2024-04-19T01:04:19.851 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.851 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.851 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.851 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.851 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.851 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 43/31 ECSubWrite(tid=43, reqid=client.4604.0:251, at_version=43'29, trim_to=0'0, roll_forward_to=43'27))", 2024-04-19T01:04:19.851 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.985908+0000", 2024-04-19T01:04:19.851 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.8473700070000003, 2024-04-19T01:04:19.851 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023748525999999999, 2024-04-19T01:04:19.851 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.851 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.852 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:19.852 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.852 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.852 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.852 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.985908+0000", 2024-04-19T01:04:19.852 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.852 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.852 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.852 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.852 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.985908+0000", 2024-04-19T01:04:19.852 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.852 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.852 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.853 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.853 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.985909+0000", 2024-04-19T01:04:19.853 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.823e-06 2024-04-19T01:04:19.853 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.853 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.853 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.853 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986673+0000", 2024-04-19T01:04:19.853 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00076342100000000002 2024-04-19T01:04:19.853 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.853 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.853 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.853 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986704+0000", 2024-04-19T01:04:19.853 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1034999999999997e-05 2024-04-19T01:04:19.854 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.854 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.854 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.854 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986720+0000", 2024-04-19T01:04:19.854 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6093e-05 2024-04-19T01:04:19.854 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.854 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.854 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.854 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986887+0000", 2024-04-19T01:04:19.854 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000167524 2024-04-19T01:04:19.854 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.854 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.854 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:19.854 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986913+0000", 2024-04-19T01:04:19.855 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5326000000000001e-05 2024-04-19T01:04:19.855 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.855 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.855 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:19.855 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.009579+0000", 2024-04-19T01:04:19.855 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022666657 2024-04-19T01:04:19.855 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.855 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.855 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:19.855 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.009656+0000", 2024-04-19T01:04:19.855 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6647e-05 2024-04-19T01:04:19.855 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.855 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.855 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.856 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.856 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.856 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:276 3.4s0 3:25634c61:::smithi15528762-21:head [read 0~196608,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:19.856 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.015933+0000", 2024-04-19T01:04:19.856 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.8173445480000003, 2024-04-19T01:04:19.856 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019236120999999998, 2024-04-19T01:04:19.856 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.856 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.856 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-04-19T01:04:19.857 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:04:19.857 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:04:19.857 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:19.857 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 276 2024-04-19T01:04:19.857 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.857 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.857 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.857 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.857 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.015933+0000", 2024-04-19T01:04:19.857 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.857 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.857 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.857 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.858 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.015933+0000", 2024-04-19T01:04:19.858 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.858 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.858 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.858 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.858 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.015936+0000", 2024-04-19T01:04:19.858 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.65e-06 2024-04-19T01:04:19.858 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.858 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.858 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.858 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.015946+0000", 2024-04-19T01:04:19.858 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0061000000000001e-05 2024-04-19T01:04:19.858 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.858 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.859 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.859 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.015971+0000", 2024-04-19T01:04:19.859 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5704999999999999e-05 2024-04-19T01:04:19.859 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.859 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.859 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.859 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.015984+0000", 2024-04-19T01:04:19.859 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2046e-05 2024-04-19T01:04:19.859 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.859 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.859 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.859 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.016097+0000", 2024-04-19T01:04:19.859 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011383999999999999 2024-04-19T01:04:19.859 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.859 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.860 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:04:19.860 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.016224+0000", 2024-04-19T01:04:19.860 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012657 2024-04-19T01:04:19.860 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.860 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.860 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:19.860 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.035169+0000", 2024-04-19T01:04:19.860 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018945249000000001 2024-04-19T01:04:19.860 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.860 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.860 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.860 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.860 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.860 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:277 3.4s0 3:25634c61:::smithi15528762-21:head [read 0~196608,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:19.860 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.016014+0000", 2024-04-19T01:04:19.861 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.8172634189999997, 2024-04-19T01:04:19.861 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020020381, 2024-04-19T01:04:19.861 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.861 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.861 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-04-19T01:04:19.861 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:04:19.861 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:04:19.861 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:19.861 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 277 2024-04-19T01:04:19.861 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.861 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.861 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.861 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.861 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.016014+0000", 2024-04-19T01:04:19.862 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.862 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.862 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.862 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.862 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.016014+0000", 2024-04-19T01:04:19.862 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.862 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.862 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.862 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.862 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.016015+0000", 2024-04-19T01:04:19.862 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1999999999999998e-07 2024-04-19T01:04:19.862 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.862 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.863 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.863 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.016021+0000", 2024-04-19T01:04:19.863 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3600000000000001e-06 2024-04-19T01:04:19.863 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.863 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.863 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.863 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.016035+0000", 2024-04-19T01:04:19.863 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3947e-05 2024-04-19T01:04:19.863 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.863 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.863 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.863 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.016046+0000", 2024-04-19T01:04:19.863 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1029999999999999e-05 2024-04-19T01:04:19.863 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.863 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.864 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.864 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.016794+0000", 2024-04-19T01:04:19.864 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00074748000000000002 2024-04-19T01:04:19.864 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.864 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.864 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:04:19.864 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.016941+0000", 2024-04-19T01:04:19.864 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000147369 2024-04-19T01:04:19.864 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.864 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.864 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:19.864 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.036035+0000", 2024-04-19T01:04:19.864 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019093275 2024-04-19T01:04:19.864 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.864 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.865 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.865 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.865 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.865 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:278 3.4s0 3:25634c61:::smithi15528762-21:head [read 0~196608,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:19.865 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.016061+0000", 2024-04-19T01:04:19.865 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.8172161520000003, 2024-04-19T01:04:19.865 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020558379000000002, 2024-04-19T01:04:19.865 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.865 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.865 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-04-19T01:04:19.865 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:04:19.865 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:04:19.866 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:19.866 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 278 2024-04-19T01:04:19.866 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.866 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.866 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.866 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.866 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.016061+0000", 2024-04-19T01:04:19.867 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.867 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.867 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.867 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.867 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.016061+0000", 2024-04-19T01:04:19.867 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.867 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.867 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.867 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.868 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.016062+0000", 2024-04-19T01:04:19.868 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9800000000000002e-07 2024-04-19T01:04:19.868 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.868 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.868 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.868 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.016067+0000", 2024-04-19T01:04:19.868 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0050000000000004e-06 2024-04-19T01:04:19.868 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.868 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.868 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.868 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.016079+0000", 2024-04-19T01:04:19.868 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1966e-05 2024-04-19T01:04:19.868 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.869 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.869 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.869 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.016089+0000", 2024-04-19T01:04:19.869 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.011e-05 2024-04-19T01:04:19.869 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.869 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.869 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.869 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.017458+0000", 2024-04-19T01:04:19.869 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001369035 2024-04-19T01:04:19.869 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.869 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.869 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:04:19.869 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.017608+0000", 2024-04-19T01:04:19.869 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000149133 2024-04-19T01:04:19.870 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.870 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.870 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:19.870 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.036620+0000", 2024-04-19T01:04:19.870 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019012232 2024-04-19T01:04:19.870 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.870 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.870 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.870 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.870 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.870 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s5 43/31 ECSubWrite(tid=76, reqid=client.4604.0:267, at_version=43'46, trim_to=0'0, roll_forward_to=43'45))", 2024-04-19T01:04:19.870 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.034778+0000", 2024-04-19T01:04:19.870 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.7984993029999998, 2024-04-19T01:04:19.871 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.021558997, 2024-04-19T01:04:19.871 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.871 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.871 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:19.871 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.871 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.871 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.871 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.034778+0000", 2024-04-19T01:04:19.871 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.871 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.871 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.871 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.871 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.034778+0000", 2024-04-19T01:04:19.871 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.871 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.872 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.872 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.872 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.034779+0000", 2024-04-19T01:04:19.872 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8400000000000005e-07 2024-04-19T01:04:19.872 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.872 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.872 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.872 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.034950+0000", 2024-04-19T01:04:19.872 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017166300000000001 2024-04-19T01:04:19.872 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.872 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.872 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.873 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.034975+0000", 2024-04-19T01:04:19.873 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4861000000000001e-05 2024-04-19T01:04:19.873 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.873 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.873 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.873 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.034987+0000", 2024-04-19T01:04:19.873 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2e-05 2024-04-19T01:04:19.873 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.873 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.873 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.873 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.035082+0000", 2024-04-19T01:04:19.873 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4611000000000006e-05 2024-04-19T01:04:19.873 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.873 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.874 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:19.874 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.035097+0000", 2024-04-19T01:04:19.874 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5529999999999999e-05 2024-04-19T01:04:19.874 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.874 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.874 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:19.874 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.056282+0000", 2024-04-19T01:04:19.874 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.021184222999999999 2024-04-19T01:04:19.874 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.874 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.874 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:19.874 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.056337+0000", 2024-04-19T01:04:19.874 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5624999999999997e-05 2024-04-19T01:04:19.874 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.875 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.875 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.875 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.875 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.875 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s5 43/31 ECSubWrite(tid=77, reqid=client.4604.0:268, at_version=43'47, trim_to=0'0, roll_forward_to=43'45))", 2024-04-19T01:04:19.875 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.040576+0000", 2024-04-19T01:04:19.875 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.7927018370000001, 2024-04-19T01:04:19.875 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016059917, 2024-04-19T01:04:19.875 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.876 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.876 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:19.876 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.876 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.876 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.876 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.040576+0000", 2024-04-19T01:04:19.876 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.876 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.876 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.876 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.876 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.040576+0000", 2024-04-19T01:04:19.876 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.876 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.876 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.877 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.877 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.040577+0000", 2024-04-19T01:04:19.877 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2860000000000001e-06 2024-04-19T01:04:19.877 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.877 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.877 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.877 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.040724+0000", 2024-04-19T01:04:19.877 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014670899999999999 2024-04-19T01:04:19.877 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.877 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.878 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.878 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.040751+0000", 2024-04-19T01:04:19.878 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7090999999999999e-05 2024-04-19T01:04:19.878 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.878 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.878 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.878 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.040767+0000", 2024-04-19T01:04:19.878 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5863999999999999e-05 2024-04-19T01:04:19.878 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.878 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.878 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.878 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.052097+0000", 2024-04-19T01:04:19.878 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011330033 2024-04-19T01:04:19.878 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.879 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.879 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:19.879 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.052113+0000", 2024-04-19T01:04:19.879 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6331999999999999e-05 2024-04-19T01:04:19.879 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.879 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.879 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:19.879 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.056576+0000", 2024-04-19T01:04:19.879 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0044627080000000001 2024-04-19T01:04:19.879 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.879 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.879 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:19.879 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.056636+0000", 2024-04-19T01:04:19.879 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9893999999999999e-05 2024-04-19T01:04:19.879 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.880 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.880 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.880 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.880 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.880 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 44/31 ECSubWrite(tid=49, reqid=osd.8.0:49, at_version=44'33, trim_to=0'0, roll_forward_to=43'29))", 2024-04-19T01:04:19.880 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.876298+0000", 2024-04-19T01:04:19.880 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.956979102, 2024-04-19T01:04:19.880 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015718606, 2024-04-19T01:04:19.880 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.881 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.881 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:19.881 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.881 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.881 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.881 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876298+0000", 2024-04-19T01:04:19.881 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.881 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.881 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.881 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.881 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876298+0000", 2024-04-19T01:04:19.881 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.881 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.882 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.882 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.882 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876301+0000", 2024-04-19T01:04:19.882 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1100000000000001e-06 2024-04-19T01:04:19.882 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.882 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.882 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.882 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876353+0000", 2024-04-19T01:04:19.882 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2565000000000002e-05 2024-04-19T01:04:19.882 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.882 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.882 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.882 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876384+0000", 2024-04-19T01:04:19.882 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1272999999999997e-05 2024-04-19T01:04:19.883 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.883 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.883 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.883 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876406+0000", 2024-04-19T01:04:19.883 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1942000000000001e-05 2024-04-19T01:04:19.883 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.883 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.883 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.883 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.879099+0000", 2024-04-19T01:04:19.883 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002692739 2024-04-19T01:04:19.883 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.883 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.883 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:19.883 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.879118+0000", 2024-04-19T01:04:19.884 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9355e-05 2024-04-19T01:04:19.884 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.884 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.884 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:19.884 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.891924+0000", 2024-04-19T01:04:19.884 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012805548 2024-04-19T01:04:19.884 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.884 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.884 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:19.884 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.892017+0000", 2024-04-19T01:04:19.884 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3073999999999997e-05 2024-04-19T01:04:19.884 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.884 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.884 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.885 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.885 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.885 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s5 44/31 ECSubWrite(tid=82, reqid=client.4604.0:280, at_version=44'50, trim_to=0'0, roll_forward_to=43'48))", 2024-04-19T01:04:19.885 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.886693+0000", 2024-04-19T01:04:19.885 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.9465848690000005, 2024-04-19T01:04:19.885 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.044983205999999998, 2024-04-19T01:04:19.885 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.885 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.885 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:19.885 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.885 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.885 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.885 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.886693+0000", 2024-04-19T01:04:19.885 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.885 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.886 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.886 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.886 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.886693+0000", 2024-04-19T01:04:19.886 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.886 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.886 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.886 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.886 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.886694+0000", 2024-04-19T01:04:19.886 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2699999999999998e-07 2024-04-19T01:04:19.886 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.886 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.886 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.886 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.887101+0000", 2024-04-19T01:04:19.886 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00040740299999999998 2024-04-19T01:04:19.887 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.887 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.887 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.887 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.887130+0000", 2024-04-19T01:04:19.887 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9479e-05 2024-04-19T01:04:19.887 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.887 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.887 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.887 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.887152+0000", 2024-04-19T01:04:19.887 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1708999999999998e-05 2024-04-19T01:04:19.888 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.888 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.888 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.888 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.887281+0000", 2024-04-19T01:04:19.888 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012845599999999999 2024-04-19T01:04:19.888 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.888 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.888 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:19.888 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.887296+0000", 2024-04-19T01:04:19.888 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5642999999999999e-05 2024-04-19T01:04:19.888 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.888 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.888 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:19.888 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.931601+0000", 2024-04-19T01:04:19.889 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.044304705 2024-04-19T01:04:19.889 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.889 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.889 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:19.889 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.931676+0000", 2024-04-19T01:04:19.889 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4983999999999999e-05 2024-04-19T01:04:19.889 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.889 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.889 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.889 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.889 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.889 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(osd.1.22:1 3.bs0 3:da0c9485:::smithi15528762-40:head [assert-version v10,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-19T01:04:19.889 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.891568+0000", 2024-04-19T01:04:19.889 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.9417092079999998, 2024-04-19T01:04:19.890 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019029371, 2024-04-19T01:04:19.890 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.890 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.890 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-04-19T01:04:19.890 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.890 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.890 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.890 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.891568+0000", 2024-04-19T01:04:19.890 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.890 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.890 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.890 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.890 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.891568+0000", 2024-04-19T01:04:19.890 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.891 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.891 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.891 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.891 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.891572+0000", 2024-04-19T01:04:19.891 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.596e-06 2024-04-19T01:04:19.891 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.891 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.891 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.891 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.891587+0000", 2024-04-19T01:04:19.891 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5130000000000001e-05 2024-04-19T01:04:19.891 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.891 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.891 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.891 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.891620+0000", 2024-04-19T01:04:19.891 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3241999999999998e-05 2024-04-19T01:04:19.892 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.892 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.892 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.892 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.891641+0000", 2024-04-19T01:04:19.892 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0639e-05 2024-04-19T01:04:19.892 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.892 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.892 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.892 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.891804+0000", 2024-04-19T01:04:19.892 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000163182 2024-04-19T01:04:19.892 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.892 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.892 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-04-19T01:04:19.892 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.892069+0000", 2024-04-19T01:04:19.892 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026467100000000001 2024-04-19T01:04:19.893 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.893 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.893 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.893 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.903406+0000", 2024-04-19T01:04:19.893 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011336946000000001 2024-04-19T01:04:19.893 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.893 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.893 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:04:19.893 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.903666+0000", 2024-04-19T01:04:19.893 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026044099999999998 2024-04-19T01:04:19.893 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.893 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.893 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:19.893 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.910598+0000", 2024-04-19T01:04:19.893 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0069315239999999997 2024-04-19T01:04:19.894 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.894 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.894 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.894 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.894 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.894 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s1 44/31 ECSubWrite(tid=189, reqid=osd.11.0:186, at_version=0'0, trim_to=0'0, roll_forward_to=44'12))", 2024-04-19T01:04:19.894 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.899026+0000", 2024-04-19T01:04:19.894 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.9342517919999995, 2024-04-19T01:04:19.894 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013877909000000001, 2024-04-19T01:04:19.894 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.894 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.894 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:19.895 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.895 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.895 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.895 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.899026+0000", 2024-04-19T01:04:19.895 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.895 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.895 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.895 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.895 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.899026+0000", 2024-04-19T01:04:19.895 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.895 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.895 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.895 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.895 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.899028+0000", 2024-04-19T01:04:19.895 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.877e-06 2024-04-19T01:04:19.896 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.896 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.896 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.896 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.899057+0000", 2024-04-19T01:04:19.896 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9575000000000001e-05 2024-04-19T01:04:19.896 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.896 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.896 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.896 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.899100+0000", 2024-04-19T01:04:19.896 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2864999999999997e-05 2024-04-19T01:04:19.896 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.896 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.896 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.897 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.899123+0000", 2024-04-19T01:04:19.897 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2623e-05 2024-04-19T01:04:19.897 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.897 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.897 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.897 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.899345+0000", 2024-04-19T01:04:19.897 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000222659 2024-04-19T01:04:19.897 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.897 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.897 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:19.897 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.899370+0000", 2024-04-19T01:04:19.897 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4715999999999998e-05 2024-04-19T01:04:19.897 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.897 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.898 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:19.898 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.912815+0000", 2024-04-19T01:04:19.898 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013444881000000001 2024-04-19T01:04:19.898 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.898 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.898 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:19.898 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.912904+0000", 2024-04-19T01:04:19.898 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8713000000000003e-05 2024-04-19T01:04:19.898 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.899 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.899 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.899 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.899 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.899 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s5 44/31 ECSubWrite(tid=84, reqid=client.4604.0:281, at_version=44'51, trim_to=0'0, roll_forward_to=43'48))", 2024-04-19T01:04:19.899 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.908596+0000", 2024-04-19T01:04:19.899 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.9246813720000002, 2024-04-19T01:04:19.900 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024500261999999998, 2024-04-19T01:04:19.900 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.900 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.900 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:19.900 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.900 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.900 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.900 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.908596+0000", 2024-04-19T01:04:19.900 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.900 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.900 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.900 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.900 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.908596+0000", 2024-04-19T01:04:19.900 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.901 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.901 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.901 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.901 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.908598+0000", 2024-04-19T01:04:19.901 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.327e-06 2024-04-19T01:04:19.901 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.901 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.901 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.901 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.908853+0000", 2024-04-19T01:04:19.901 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00025536100000000001 2024-04-19T01:04:19.901 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.901 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.901 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.902 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.908881+0000", 2024-04-19T01:04:19.902 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7889999999999999e-05 2024-04-19T01:04:19.902 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.902 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.902 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.902 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.908901+0000", 2024-04-19T01:04:19.902 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0112e-05 2024-04-19T01:04:19.902 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.902 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.902 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.902 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.921159+0000", 2024-04-19T01:04:19.902 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012258349 2024-04-19T01:04:19.902 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.902 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.902 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:19.903 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.921189+0000", 2024-04-19T01:04:19.903 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9743e-05 2024-04-19T01:04:19.903 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.903 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.903 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:19.903 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.933036+0000", 2024-04-19T01:04:19.903 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011847068000000001 2024-04-19T01:04:19.903 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.903 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.903 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:19.903 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.933096+0000", 2024-04-19T01:04:19.903 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0411999999999997e-05 2024-04-19T01:04:19.903 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.904 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.904 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.904 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.904 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.904 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s5 44/31 ECSubWrite(tid=85, reqid=client.4604.0:282, at_version=44'52, trim_to=0'0, roll_forward_to=43'48))", 2024-04-19T01:04:19.904 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.917333+0000", 2024-04-19T01:04:19.904 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.915944874, 2024-04-19T01:04:19.904 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015816497999999998, 2024-04-19T01:04:19.904 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.904 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.904 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:19.904 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.904 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.904 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.904 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.917333+0000", 2024-04-19T01:04:19.905 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.905 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.905 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.905 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.905 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.917333+0000", 2024-04-19T01:04:19.905 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.905 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.905 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.905 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.905 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.917334+0000", 2024-04-19T01:04:19.905 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.716e-06 2024-04-19T01:04:19.905 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.905 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.905 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.906 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.917370+0000", 2024-04-19T01:04:19.906 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5895000000000001e-05 2024-04-19T01:04:19.906 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.906 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.906 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.906 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.917409+0000", 2024-04-19T01:04:19.906 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8390999999999999e-05 2024-04-19T01:04:19.906 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.906 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.906 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.906 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.917428+0000", 2024-04-19T01:04:19.906 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9177e-05 2024-04-19T01:04:19.906 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.906 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.906 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.907 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.931030+0000", 2024-04-19T01:04:19.907 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013602553 2024-04-19T01:04:19.907 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.907 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.907 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:19.907 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.931048+0000", 2024-04-19T01:04:19.907 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7081000000000001e-05 2024-04-19T01:04:19.907 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.907 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.907 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:19.907 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.933103+0000", 2024-04-19T01:04:19.907 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002055795 2024-04-19T01:04:19.907 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.907 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.908 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:19.908 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.933149+0000", 2024-04-19T01:04:19.908 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5890000000000003e-05 2024-04-19T01:04:19.908 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.908 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.908 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.908 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.908 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.908 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.0s4 44/31 ECSubWrite(tid=25, reqid=client.4604.0:284, at_version=44'22, trim_to=0'0, roll_forward_to=44'20))", 2024-04-19T01:04:19.909 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.921604+0000", 2024-04-19T01:04:19.909 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.9116735409999999, 2024-04-19T01:04:19.909 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023370134000000001, 2024-04-19T01:04:19.909 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.909 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.909 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:19.909 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.909 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.909 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.909 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.921604+0000", 2024-04-19T01:04:19.909 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.909 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.909 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.909 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.910 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.921604+0000", 2024-04-19T01:04:19.910 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.910 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.910 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.910 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.910 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.921606+0000", 2024-04-19T01:04:19.910 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.508e-06 2024-04-19T01:04:19.910 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.910 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.910 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.910 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.922858+0000", 2024-04-19T01:04:19.910 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0012527829999999999 2024-04-19T01:04:19.910 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.910 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.911 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.911 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.922889+0000", 2024-04-19T01:04:19.911 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0822999999999999e-05 2024-04-19T01:04:19.911 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.911 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.911 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.911 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.922907+0000", 2024-04-19T01:04:19.911 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8065999999999999e-05 2024-04-19T01:04:19.911 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.911 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.911 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.911 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.923065+0000", 2024-04-19T01:04:19.911 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015793799999999999 2024-04-19T01:04:19.911 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.912 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.912 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:19.912 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.923086+0000", 2024-04-19T01:04:19.912 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0874e-05 2024-04-19T01:04:19.912 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.912 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.912 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:19.912 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.944897+0000", 2024-04-19T01:04:19.912 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.021811185 2024-04-19T01:04:19.912 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.912 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.912 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:19.912 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.944974+0000", 2024-04-19T01:04:19.912 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6957000000000002e-05 2024-04-19T01:04:19.913 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.913 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.913 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.913 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.913 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.913 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s5 49/31 ECSubWrite(tid=124, reqid=osd.13.0:124, at_version=49'75, trim_to=0'0, roll_forward_to=48'73))", 2024-04-19T01:04:19.913 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:19.537588+0000", 2024-04-19T01:04:19.913 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.29568989400000001, 2024-04-19T01:04:19.913 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013656010999999999, 2024-04-19T01:04:19.913 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.913 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.913 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:19.913 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.913 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.913 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.914 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.537588+0000", 2024-04-19T01:04:19.914 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.914 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.914 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.914 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.914 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.537588+0000", 2024-04-19T01:04:19.914 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.914 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.914 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.914 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.914 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.537589+0000", 2024-04-19T01:04:19.914 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5400000000000001e-06 2024-04-19T01:04:19.914 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.914 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.915 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.915 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.537638+0000", 2024-04-19T01:04:19.915 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9067999999999998e-05 2024-04-19T01:04:19.915 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.915 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.915 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.915 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.537666+0000", 2024-04-19T01:04:19.915 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7793e-05 2024-04-19T01:04:19.915 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.915 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.915 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.915 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.537686+0000", 2024-04-19T01:04:19.915 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9983000000000002e-05 2024-04-19T01:04:19.915 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.915 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.916 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.916 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.537845+0000", 2024-04-19T01:04:19.916 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000158925 2024-04-19T01:04:19.916 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.916 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.916 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:19.916 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.537881+0000", 2024-04-19T01:04:19.916 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5695999999999998e-05 2024-04-19T01:04:19.916 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.916 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.916 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:19.916 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.551146+0000", 2024-04-19T01:04:19.916 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013265371999999999 2024-04-19T01:04:19.918 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.918 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.918 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:19.918 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.551244+0000", 2024-04-19T01:04:19.918 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7633999999999994e-05 2024-04-19T01:04:19.918 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.918 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.918 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.919 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.919 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.919 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s1 49/31 ECSubWrite(tid=203, reqid=client.4604.0:360, at_version=49'25, trim_to=0'0, roll_forward_to=49'24))", 2024-04-19T01:04:19.919 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:19.539012+0000", 2024-04-19T01:04:19.919 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.294265112, 2024-04-19T01:04:19.919 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017635093000000001, 2024-04-19T01:04:19.919 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.919 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.919 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:19.919 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.920 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.920 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.920 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.539012+0000", 2024-04-19T01:04:19.920 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.920 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.920 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.920 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.920 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.539012+0000", 2024-04-19T01:04:19.920 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.920 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.920 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.920 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.920 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.539014+0000", 2024-04-19T01:04:19.920 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.81e-06 2024-04-19T01:04:19.921 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.921 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.921 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.921 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.539479+0000", 2024-04-19T01:04:19.921 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00046503700000000001 2024-04-19T01:04:19.921 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.921 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.921 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.921 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.539519+0000", 2024-04-19T01:04:19.921 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9965000000000001e-05 2024-04-19T01:04:19.921 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.921 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.921 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.921 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.539540+0000", 2024-04-19T01:04:19.922 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0733e-05 2024-04-19T01:04:19.922 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.922 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.922 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.922 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.539713+0000", 2024-04-19T01:04:19.922 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017261800000000001 2024-04-19T01:04:19.922 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.922 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.922 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:19.922 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.539739+0000", 2024-04-19T01:04:19.923 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6769999999999999e-05 2024-04-19T01:04:19.923 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.923 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.923 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:19.923 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.556607+0000", 2024-04-19T01:04:19.923 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016867608999999999 2024-04-19T01:04:19.923 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.923 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.923 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:19.923 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.556648+0000", 2024-04-19T01:04:19.923 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0550999999999999e-05 2024-04-19T01:04:19.923 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.923 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.923 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.924 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.924 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.924 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.0s4 49/31 ECSubWrite(tid=39, reqid=osd.1.0:39, at_version=49'35, trim_to=0'0, roll_forward_to=48'33))", 2024-04-19T01:04:19.924 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:19.541020+0000", 2024-04-19T01:04:19.924 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.29225757600000002, 2024-04-19T01:04:19.924 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014472379, 2024-04-19T01:04:19.924 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.924 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.924 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:19.924 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.925 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.925 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.925 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.541020+0000", 2024-04-19T01:04:19.925 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.925 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.925 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.925 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.925 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.541020+0000", 2024-04-19T01:04:19.925 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.925 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.925 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.925 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.925 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.541021+0000", 2024-04-19T01:04:19.925 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5000000000000001e-07 2024-04-19T01:04:19.926 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.926 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.926 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.926 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.541069+0000", 2024-04-19T01:04:19.926 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8288000000000003e-05 2024-04-19T01:04:19.926 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.926 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.926 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.926 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.541093+0000", 2024-04-19T01:04:19.926 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4263000000000001e-05 2024-04-19T01:04:19.926 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.927 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.927 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.927 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.541111+0000", 2024-04-19T01:04:19.927 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7535999999999999e-05 2024-04-19T01:04:19.927 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.927 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.927 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.927 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.541259+0000", 2024-04-19T01:04:19.927 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014775300000000001 2024-04-19T01:04:19.927 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.927 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.927 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:19.927 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.541285+0000", 2024-04-19T01:04:19.927 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6064999999999999e-05 2024-04-19T01:04:19.927 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.928 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.928 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:19.928 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.555383+0000", 2024-04-19T01:04:19.928 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.01409843 2024-04-19T01:04:19.928 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.928 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.928 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:19.928 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.555492+0000", 2024-04-19T01:04:19.928 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000109194 2024-04-19T01:04:19.928 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.928 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.928 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.928 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.928 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.929 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s1 49/31 ECSubWrite(tid=204, reqid=client.4604.0:361, at_version=49'26, trim_to=0'0, roll_forward_to=49'24))", 2024-04-19T01:04:19.929 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:19.561289+0000", 2024-04-19T01:04:19.929 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.27198895000000001, 2024-04-19T01:04:19.929 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013224237999999999, 2024-04-19T01:04:19.929 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:19.929 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:19.929 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:19.929 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:19.929 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.930 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:19.930 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.561289+0000", 2024-04-19T01:04:19.930 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.930 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.930 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.930 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:19.930 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.561289+0000", 2024-04-19T01:04:19.930 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:19.930 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.930 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.930 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:19.930 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.561290+0000", 2024-04-19T01:04:19.930 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1060000000000001e-06 2024-04-19T01:04:19.930 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.931 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.931 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:19.931 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.561430+0000", 2024-04-19T01:04:19.931 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013995899999999999 2024-04-19T01:04:19.931 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.931 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.931 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:19.931 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.561453+0000", 2024-04-19T01:04:19.931 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3502e-05 2024-04-19T01:04:19.931 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.931 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.931 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:19.931 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.561466+0000", 2024-04-19T01:04:19.931 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2347999999999999e-05 2024-04-19T01:04:19.931 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.932 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.932 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:19.932 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.561602+0000", 2024-04-19T01:04:19.932 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013648800000000001 2024-04-19T01:04:19.932 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.932 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.932 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:19.932 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.561630+0000", 2024-04-19T01:04:19.932 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8189999999999999e-05 2024-04-19T01:04:19.932 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.932 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.932 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:19.932 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.574434+0000", 2024-04-19T01:04:19.932 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012803281 2024-04-19T01:04:19.932 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:19.933 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:19.933 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:19.933 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.574513+0000", 2024-04-19T01:04:19.933 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9363999999999994e-05 2024-04-19T01:04:19.933 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.933 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.933 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.933 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:19.933 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:19.934 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:04:19.934 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-19T01:04:19.935 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:19.935+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:19.936 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-04-19T01:04:20.037 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:04:20.037+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:04:20.090 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-04-19T01:04:20.090 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T01:04:20.103 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:04:20.104 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-04-19T01:04:20.104 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-04-19T01:04:20.104 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:04:20.110 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-04-19T01:04:20.111 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T01:04:20.118 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-04-19T01:04:20.130 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-04-19T01:04:20.130 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T01:04:20.137 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:04:20.137+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:04:20.150 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-04-19T01:04:20.151 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T01:04:20.167 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-04-19T01:04:20.168 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T01:04:20.189 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-04-19T01:04:20.190 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T01:04:20.209 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-04-19T01:04:20.210 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T01:04:20.231 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-04-19T01:04:20.231 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T01:04:20.234 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:04:20.234 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-04-19T01:04:20.234 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-04-19T01:04:20.234 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-04-19T01:04:20.234 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:04:20.238 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:04:20.237+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:04:20.248 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-04-19T01:04:20.249 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2024-04-19T01:04:20.249 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T01:04:20.271 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2024-04-19T01:04:20.271 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T01:04:20.290 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2024-04-19T01:04:20.290 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T01:04:20.311 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2024-04-19T01:04:20.312 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T01:04:20.331 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2024-04-19T01:04:20.331 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-19T01:04:20.339 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:04:20.337+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:04:20.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:04:20.351+0000 7f50eed9d640 0 -- 172.21.15.33:0/2629202503 >> [v2:172.21.15.110:6801/1583232449,v1:172.21.15.110:6803/1583232449] conn(0x7f50f021b420 msgr2=0x7f50f021d810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-19T01:04:20.358 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2024-04-19T01:04:20.376 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:04:20.376 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-04-19T01:04:20.376 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-04-19T01:04:20.376 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-04-19T01:04:20.376 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.376 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s2 39/31 ECSubWrite(tid=32, reqid=client.4604.0:181, at_version=39'16, trim_to=0'0, roll_forward_to=38'14))", 2024-04-19T01:04:20.376 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:09.821438+0000", 2024-04-19T01:04:20.376 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.554373031000001, 2024-04-19T01:04:20.376 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020304547999999999, 2024-04-19T01:04:20.376 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:20.377 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:20.377 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:20.377 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:20.377 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.377 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:20.377 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.821438+0000", 2024-04-19T01:04:20.377 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:20.377 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.377 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.377 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:20.377 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.821438+0000", 2024-04-19T01:04:20.377 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:20.377 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.378 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.378 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:20.378 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.821439+0000", 2024-04-19T01:04:20.378 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2699999999999999e-07 2024-04-19T01:04:20.378 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.378 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.378 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:20.378 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.822000+0000", 2024-04-19T01:04:20.378 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00056130600000000002 2024-04-19T01:04:20.378 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.378 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.379 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:20.379 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.822034+0000", 2024-04-19T01:04:20.379 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3785000000000003e-05 2024-04-19T01:04:20.379 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.379 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.379 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:20.379 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.822057+0000", 2024-04-19T01:04:20.379 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3289e-05 2024-04-19T01:04:20.379 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.379 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.379 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:20.379 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.822234+0000", 2024-04-19T01:04:20.380 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017660000000000001 2024-04-19T01:04:20.380 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.380 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.380 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:20.380 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.822249+0000", 2024-04-19T01:04:20.380 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5415999999999999e-05 2024-04-19T01:04:20.380 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.380 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.380 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:20.380 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.841628+0000", 2024-04-19T01:04:20.380 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019379159999999999 2024-04-19T01:04:20.380 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.380 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.381 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:20.381 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.841743+0000", 2024-04-19T01:04:20.381 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000114265 2024-04-19T01:04:20.381 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:20.381 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:20.381 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:20.381 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.381 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.381 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s2 39/31 ECSubWrite(tid=33, reqid=client.4604.0:182, at_version=39'17, trim_to=0'0, roll_forward_to=38'14))", 2024-04-19T01:04:20.381 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:09.829850+0000", 2024-04-19T01:04:20.382 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.545961508, 2024-04-19T01:04:20.382 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012982403999999999, 2024-04-19T01:04:20.382 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:20.384 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:20.384 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:20.384 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:20.384 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.384 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:20.384 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.829850+0000", 2024-04-19T01:04:20.385 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:20.385 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.385 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.385 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:20.385 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.829850+0000", 2024-04-19T01:04:20.385 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:20.385 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.385 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.385 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:20.385 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.829851+0000", 2024-04-19T01:04:20.385 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7269999999999999e-06 2024-04-19T01:04:20.385 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.385 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.386 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:20.386 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.830044+0000", 2024-04-19T01:04:20.386 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000192449 2024-04-19T01:04:20.386 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.386 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.386 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:20.386 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.830072+0000", 2024-04-19T01:04:20.386 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8492000000000001e-05 2024-04-19T01:04:20.386 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.386 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.386 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:20.386 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.830088+0000", 2024-04-19T01:04:20.386 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5423000000000001e-05 2024-04-19T01:04:20.386 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.387 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.387 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:20.387 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.834720+0000", 2024-04-19T01:04:20.387 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0046326320000000002 2024-04-19T01:04:20.387 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.387 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.387 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:20.387 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.834735+0000", 2024-04-19T01:04:20.387 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.468e-05 2024-04-19T01:04:20.387 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.388 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.388 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:20.388 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.842759+0000", 2024-04-19T01:04:20.388 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0080238400000000008 2024-04-19T01:04:20.388 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.388 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.388 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:20.388 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.842832+0000", 2024-04-19T01:04:20.388 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3161e-05 2024-04-19T01:04:20.388 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:20.388 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:20.388 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:20.388 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.389 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.389 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s2 39/31 ECSubWrite(tid=38, reqid=client.4604.0:187, at_version=39'20, trim_to=0'0, roll_forward_to=39'18))", 2024-04-19T01:04:20.389 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:09.890157+0000", 2024-04-19T01:04:20.389 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.485654543000001, 2024-04-19T01:04:20.389 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025377373000000002, 2024-04-19T01:04:20.389 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:20.389 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:20.389 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:20.389 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:20.389 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.389 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:20.389 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.890157+0000", 2024-04-19T01:04:20.389 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:20.389 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.390 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.390 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:20.390 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.890157+0000", 2024-04-19T01:04:20.390 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:20.390 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.390 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.390 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:20.390 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.890159+0000", 2024-04-19T01:04:20.390 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3140000000000002e-06 2024-04-19T01:04:20.390 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.390 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.390 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:20.390 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.890514+0000", 2024-04-19T01:04:20.390 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035502899999999998 2024-04-19T01:04:20.391 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.391 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.391 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:20.391 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.890556+0000", 2024-04-19T01:04:20.391 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1819000000000001e-05 2024-04-19T01:04:20.391 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.391 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.391 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:20.391 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.890581+0000", 2024-04-19T01:04:20.391 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5548999999999999e-05 2024-04-19T01:04:20.391 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.391 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.391 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:20.391 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.890752+0000", 2024-04-19T01:04:20.391 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000170147 2024-04-19T01:04:20.391 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.392 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.392 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:20.392 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.890781+0000", 2024-04-19T01:04:20.392 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9275e-05 2024-04-19T01:04:20.392 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.392 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.392 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:20.392 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.915444+0000", 2024-04-19T01:04:20.392 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024662726999999999 2024-04-19T01:04:20.392 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.392 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.392 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:20.392 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.915534+0000", 2024-04-19T01:04:20.392 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0513000000000006e-05 2024-04-19T01:04:20.393 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:20.393 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:20.393 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:20.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.393 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.393 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s2 39/31 ECSubWrite(tid=39, reqid=client.4604.0:188, at_version=39'21, trim_to=0'0, roll_forward_to=39'18))", 2024-04-19T01:04:20.393 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:09.897182+0000", 2024-04-19T01:04:20.393 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.478628948000001, 2024-04-19T01:04:20.393 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018721683999999999, 2024-04-19T01:04:20.394 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:20.394 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:20.394 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:20.394 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:20.394 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.394 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:20.394 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.897182+0000", 2024-04-19T01:04:20.394 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:20.394 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.394 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.394 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:20.395 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.897182+0000", 2024-04-19T01:04:20.395 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:20.395 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.395 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.395 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:20.395 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.897185+0000", 2024-04-19T01:04:20.395 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4099999999999998e-06 2024-04-19T01:04:20.395 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.396 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.396 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:20.396 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.897464+0000", 2024-04-19T01:04:20.396 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00027915900000000002 2024-04-19T01:04:20.396 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.396 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.396 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:20.396 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.897515+0000", 2024-04-19T01:04:20.396 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0778000000000001e-05 2024-04-19T01:04:20.396 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.396 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.396 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:20.396 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.897536+0000", 2024-04-19T01:04:20.396 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1685999999999999e-05 2024-04-19T01:04:20.397 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.397 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.397 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:20.397 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.905546+0000", 2024-04-19T01:04:20.397 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0080096409999999996 2024-04-19T01:04:20.397 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.397 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.397 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:20.397 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.905571+0000", 2024-04-19T01:04:20.397 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.482e-05 2024-04-19T01:04:20.397 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.397 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.397 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:20.397 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.915777+0000", 2024-04-19T01:04:20.397 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010205881999999999 2024-04-19T01:04:20.398 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.398 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.398 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:20.398 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.915904+0000", 2024-04-19T01:04:20.398 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012730799999999999 2024-04-19T01:04:20.398 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:20.398 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:20.398 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:20.398 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.398 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.398 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s2 39/31 ECSubWrite(tid=40, reqid=client.4604.0:189, at_version=39'22, trim_to=0'0, roll_forward_to=39'18))", 2024-04-19T01:04:20.398 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:09.900556+0000", 2024-04-19T01:04:20.398 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.475255401, 2024-04-19T01:04:20.398 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016123281, 2024-04-19T01:04:20.399 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:20.399 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:20.399 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:20.399 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:20.399 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.399 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:20.399 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.900556+0000", 2024-04-19T01:04:20.399 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:20.399 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.399 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.399 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:20.399 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.900556+0000", 2024-04-19T01:04:20.399 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:20.399 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.399 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.400 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:20.400 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.900558+0000", 2024-04-19T01:04:20.400 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0159999999999998e-06 2024-04-19T01:04:20.400 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.400 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.400 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:20.400 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.900599+0000", 2024-04-19T01:04:20.400 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1424999999999997e-05 2024-04-19T01:04:20.400 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.400 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.400 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:20.400 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.900639+0000", 2024-04-19T01:04:20.400 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9916000000000002e-05 2024-04-19T01:04:20.400 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.400 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.401 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:20.401 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.900662+0000", 2024-04-19T01:04:20.401 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3295e-05 2024-04-19T01:04:20.401 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.401 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.401 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:20.401 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.915152+0000", 2024-04-19T01:04:20.401 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014489837 2024-04-19T01:04:20.401 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.401 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.401 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:20.401 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.915168+0000", 2024-04-19T01:04:20.401 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5233e-05 2024-04-19T01:04:20.401 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.401 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.402 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:20.402 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.916606+0000", 2024-04-19T01:04:20.402 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001437993 2024-04-19T01:04:20.402 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.402 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.402 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:20.402 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.916679+0000", 2024-04-19T01:04:20.402 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3565999999999993e-05 2024-04-19T01:04:20.402 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:20.402 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:20.402 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:20.402 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.402 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.402 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 43/31 ECSubWrite(tid=38, reqid=osd.8.0:38, at_version=43'27, trim_to=0'0, roll_forward_to=42'25))", 2024-04-19T01:04:20.403 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.847035+0000", 2024-04-19T01:04:20.403 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.5287760339999998, 2024-04-19T01:04:20.403 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010311450999999999, 2024-04-19T01:04:20.403 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:20.403 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:20.403 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:20.403 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:20.403 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.403 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:20.403 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.847035+0000", 2024-04-19T01:04:20.403 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:20.403 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.403 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.403 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:20.403 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.847035+0000", 2024-04-19T01:04:20.404 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:20.404 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.404 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.404 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:20.404 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.847036+0000", 2024-04-19T01:04:20.404 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8000000000000005e-07 2024-04-19T01:04:20.404 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.404 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.404 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:20.404 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.847113+0000", 2024-04-19T01:04:20.404 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6667000000000003e-05 2024-04-19T01:04:20.404 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.404 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.404 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:20.405 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.847140+0000", 2024-04-19T01:04:20.405 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7146000000000001e-05 2024-04-19T01:04:20.405 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.405 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.405 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:20.405 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.847162+0000", 2024-04-19T01:04:20.405 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2079e-05 2024-04-19T01:04:20.406 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.406 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.406 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:20.406 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.847314+0000", 2024-04-19T01:04:20.406 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000152175 2024-04-19T01:04:20.406 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.406 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.406 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:20.406 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.847337+0000", 2024-04-19T01:04:20.406 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3275999999999999e-05 2024-04-19T01:04:20.406 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.406 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.406 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:20.406 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.857249+0000", 2024-04-19T01:04:20.406 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0099113880000000001 2024-04-19T01:04:20.407 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.407 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.407 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:20.407 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.857347+0000", 2024-04-19T01:04:20.407 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7940000000000001e-05 2024-04-19T01:04:20.407 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:20.407 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:20.407 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:20.407 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.407 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.407 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s2 43/31 ECSubWrite(tid=70, reqid=client.4604.0:235, at_version=43'43, trim_to=0'0, roll_forward_to=41'37))", 2024-04-19T01:04:20.407 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.874738+0000", 2024-04-19T01:04:20.407 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.5010729759999997, 2024-04-19T01:04:20.407 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024645897999999999, 2024-04-19T01:04:20.407 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:20.408 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:20.408 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:20.408 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:20.408 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.408 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:20.408 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.874738+0000", 2024-04-19T01:04:20.408 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:20.408 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.408 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.408 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:20.408 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.874738+0000", 2024-04-19T01:04:20.408 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:20.408 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.408 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.409 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:20.409 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.874739+0000", 2024-04-19T01:04:20.409 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.203e-06 2024-04-19T01:04:20.409 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.409 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.409 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:20.409 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.875466+0000", 2024-04-19T01:04:20.409 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00072702499999999996 2024-04-19T01:04:20.409 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.409 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.409 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:20.409 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.875489+0000", 2024-04-19T01:04:20.409 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2932e-05 2024-04-19T01:04:20.409 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.410 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.410 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:20.410 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.875509+0000", 2024-04-19T01:04:20.410 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9996999999999998e-05 2024-04-19T01:04:20.410 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.410 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.410 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:20.410 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.875656+0000", 2024-04-19T01:04:20.410 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014699099999999999 2024-04-19T01:04:20.410 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.410 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.410 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:20.410 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.875683+0000", 2024-04-19T01:04:20.410 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6435999999999999e-05 2024-04-19T01:04:20.411 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.411 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.411 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:20.411 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.899339+0000", 2024-04-19T01:04:20.411 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023656074999999999 2024-04-19T01:04:20.411 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.411 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.411 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:20.411 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.899384+0000", 2024-04-19T01:04:20.411 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5238999999999998e-05 2024-04-19T01:04:20.411 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:20.411 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:20.411 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:20.411 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.411 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.412 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s2 43/31 ECSubWrite(tid=72, reqid=client.4604.0:236, at_version=43'44, trim_to=0'0, roll_forward_to=41'37))", 2024-04-19T01:04:20.412 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.890996+0000", 2024-04-19T01:04:20.412 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.4848148339999998, 2024-04-19T01:04:20.412 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0093733779999999999, 2024-04-19T01:04:20.412 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:20.412 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:20.412 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:20.412 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:20.412 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.412 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:20.412 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.890996+0000", 2024-04-19T01:04:20.412 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:20.412 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.412 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.412 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:20.413 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.890996+0000", 2024-04-19T01:04:20.413 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:20.413 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.413 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.413 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:20.413 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.890998+0000", 2024-04-19T01:04:20.413 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8530000000000001e-06 2024-04-19T01:04:20.413 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.413 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.413 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:20.413 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.891336+0000", 2024-04-19T01:04:20.413 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00033820200000000001 2024-04-19T01:04:20.413 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.413 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.414 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:20.414 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.891374+0000", 2024-04-19T01:04:20.414 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7795e-05 2024-04-19T01:04:20.414 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.414 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.414 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:20.414 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.891397+0000", 2024-04-19T01:04:20.414 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2742999999999999e-05 2024-04-19T01:04:20.414 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.414 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.414 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:20.414 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.893872+0000", 2024-04-19T01:04:20.414 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002474979 2024-04-19T01:04:20.414 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.415 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.415 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:20.415 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.893891+0000", 2024-04-19T01:04:20.415 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9465000000000001e-05 2024-04-19T01:04:20.415 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.415 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.415 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:20.415 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.900302+0000", 2024-04-19T01:04:20.415 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.006410692 2024-04-19T01:04:20.415 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.415 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.415 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:20.415 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.900370+0000", 2024-04-19T01:04:20.416 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7649000000000002e-05 2024-04-19T01:04:20.416 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:20.416 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:20.416 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:20.416 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.416 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.416 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 43/31 ECSubWrite(tid=43, reqid=client.4604.0:251, at_version=43'29, trim_to=0'0, roll_forward_to=43'27))", 2024-04-19T01:04:20.416 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.985922+0000", 2024-04-19T01:04:20.417 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.3898894100000003, 2024-04-19T01:04:20.417 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.021650376999999998, 2024-04-19T01:04:20.417 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:20.417 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:20.417 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:20.417 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:20.417 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.417 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:20.417 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.985922+0000", 2024-04-19T01:04:20.417 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:20.417 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.417 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.417 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:20.418 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.985922+0000", 2024-04-19T01:04:20.418 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:20.418 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.418 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.418 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:20.418 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.985924+0000", 2024-04-19T01:04:20.418 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1169999999999998e-06 2024-04-19T01:04:20.418 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.418 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.418 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:20.418 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986891+0000", 2024-04-19T01:04:20.418 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00096720599999999997 2024-04-19T01:04:20.418 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.419 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.419 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:20.419 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986935+0000", 2024-04-19T01:04:20.419 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3776000000000002e-05 2024-04-19T01:04:20.419 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.419 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.419 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:20.419 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986960+0000", 2024-04-19T01:04:20.419 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5418999999999998e-05 2024-04-19T01:04:20.419 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.419 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.419 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:20.419 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987142+0000", 2024-04-19T01:04:20.420 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018212 2024-04-19T01:04:20.420 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.420 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.420 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:20.420 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987174+0000", 2024-04-19T01:04:20.420 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1782000000000002e-05 2024-04-19T01:04:20.420 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.420 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.420 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:20.420 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.007479+0000", 2024-04-19T01:04:20.420 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020304543000000001 2024-04-19T01:04:20.420 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.420 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.420 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:20.421 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.007572+0000", 2024-04-19T01:04:20.421 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3413999999999997e-05 2024-04-19T01:04:20.421 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:20.421 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:20.421 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:20.421 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.421 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.421 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s2 43/31 ECSubWrite(tid=76, reqid=client.4604.0:267, at_version=43'46, trim_to=0'0, roll_forward_to=43'45))", 2024-04-19T01:04:20.421 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.026909+0000", 2024-04-19T01:04:20.421 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.3489022789999998, 2024-04-19T01:04:20.421 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010819211, 2024-04-19T01:04:20.421 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:20.421 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:20.421 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:20.421 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:20.422 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.422 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:20.422 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.026909+0000", 2024-04-19T01:04:20.422 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:20.422 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.422 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.422 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:20.422 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.026909+0000", 2024-04-19T01:04:20.422 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:20.422 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.422 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.422 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:20.422 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.026911+0000", 2024-04-19T01:04:20.423 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3080000000000002e-06 2024-04-19T01:04:20.423 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.423 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.423 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:20.423 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.027220+0000", 2024-04-19T01:04:20.423 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00030838399999999998 2024-04-19T01:04:20.423 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.423 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.423 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:20.423 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.027247+0000", 2024-04-19T01:04:20.423 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7634e-05 2024-04-19T01:04:20.423 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.423 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.423 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:20.423 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.027268+0000", 2024-04-19T01:04:20.424 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0398999999999999e-05 2024-04-19T01:04:20.424 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.424 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.424 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:20.424 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.027373+0000", 2024-04-19T01:04:20.424 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000105709 2024-04-19T01:04:20.424 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.424 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.424 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:20.424 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.027384+0000", 2024-04-19T01:04:20.424 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1151e-05 2024-04-19T01:04:20.424 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.424 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.424 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:20.425 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.037668+0000", 2024-04-19T01:04:20.425 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010283494000000001 2024-04-19T01:04:20.425 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.425 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.425 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:20.425 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.037728+0000", 2024-04-19T01:04:20.425 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0132e-05 2024-04-19T01:04:20.425 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:20.425 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:20.425 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:20.425 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.425 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.425 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 44/31 ECSubWrite(tid=48, reqid=osd.8.0:48, at_version=44'31, trim_to=0'0, roll_forward_to=43'29))", 2024-04-19T01:04:20.425 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.875721+0000", 2024-04-19T01:04:20.426 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.5000904720000001, 2024-04-19T01:04:20.426 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011047829, 2024-04-19T01:04:20.426 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:20.426 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:20.426 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:20.426 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:20.426 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.426 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:20.427 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.875721+0000", 2024-04-19T01:04:20.427 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:20.427 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.427 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.427 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:20.427 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.875721+0000", 2024-04-19T01:04:20.427 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:20.427 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.427 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.427 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:20.427 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.875721+0000", 2024-04-19T01:04:20.427 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4000000000000001e-07 2024-04-19T01:04:20.427 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.427 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.428 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:20.428 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.875777+0000", 2024-04-19T01:04:20.428 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5352000000000001e-05 2024-04-19T01:04:20.428 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.428 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.428 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:20.428 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.875812+0000", 2024-04-19T01:04:20.428 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5274999999999997e-05 2024-04-19T01:04:20.428 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.428 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.428 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:20.428 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.875837+0000", 2024-04-19T01:04:20.428 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5035999999999999e-05 2024-04-19T01:04:20.428 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.428 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.429 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:20.429 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876027+0000", 2024-04-19T01:04:20.429 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000190354 2024-04-19T01:04:20.429 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.429 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.429 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:20.429 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876055+0000", 2024-04-19T01:04:20.429 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7866000000000001e-05 2024-04-19T01:04:20.429 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.429 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.429 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:20.429 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.886413+0000", 2024-04-19T01:04:20.429 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010357857 2024-04-19T01:04:20.429 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.430 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.430 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:20.430 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.886769+0000", 2024-04-19T01:04:20.430 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035544899999999998 2024-04-19T01:04:20.430 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:20.430 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:20.430 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:20.430 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.430 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.430 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 44/31 ECSubWrite(tid=49, reqid=osd.8.0:49, at_version=44'33, trim_to=0'0, roll_forward_to=43'29))", 2024-04-19T01:04:20.430 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.876887+0000", 2024-04-19T01:04:20.430 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.4989238870000001, 2024-04-19T01:04:20.430 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017055688999999999, 2024-04-19T01:04:20.430 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:20.431 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:20.431 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:20.431 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:20.431 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.431 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:20.431 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876887+0000", 2024-04-19T01:04:20.431 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:20.431 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.431 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.431 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:20.431 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876887+0000", 2024-04-19T01:04:20.431 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:20.431 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.431 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.431 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:20.432 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876889+0000", 2024-04-19T01:04:20.432 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.827e-06 2024-04-19T01:04:20.432 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.432 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.432 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:20.432 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876942+0000", 2024-04-19T01:04:20.432 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3248000000000002e-05 2024-04-19T01:04:20.432 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.432 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.432 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:20.432 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876984+0000", 2024-04-19T01:04:20.432 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1598999999999999e-05 2024-04-19T01:04:20.432 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.432 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.432 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:20.433 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.877007+0000", 2024-04-19T01:04:20.433 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3209000000000001e-05 2024-04-19T01:04:20.433 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.433 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.433 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:20.433 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.878441+0000", 2024-04-19T01:04:20.433 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0014341790000000001 2024-04-19T01:04:20.433 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.433 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.433 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:20.433 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.878457+0000", 2024-04-19T01:04:20.433 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5787e-05 2024-04-19T01:04:20.433 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.433 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.434 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:20.434 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.893844+0000", 2024-04-19T01:04:20.434 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015387283 2024-04-19T01:04:20.434 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.434 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.434 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:20.434 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.893943+0000", 2024-04-19T01:04:20.434 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8556999999999998e-05 2024-04-19T01:04:20.434 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:20.434 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:20.434 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:20.434 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.434 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.434 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s2 44/31 ECSubWrite(tid=82, reqid=client.4604.0:280, at_version=44'50, trim_to=0'0, roll_forward_to=43'48))", 2024-04-19T01:04:20.435 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.882222+0000", 2024-04-19T01:04:20.435 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.4935896020000001, 2024-04-19T01:04:20.435 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.035827673999999997, 2024-04-19T01:04:20.435 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:20.435 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:20.435 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:20.435 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:20.435 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.435 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:20.435 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882222+0000", 2024-04-19T01:04:20.435 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:20.435 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.435 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.435 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:20.435 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882222+0000", 2024-04-19T01:04:20.436 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:20.436 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.436 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.436 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:20.436 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882222+0000", 2024-04-19T01:04:20.436 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7999999999999995e-07 2024-04-19T01:04:20.436 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.436 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.436 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:20.436 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882709+0000", 2024-04-19T01:04:20.436 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00048730799999999999 2024-04-19T01:04:20.437 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.437 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.437 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:20.437 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882742+0000", 2024-04-19T01:04:20.437 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2428000000000002e-05 2024-04-19T01:04:20.437 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.437 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.437 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:20.437 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882765+0000", 2024-04-19T01:04:20.438 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2693000000000001e-05 2024-04-19T01:04:20.438 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.438 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.438 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:20.438 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.883062+0000", 2024-04-19T01:04:20.438 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029777100000000001 2024-04-19T01:04:20.438 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.438 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.438 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:20.438 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.883088+0000", 2024-04-19T01:04:20.438 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6018999999999999e-05 2024-04-19T01:04:20.438 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.438 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.439 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:20.439 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.917995+0000", 2024-04-19T01:04:20.439 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.034906988999999999 2024-04-19T01:04:20.439 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.439 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.439 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:20.439 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.918049+0000", 2024-04-19T01:04:20.439 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3885999999999999e-05 2024-04-19T01:04:20.439 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:20.439 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:20.439 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:20.439 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.439 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.439 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s4 44/31 ECSubWrite(tid=186, reqid=osd.11.0:186, at_version=44'12, trim_to=0'0, roll_forward_to=43'10))", 2024-04-19T01:04:20.440 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.883784+0000", 2024-04-19T01:04:20.440 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.4920274459999998, 2024-04-19T01:04:20.440 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.01028681, 2024-04-19T01:04:20.440 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:20.440 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:20.440 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:20.440 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:20.440 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.440 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:20.440 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.883784+0000", 2024-04-19T01:04:20.440 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:20.440 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.440 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.440 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:20.441 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.883784+0000", 2024-04-19T01:04:20.441 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:20.441 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.441 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.441 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:20.441 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.883784+0000", 2024-04-19T01:04:20.441 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1600000000000001e-07 2024-04-19T01:04:20.441 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.441 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.441 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:20.441 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.883860+0000", 2024-04-19T01:04:20.441 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5955e-05 2024-04-19T01:04:20.441 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.442 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.442 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:20.442 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.883889+0000", 2024-04-19T01:04:20.442 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9116e-05 2024-04-19T01:04:20.442 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.442 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.442 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:20.442 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.883911+0000", 2024-04-19T01:04:20.442 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1282000000000001e-05 2024-04-19T01:04:20.442 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.442 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.442 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:20.442 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.885494+0000", 2024-04-19T01:04:20.442 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001583575 2024-04-19T01:04:20.442 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.443 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.443 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:20.443 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.885519+0000", 2024-04-19T01:04:20.443 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5092999999999998e-05 2024-04-19T01:04:20.443 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.443 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.443 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:20.443 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.893986+0000", 2024-04-19T01:04:20.443 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0084664300000000005 2024-04-19T01:04:20.443 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.443 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.443 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:20.443 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.894071+0000", 2024-04-19T01:04:20.443 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4742999999999998e-05 2024-04-19T01:04:20.444 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:20.444 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:20.444 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:20.444 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.444 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.444 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.bs6 44/31 ECSubWrite(tid=42, reqid=osd.10.0:42, at_version=44'16, trim_to=0'0, roll_forward_to=43'14))", 2024-04-19T01:04:20.444 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.887817+0000", 2024-04-19T01:04:20.444 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.4879937180000002, 2024-04-19T01:04:20.444 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011064671, 2024-04-19T01:04:20.444 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:20.444 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:20.444 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:20.444 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:20.444 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.444 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:20.445 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.887817+0000", 2024-04-19T01:04:20.445 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:20.445 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.445 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.445 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:20.445 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.887817+0000", 2024-04-19T01:04:20.445 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:20.445 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.445 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.445 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:20.445 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.887819+0000", 2024-04-19T01:04:20.445 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6029999999999999e-06 2024-04-19T01:04:20.445 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.445 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.446 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:20.446 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.887875+0000", 2024-04-19T01:04:20.446 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5828999999999997e-05 2024-04-19T01:04:20.446 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.446 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.446 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:20.446 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.887911+0000", 2024-04-19T01:04:20.446 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6191e-05 2024-04-19T01:04:20.446 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.446 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.446 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:20.446 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.887938+0000", 2024-04-19T01:04:20.446 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6950000000000001e-05 2024-04-19T01:04:20.446 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.447 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.447 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:20.447 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.888169+0000", 2024-04-19T01:04:20.447 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000230609 2024-04-19T01:04:20.447 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.447 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.447 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:20.447 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.888195+0000", 2024-04-19T01:04:20.447 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6018999999999999e-05 2024-04-19T01:04:20.447 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.447 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.447 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:20.447 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.898790+0000", 2024-04-19T01:04:20.447 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010594927000000001 2024-04-19T01:04:20.448 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.448 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.448 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:20.448 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.898882+0000", 2024-04-19T01:04:20.448 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2542999999999998e-05 2024-04-19T01:04:20.448 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:20.448 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:20.448 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:20.448 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.448 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.448 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 44/31 ECSubWrite(tid=50, reqid=osd.8.0:49, at_version=0'0, trim_to=0'0, roll_forward_to=44'33))", 2024-04-19T01:04:20.448 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.901832+0000", 2024-04-19T01:04:20.448 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.4739791000000002, 2024-04-19T01:04:20.448 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011627197000000001, 2024-04-19T01:04:20.449 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:20.449 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:20.449 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:20.449 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:20.449 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.449 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:20.449 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.901832+0000", 2024-04-19T01:04:20.449 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:20.449 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.449 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.449 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:20.449 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.901832+0000", 2024-04-19T01:04:20.449 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:20.449 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.449 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.450 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:20.450 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.901834+0000", 2024-04-19T01:04:20.450 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.08e-06 2024-04-19T01:04:20.450 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.450 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.450 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:20.450 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.901863+0000", 2024-04-19T01:04:20.450 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8445999999999998e-05 2024-04-19T01:04:20.450 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.450 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.450 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:20.450 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.901895+0000", 2024-04-19T01:04:20.450 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2308999999999998e-05 2024-04-19T01:04:20.450 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.451 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.451 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:20.451 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.901917+0000", 2024-04-19T01:04:20.451 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2532999999999999e-05 2024-04-19T01:04:20.451 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.451 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.451 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:20.451 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.902071+0000", 2024-04-19T01:04:20.451 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015368099999999999 2024-04-19T01:04:20.451 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.451 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.451 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:20.451 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.902161+0000", 2024-04-19T01:04:20.451 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9703999999999994e-05 2024-04-19T01:04:20.452 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.452 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.452 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:20.452 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.913373+0000", 2024-04-19T01:04:20.452 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011212353 2024-04-19T01:04:20.452 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.452 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.452 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:20.452 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.913459+0000", 2024-04-19T01:04:20.452 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6091e-05 2024-04-19T01:04:20.452 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:20.452 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:20.452 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:20.452 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.452 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.453 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s2 44/31 ECSubWrite(tid=84, reqid=client.4604.0:281, at_version=44'51, trim_to=0'0, roll_forward_to=43'48))", 2024-04-19T01:04:20.453 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.907234+0000", 2024-04-19T01:04:20.453 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.4685776669999999, 2024-04-19T01:04:20.453 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011122504, 2024-04-19T01:04:20.453 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:20.453 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:20.453 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:20.453 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:20.453 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.453 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:20.453 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.907234+0000", 2024-04-19T01:04:20.453 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:20.453 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.453 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.454 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:20.454 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.907234+0000", 2024-04-19T01:04:20.454 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:20.454 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.454 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.454 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:20.454 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.907235+0000", 2024-04-19T01:04:20.454 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.824e-06 2024-04-19T01:04:20.454 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.454 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.454 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:20.454 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.907488+0000", 2024-04-19T01:04:20.454 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00025270499999999999 2024-04-19T01:04:20.454 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.454 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.455 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:20.455 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.907520+0000", 2024-04-19T01:04:20.455 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2308999999999998e-05 2024-04-19T01:04:20.455 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.455 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.455 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:20.455 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.907544+0000", 2024-04-19T01:04:20.455 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3519e-05 2024-04-19T01:04:20.455 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.455 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.455 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:20.455 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.908322+0000", 2024-04-19T01:04:20.455 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00077773499999999999 2024-04-19T01:04:20.455 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.455 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.456 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:20.456 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.908349+0000", 2024-04-19T01:04:20.456 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6916000000000001e-05 2024-04-19T01:04:20.456 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.456 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.456 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:20.456 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.918305+0000", 2024-04-19T01:04:20.456 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0099564100000000006 2024-04-19T01:04:20.456 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.456 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.456 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:20.456 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.918356+0000", 2024-04-19T01:04:20.456 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1085999999999998e-05 2024-04-19T01:04:20.456 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:20.457 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:20.457 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:20.457 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.457 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.457 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s4 49/31 ECSubWrite(tid=202, reqid=osd.11.0:201, at_version=0'0, trim_to=0'0, roll_forward_to=49'24))", 2024-04-19T01:04:20.457 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:19.537002+0000", 2024-04-19T01:04:20.457 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.83880890600000002, 2024-04-19T01:04:20.457 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023787906000000001, 2024-04-19T01:04:20.457 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:20.457 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:20.457 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:20.457 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:20.457 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.457 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:20.458 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.537002+0000", 2024-04-19T01:04:20.458 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:20.458 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.458 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.458 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:20.458 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.537002+0000", 2024-04-19T01:04:20.458 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:20.458 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.458 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.458 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:20.458 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.537004+0000", 2024-04-19T01:04:20.458 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.553e-06 2024-04-19T01:04:20.458 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.458 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.459 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:20.459 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.537031+0000", 2024-04-19T01:04:20.459 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7039999999999999e-05 2024-04-19T01:04:20.459 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.459 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.459 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:20.459 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.537063+0000", 2024-04-19T01:04:20.459 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1705000000000002e-05 2024-04-19T01:04:20.459 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.459 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.459 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:20.459 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.537084+0000", 2024-04-19T01:04:20.459 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1852999999999999e-05 2024-04-19T01:04:20.459 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.460 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.460 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:20.460 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.537559+0000", 2024-04-19T01:04:20.460 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00047440999999999998 2024-04-19T01:04:20.460 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.460 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.460 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:20.460 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.537585+0000", 2024-04-19T01:04:20.460 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6319e-05 2024-04-19T01:04:20.460 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.460 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.460 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:20.460 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.560733+0000", 2024-04-19T01:04:20.460 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023147581 2024-04-19T01:04:20.461 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.461 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.461 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:20.461 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.560790+0000", 2024-04-19T01:04:20.461 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7445000000000003e-05 2024-04-19T01:04:20.461 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:20.461 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:20.461 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:20.461 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.461 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.461 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s4 49/31 ECSubWrite(tid=203, reqid=client.4604.0:360, at_version=49'25, trim_to=0'0, roll_forward_to=49'24))", 2024-04-19T01:04:20.461 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:19.539163+0000", 2024-04-19T01:04:20.461 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.83664784199999997, 2024-04-19T01:04:20.461 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022464522000000001, 2024-04-19T01:04:20.461 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:20.462 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:20.462 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:20.462 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:20.462 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.462 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:20.462 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.539163+0000", 2024-04-19T01:04:20.462 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:20.462 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.462 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.462 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:20.462 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.539163+0000", 2024-04-19T01:04:20.462 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:20.462 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.462 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.463 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:20.463 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.539165+0000", 2024-04-19T01:04:20.463 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.95e-06 2024-04-19T01:04:20.463 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.463 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.463 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:20.463 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.539991+0000", 2024-04-19T01:04:20.463 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00082553499999999996 2024-04-19T01:04:20.463 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.463 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.463 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:20.463 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.540003+0000", 2024-04-19T01:04:20.463 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1838000000000001e-05 2024-04-19T01:04:20.463 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.463 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.464 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:20.464 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.540011+0000", 2024-04-19T01:04:20.464 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3829999999999992e-06 2024-04-19T01:04:20.464 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.464 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.464 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:20.464 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.540704+0000", 2024-04-19T01:04:20.464 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00069282000000000002 2024-04-19T01:04:20.464 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.464 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.464 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:20.464 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.540730+0000", 2024-04-19T01:04:20.464 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5649000000000001e-05 2024-04-19T01:04:20.464 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.465 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.465 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:20.465 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.561559+0000", 2024-04-19T01:04:20.465 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020829555 2024-04-19T01:04:20.465 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.465 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.465 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:20.465 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.561628+0000", 2024-04-19T01:04:20.465 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8792000000000001e-05 2024-04-19T01:04:20.465 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:20.465 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:20.465 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:20.465 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.465 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.466 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s4 49/31 ECSubWrite(tid=204, reqid=client.4604.0:361, at_version=49'26, trim_to=0'0, roll_forward_to=49'24))", 2024-04-19T01:04:20.466 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:19.561460+0000", 2024-04-19T01:04:20.466 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.81435106599999996, 2024-04-19T01:04:20.466 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010050376, 2024-04-19T01:04:20.466 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:20.466 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:20.466 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:20.466 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:20.466 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.466 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:20.466 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.561460+0000", 2024-04-19T01:04:20.466 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:20.466 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.467 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.467 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:20.467 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.561460+0000", 2024-04-19T01:04:20.467 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:20.467 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.467 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.467 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:20.467 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.561462+0000", 2024-04-19T01:04:20.467 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.567e-06 2024-04-19T01:04:20.467 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.467 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.467 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:20.467 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.561717+0000", 2024-04-19T01:04:20.467 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00025506599999999997 2024-04-19T01:04:20.467 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.468 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.468 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:20.468 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.561741+0000", 2024-04-19T01:04:20.468 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3788999999999999e-05 2024-04-19T01:04:20.468 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.468 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.468 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:20.468 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.561755+0000", 2024-04-19T01:04:20.468 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4059999999999999e-05 2024-04-19T01:04:20.468 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.468 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.468 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:20.468 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.561912+0000", 2024-04-19T01:04:20.468 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000157429 2024-04-19T01:04:20.468 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.469 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.469 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:20.469 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.561941+0000", 2024-04-19T01:04:20.469 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8756e-05 2024-04-19T01:04:20.469 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.469 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.469 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:20.469 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.571478+0000", 2024-04-19T01:04:20.469 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0095372459999999992 2024-04-19T01:04:20.469 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:20.469 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:20.469 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:20.469 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.571511+0000", 2024-04-19T01:04:20.469 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2462999999999997e-05 2024-04-19T01:04:20.470 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:20.470 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:20.470 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:20.470 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:20.470 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:20.470 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:04:20.471 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2024-04-19T01:04:20.471 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-19T01:04:20.472 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2024-04-19T01:04:20.472 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-19T01:04:20.473 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-19T01:04:20.474 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:20.471+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:20.515 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2024-04-19T01:04:20.515 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.14 to 1 (10000) 2024-04-19T01:04:20.521 INFO:tasks.rados.rados.0.smithi155.stdout:140: done (2 left) 2024-04-19T01:04:20.522 INFO:tasks.rados.rados.0.smithi155.stdout:141: done (1 left) 2024-04-19T01:04:20.522 INFO:tasks.rados.rados.0.smithi155.stdout:142: done (0 left) 2024-04-19T01:04:20.522 INFO:tasks.rados.rados.0.smithi155.stdout:143: snap_create 2024-04-19T01:04:20.543 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 15 1 2024-04-19T01:04:20.572 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:04:20.571+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:04:20.606 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:20.606 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:20.619 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-19T01:04:20.673 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:04:20.673+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:04:20.740 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:20.740 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:20.755 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-19T01:04:20.774 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:04:20.772+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:04:20.875 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:04:20.871+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:04:20.923 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:20.923 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:20.937 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-19T01:04:20.976 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:04:20.976+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:04:21.055 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:21.055 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:21.069 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-19T01:04:21.077 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:21.076+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:21.180 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:04:21.176+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:04:21.186 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:21.187 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:21.201 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-19T01:04:21.279 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:04:21.279+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:04:21.319 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:21.319 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:21.334 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-19T01:04:21.381 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:04:21.379+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:04:21.451 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:21.451 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:21.471 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-19T01:04:21.496 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.15 to 1 (10000) 2024-04-19T01:04:21.502 INFO:tasks.rados.rados.0.smithi155.stdout:143: done (0 left) 2024-04-19T01:04:21.502 INFO:tasks.rados.rados.0.smithi155.stdout:144: copy_from oid 26 from oid 1 current snap is 10 2024-04-19T01:04:21.502 INFO:tasks.rados.rados.0.smithi155.stdout:145: snap_remove snap 5 2024-04-19T01:04:21.502 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:04:21.501+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:04:21.526 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-19T01:04:21.526 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T01:04:21.583 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:04:21.584+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:04:21.600 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:21.600 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:21.614 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-19T01:04:21.684 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:04:21.684+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:04:21.777 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:21.777 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:21.785 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:21.784+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:21.791 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-04-19T01:04:21.877 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:04:21.878 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-04-19T01:04:21.886 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:04:21.887+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:04:21.896 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":53,"stamp":"2024-04-19T01:04:21.525165+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":39780896,"num_objects":68,"num_object_clones":18,"num_object_copies":466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":2,"num_read":316,"num_read_kb":20165,"num_write":524,"num_write_kb":44404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":552,"ondisk_log_size":552,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":524924,"kb_used_data":75820,"kb_used_omap":24,"kb_used_meta":448743,"kb_avail":1509424516,"statfs":{"total":1546188226560,"available":1545650704384,"internally_reserved":0,"allocated":77639680,"data_stored":91255739,"data_compressed":759887,"data_compressed_allocated":1617920,"data_compressed_original":4128768,"omap_allocated":25520,"internal_metadata":459512912},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4046848,"num_objects":1,"num_object_clones":2,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":5526,"num_write":119,"num_write_kb":6256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001607"},"pg_stats":[{"pgid":"3.f","version":"45'14","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-19T01:04:15.881381+0000","last_change":"2024-04-19T01:04:15.882117+0000","last_active":"2024-04-19T01:04:15.881381+0000","last_peered":"2024-04-19T01:04:15.881381+0000","last_clean":"2024-04-19T01:04:15.881381+0000","last_became_active":"2024-04-19T01:04:03.238651+0000","last_became_peered":"2024-04-19T01:04:03.238651+0000","last_unstale":"2024-04-19T01:04:15.881381+0000","last_undegraded":"2024-04-19T01:04:15.881381+0000","last_fullsized":"2024-04-19T01:04:15.881381+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:29.932463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01334835,"stat_sum":{"num_bytes":1900544,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":1397,"num_write":9,"num_write_kb":1856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,5,11,7,1,2],"acting":[0,13,5,11,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"5"}]},{"pgid":"3.e","version":"45'16","reported_seq":94,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-19T01:04:15.888767+0000","last_change":"2024-04-19T01:04:15.890755+0000","last_active":"2024-04-19T01:04:15.888767+0000","last_peered":"2024-04-19T01:04:15.888767+0000","last_clean":"2024-04-19T01:04:15.888767+0000","last_became_active":"2024-04-19T01:04:03.238386+0000","last_became_peered":"2024-04-19T01:04:03.238386+0000","last_unstale":"2024-04-19T01:04:15.888767+0000","last_undegraded":"2024-04-19T01:04:15.888767+0000","last_fullsized":"2024-04-19T01:04:15.888767+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:14.290256+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.01267603,"stat_sum":{"num_bytes":1703936,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":3700,"num_write":7,"num_write_kb":1664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11,5,4,2,9,15],"acting":[13,11,5,4,2,9,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"5"}]},{"pgid":"3.d","version":"49'43","reported_seq":133,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T01:04:19.547108+0000","last_change":"2024-04-19T01:04:19.548488+0000","last_active":"2024-04-19T01:04:19.547108+0000","last_peered":"2024-04-19T01:04:19.547108+0000","last_clean":"2024-04-19T01:04:19.547108+0000","last_became_active":"2024-04-19T01:04:03.025363+0000","last_became_peered":"2024-04-19T01:04:03.025363+0000","last_unstale":"2024-04-19T01:04:19.547108+0000","last_undegraded":"2024-04-19T01:04:19.547108+0000","last_fullsized":"2024-04-19T01:04:19.547108+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:02.599042+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.019013540999999998,"stat_sum":{"num_bytes":1540096,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":2560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,9,0,8,2,6],"acting":[12,4,9,0,8,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"},{"start":"8","length":"2"}]},{"pgid":"3.c","version":"45'29","reported_seq":103,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-19T01:04:15.893313+0000","last_change":"2024-04-19T01:04:15.893950+0000","last_active":"2024-04-19T01:04:15.893313+0000","last_peered":"2024-04-19T01:04:15.893313+0000","last_clean":"2024-04-19T01:04:15.893313+0000","last_became_active":"2024-04-19T01:04:03.025808+0000","last_became_peered":"2024-04-19T01:04:03.025808+0000","last_unstale":"2024-04-19T01:04:15.893313+0000","last_undegraded":"2024-04-19T01:04:15.893313+0000","last_fullsized":"2024-04-19T01:04:15.893313+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:09.514383+0000","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0.025445307,"stat_sum":{"num_bytes":1884160,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":13,"num_read_kb":2453,"num_write":32,"num_write_kb":1904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,12,1,9,2,6],"acting":[0,10,12,1,9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"5"}]},{"pgid":"3.b","version":"45'18","reported_seq":108,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-19T01:04:18.518324+0000","last_change":"2024-04-19T01:04:18.519279+0000","last_active":"2024-04-19T01:04:18.518324+0000","last_peered":"2024-04-19T01:04:18.518324+0000","last_clean":"2024-04-19T01:04:18.518324+0000","last_became_active":"2024-04-19T01:04:02.781569+0000","last_became_peered":"2024-04-19T01:04:02.781569+0000","last_unstale":"2024-04-19T01:04:18.518324+0000","last_undegraded":"2024-04-19T01:04:18.518324+0000","last_fullsized":"2024-04-19T01:04:18.518324+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:16.849719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00152271,"stat_sum":{"num_bytes":2129920,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":7,"num_read_kb":4,"num_write":9,"num_write_kb":2080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11,15,1,4,14],"acting":[10,6,11,15,1,4,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"},{"start":"9","length":"1"}]},{"pgid":"3.a","version":"49'12","reported_seq":124,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T01:04:19.557884+0000","last_change":"2024-04-19T01:04:19.558656+0000","last_active":"2024-04-19T01:04:19.557884+0000","last_peered":"2024-04-19T01:04:19.557884+0000","last_clean":"2024-04-19T01:04:19.557884+0000","last_became_active":"2024-04-19T01:04:03.323177+0000","last_became_peered":"2024-04-19T01:04:03.323177+0000","last_unstale":"2024-04-19T01:04:19.557884+0000","last_undegraded":"2024-04-19T01:04:19.557884+0000","last_fullsized":"2024-04-19T01:04:19.557884+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:14.771327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.029493378000000001,"stat_sum":{"num_bytes":3440640,"num_objects":5,"num_object_clones":1,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":5816,"num_write":9,"num_write_kb":3360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,7,3,13,1,8],"acting":[12,4,7,3,13,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"},{"start":"8","length":"2"}]},{"pgid":"3.9","version":"45'64","reported_seq":142,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-19T01:04:15.902226+0000","last_change":"2024-04-19T01:04:15.903771+0000","last_active":"2024-04-19T01:04:15.902226+0000","last_peered":"2024-04-19T01:04:15.902226+0000","last_clean":"2024-04-19T01:04:15.902226+0000","last_became_active":"2024-04-19T01:04:03.027372+0000","last_became_peered":"2024-04-19T01:04:03.027372+0000","last_unstale":"2024-04-19T01:04:15.902226+0000","last_undegraded":"2024-04-19T01:04:15.902226+0000","last_fullsized":"2024-04-19T01:04:15.902226+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:06.869418+0000","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0.025485052000000001,"stat_sum":{"num_bytes":9240576,"num_objects":8,"num_object_clones":2,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":50,"num_write_kb":9024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11,14,9,4,10,6],"acting":[13,11,14,9,4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"5"}]},{"pgid":"3.8","version":"49'25","reported_seq":116,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T01:04:19.534323+0000","last_change":"2024-04-19T01:04:19.549872+0000","last_active":"2024-04-19T01:04:19.534323+0000","last_peered":"2024-04-19T01:04:19.534323+0000","last_clean":"2024-04-19T01:04:19.534323+0000","last_became_active":"2024-04-19T01:04:03.237653+0000","last_became_peered":"2024-04-19T01:04:03.237653+0000","last_unstale":"2024-04-19T01:04:19.534323+0000","last_undegraded":"2024-04-19T01:04:19.534323+0000","last_fullsized":"2024-04-19T01:04:19.534323+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:16.852722+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.030904131000000001,"stat_sum":{"num_bytes":2195456,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":30,"num_write_kb":2144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,1,15,14,0,5],"acting":[11,10,1,15,14,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"},{"start":"8","length":"2"}]},{"pgid":"1.0","version":"25'157","reported_seq":256,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T01:04:19.518472+0000","last_change":"2024-04-19T01:03:50.776344+0000","last_active":"2024-04-19T01:04:19.518472+0000","last_peered":"2024-04-19T01:04:19.518472+0000","last_clean":"2024-04-19T01:04:19.518472+0000","last_became_active":"2024-04-19T01:03:50.774078+0000","last_became_peered":"2024-04-19T01:03:50.774078+0000","last_unstale":"2024-04-19T01:04:19.518472+0000","last_undegraded":"2024-04-19T01:04:19.518472+0000","last_fullsized":"2024-04-19T01:04:19.518472+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:49.580851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:49.580851+0000","last_clean_scrub_stamp":"2024-04-19T01:03:49.580851+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:07.495976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":54,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T01:04:19.518443+0000","last_change":"2024-04-19T01:03:53.766587+0000","last_active":"2024-04-19T01:04:19.518443+0000","last_peered":"2024-04-19T01:04:19.518443+0000","last_clean":"2024-04-19T01:04:19.518443+0000","last_became_active":"2024-04-19T01:03:53.765967+0000","last_became_peered":"2024-04-19T01:03:53.765967+0000","last_unstale":"2024-04-19T01:04:19.518443+0000","last_undegraded":"2024-04-19T01:04:19.518443+0000","last_fullsized":"2024-04-19T01:04:19.518443+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:18.833542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"45'14","reported_seq":108,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T01:04:19.519270+0000","last_change":"2024-04-19T01:04:19.520142+0000","last_active":"2024-04-19T01:04:19.519270+0000","last_peered":"2024-04-19T01:04:19.519270+0000","last_clean":"2024-04-19T01:04:19.519270+0000","last_became_active":"2024-04-19T01:04:03.322437+0000","last_became_peered":"2024-04-19T01:04:03.322437+0000","last_unstale":"2024-04-19T01:04:19.519270+0000","last_undegraded":"2024-04-19T01:04:19.519270+0000","last_fullsized":"2024-04-19T01:04:19.519270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:06.317438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001368743,"stat_sum":{"num_bytes":344064,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":6,"num_write_kb":336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0,4,8,3,7],"acting":[11,5,0,4,8,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"},{"start":"8","length":"2"}]},{"pgid":"2.0","version":"0'0","reported_seq":54,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T01:04:19.517419+0000","last_change":"2024-04-19T01:03:53.760075+0000","last_active":"2024-04-19T01:04:19.517419+0000","last_peered":"2024-04-19T01:04:19.517419+0000","last_clean":"2024-04-19T01:04:19.517419+0000","last_became_active":"2024-04-19T01:03:53.759326+0000","last_became_peered":"2024-04-19T01:03:53.759326+0000","last_unstale":"2024-04-19T01:04:19.517419+0000","last_undegraded":"2024-04-19T01:04:19.517419+0000","last_fullsized":"2024-04-19T01:04:19.517419+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:02.718012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"47'12","reported_seq":103,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-19T01:04:18.517899+0000","last_change":"2024-04-19T01:04:18.518427+0000","last_active":"2024-04-19T01:04:18.517899+0000","last_peered":"2024-04-19T01:04:18.517899+0000","last_clean":"2024-04-19T01:04:18.517899+0000","last_became_active":"2024-04-19T01:04:03.323727+0000","last_became_peered":"2024-04-19T01:04:03.323727+0000","last_unstale":"2024-04-19T01:04:18.517899+0000","last_undegraded":"2024-04-19T01:04:18.517899+0000","last_fullsized":"2024-04-19T01:04:18.517899+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:29.482112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011327349999999999,"stat_sum":{"num_bytes":1523712,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":2020,"num_write":27,"num_write_kb":1488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,11,9,1,7],"acting":[6,4,3,11,9,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"},{"start":"9","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":54,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T01:04:19.528833+0000","last_change":"2024-04-19T01:03:53.650434+0000","last_active":"2024-04-19T01:04:19.528833+0000","last_peered":"2024-04-19T01:04:19.528833+0000","last_clean":"2024-04-19T01:04:19.528833+0000","last_became_active":"2024-04-19T01:03:53.649606+0000","last_became_peered":"2024-04-19T01:03:53.649606+0000","last_unstale":"2024-04-19T01:04:19.528833+0000","last_undegraded":"2024-04-19T01:04:19.528833+0000","last_fullsized":"2024-04-19T01:04:19.528833+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:17.424729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"49'35","reported_seq":127,"reported_epoch":49,"state":"active+clean+snaptrim","last_fresh":"2024-04-19T01:04:19.565237+0000","last_change":"2024-04-19T01:04:19.532007+0000","last_active":"2024-04-19T01:04:19.565237+0000","last_peered":"2024-04-19T01:04:19.565237+0000","last_clean":"2024-04-19T01:04:19.565237+0000","last_became_active":"2024-04-19T01:04:03.322503+0000","last_became_peered":"2024-04-19T01:04:03.322503+0000","last_unstale":"2024-04-19T01:04:19.565237+0000","last_undegraded":"2024-04-19T01:04:19.565237+0000","last_fullsized":"2024-04-19T01:04:19.565237+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:13.454365+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.012386256,"stat_sum":{"num_bytes":3817472,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":15,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4,3,10,11,6],"acting":[1,9,4,3,10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"},{"start":"9","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":54,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T01:04:19.517767+0000","last_change":"2024-04-19T01:03:54.012138+0000","last_active":"2024-04-19T01:04:19.517767+0000","last_peered":"2024-04-19T01:04:19.517767+0000","last_clean":"2024-04-19T01:04:19.517767+0000","last_became_active":"2024-04-19T01:03:54.011460+0000","last_became_peered":"2024-04-19T01:03:54.011460+0000","last_unstale":"2024-04-19T01:04:19.517767+0000","last_undegraded":"2024-04-19T01:04:19.517767+0000","last_fullsized":"2024-04-19T01:04:19.517767+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:04:58.336612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"49'42","reported_seq":137,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T01:04:19.561330+0000","last_change":"2024-04-19T01:04:19.563217+0000","last_active":"2024-04-19T01:04:19.561330+0000","last_peered":"2024-04-19T01:04:19.561330+0000","last_clean":"2024-04-19T01:04:19.561330+0000","last_became_active":"2024-04-19T01:04:02.835716+0000","last_became_peered":"2024-04-19T01:04:02.835716+0000","last_unstale":"2024-04-19T01:04:19.561330+0000","last_undegraded":"2024-04-19T01:04:19.561330+0000","last_fullsized":"2024-04-19T01:04:19.561330+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:14.688145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032384638,"stat_sum":{"num_bytes":3833856,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":13,"num_read_kb":1110,"num_write":55,"num_write_kb":3744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2,0,1,4,12],"acting":[15,8,2,0,1,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"},{"start":"8","length":"2"}]},{"pgid":"3.5","version":"45'43","reported_seq":147,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T01:04:19.518177+0000","last_change":"2024-04-19T01:04:19.518641+0000","last_active":"2024-04-19T01:04:19.518177+0000","last_peered":"2024-04-19T01:04:19.518177+0000","last_clean":"2024-04-19T01:04:19.518177+0000","last_became_active":"2024-04-19T01:04:03.237886+0000","last_became_peered":"2024-04-19T01:04:03.237886+0000","last_unstale":"2024-04-19T01:04:19.518177+0000","last_undegraded":"2024-04-19T01:04:19.518177+0000","last_fullsized":"2024-04-19T01:04:19.518177+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:13.565446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068062999999999999,"stat_sum":{"num_bytes":3293184,"num_objects":9,"num_object_clones":3,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":30,"num_read_kb":2938,"num_write":39,"num_write_kb":4432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,14,11,10,2,5],"acting":[8,6,14,11,10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"},{"start":"8","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":52,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-19T01:04:18.517402+0000","last_change":"2024-04-19T01:03:53.716606+0000","last_active":"2024-04-19T01:04:18.517402+0000","last_peered":"2024-04-19T01:04:18.517402+0000","last_clean":"2024-04-19T01:04:18.517402+0000","last_became_active":"2024-04-19T01:03:53.716040+0000","last_became_peered":"2024-04-19T01:03:53.716040+0000","last_unstale":"2024-04-19T01:04:18.517402+0000","last_undegraded":"2024-04-19T01:04:18.517402+0000","last_fullsized":"2024-04-19T01:04:18.517402+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:01.975515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"48'16","reported_seq":111,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-19T01:04:18.536522+0000","last_change":"2024-04-19T01:04:18.536522+0000","last_active":"2024-04-19T01:04:18.536522+0000","last_peered":"2024-04-19T01:04:18.536522+0000","last_clean":"2024-04-19T01:04:18.536522+0000","last_became_active":"2024-04-19T01:04:03.322815+0000","last_became_peered":"2024-04-19T01:04:03.322815+0000","last_unstale":"2024-04-19T01:04:18.536522+0000","last_undegraded":"2024-04-19T01:04:18.536522+0000","last_fullsized":"2024-04-19T01:04:18.536522+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:12.411696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018823612,"stat_sum":{"num_bytes":753664,"num_objects":5,"num_object_clones":1,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":582,"num_write":12,"num_write_kb":1072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,13,14,3,15,1,5],"acting":[10,13,14,3,15,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"},{"start":"9","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":48,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-19T01:04:16.499015+0000","last_change":"2024-04-19T01:03:53.772709+0000","last_active":"2024-04-19T01:04:16.499015+0000","last_peered":"2024-04-19T01:04:16.499015+0000","last_clean":"2024-04-19T01:04:16.499015+0000","last_became_active":"2024-04-19T01:03:53.771887+0000","last_became_peered":"2024-04-19T01:03:53.771887+0000","last_unstale":"2024-04-19T01:04:16.499015+0000","last_undegraded":"2024-04-19T01:04:16.499015+0000","last_fullsized":"2024-04-19T01:04:16.499015+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:17.479766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"45'12","reported_seq":86,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-19T01:04:15.880263+0000","last_change":"2024-04-19T01:04:15.883168+0000","last_active":"2024-04-19T01:04:15.880263+0000","last_peered":"2024-04-19T01:04:15.880263+0000","last_clean":"2024-04-19T01:04:15.880263+0000","last_became_active":"2024-04-19T01:04:03.238517+0000","last_became_peered":"2024-04-19T01:04:03.238517+0000","last_unstale":"2024-04-19T01:04:15.880263+0000","last_undegraded":"2024-04-19T01:04:15.880263+0000","last_fullsized":"2024-04-19T01:04:15.880263+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:23.314696+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.016351998,"stat_sum":{"num_bytes":1589248,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":5,"num_write_kb":1552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,7,5,11,6,0],"acting":[14,13,7,5,11,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"5"}]},{"pgid":"2.6","version":"0'0","reported_seq":46,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-19T01:04:15.866554+0000","last_change":"2024-04-19T01:03:53.736290+0000","last_active":"2024-04-19T01:04:15.866554+0000","last_peered":"2024-04-19T01:04:15.866554+0000","last_clean":"2024-04-19T01:04:15.866554+0000","last_became_active":"2024-04-19T01:03:53.735549+0000","last_became_peered":"2024-04-19T01:03:53.735549+0000","last_unstale":"2024-04-19T01:04:15.866554+0000","last_undegraded":"2024-04-19T01:04:15.866554+0000","last_fullsized":"2024-04-19T01:04:15.866554+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:07.640918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13],"acting":[14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":73,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-19T01:04:15.866385+0000","last_change":"2024-04-19T01:04:15.867502+0000","last_active":"2024-04-19T01:04:15.866385+0000","last_peered":"2024-04-19T01:04:15.866385+0000","last_clean":"2024-04-19T01:04:15.866385+0000","last_became_active":"2024-04-19T01:04:02.889974+0000","last_became_peered":"2024-04-19T01:04:02.889974+0000","last_unstale":"2024-04-19T01:04:15.866385+0000","last_undegraded":"2024-04-19T01:04:15.866385+0000","last_fullsized":"2024-04-19T01:04:15.866385+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:17.036263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017315200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7,15,8,11,0,1],"acting":[14,7,15,8,11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"5"}]},{"pgid":"2.7","version":"0'0","reported_seq":54,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T01:04:19.517740+0000","last_change":"2024-04-19T01:03:53.939441+0000","last_active":"2024-04-19T01:04:19.517740+0000","last_peered":"2024-04-19T01:04:19.517740+0000","last_clean":"2024-04-19T01:04:19.517740+0000","last_became_active":"2024-04-19T01:03:53.938774+0000","last_became_peered":"2024-04-19T01:03:53.938774+0000","last_unstale":"2024-04-19T01:04:19.517740+0000","last_undegraded":"2024-04-19T01:04:19.517740+0000","last_fullsized":"2024-04-19T01:04:19.517740+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:14.081709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":39190528,"num_objects":66,"num_object_clones":18,"num_object_copies":462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":2,"num_read":180,"num_read_kb":20047,"num_write":321,"num_write_kb":40944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":68788224,"data_stored":82378752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":13},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":15546,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":14,"up_from":24,"seq":103079215113,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36192,"kb_used_data":5012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335648,"statfs":{"total":96636764160,"available":96599703552,"internally_reserved":0,"allocated":5132288,"data_stored":6184199,"data_compressed":43152,"data_compressed_allocated":90112,"data_compressed_original":180224,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,5,6,7,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":13,"up_from":24,"seq":103079215114,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35824,"kb_used_data":4640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336016,"statfs":{"total":96636764160,"available":96600080384,"internally_reserved":0,"allocated":4751360,"data_stored":6382790,"data_compressed":41151,"data_compressed_allocated":86016,"data_compressed_original":172032,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,4,5,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":9,"up_from":24,"seq":103079215115,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36120,"kb_used_data":4932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335720,"statfs":{"total":96636764160,"available":96599777280,"internally_reserved":0,"allocated":5050368,"data_stored":5926151,"data_compressed":43152,"data_compressed_allocated":90112,"data_compressed_original":180224,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313226,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33068,"kb_used_data":6040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338772,"statfs":{"total":96636764160,"available":96602902528,"internally_reserved":0,"allocated":6184960,"data_stored":7486833,"data_compressed":47144,"data_compressed_allocated":98304,"data_compressed_original":196608,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,3,4,5,6,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280520,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33904,"kb_used_data":6872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337936,"statfs":{"total":96636764160,"available":96602046464,"internally_reserved":0,"allocated":7036928,"data_stored":8869330,"data_compressed":49143,"data_compressed_allocated":102400,"data_compressed_original":204800,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280522,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33264,"kb_used_data":6232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338576,"statfs":{"total":96636764160,"available":96602701824,"internally_reserved":0,"allocated":6381568,"data_stored":6997458,"data_compressed":49143,"data_compressed_allocated":102400,"data_compressed_original":204800,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280521,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29944,"kb_used_data":2912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341896,"statfs":{"total":96636764160,"available":96606101504,"internally_reserved":0,"allocated":2981888,"data_stored":2897362,"data_compressed":49143,"data_compressed_allocated":102400,"data_compressed_original":204800,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[2,4,5,6,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280521,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34528,"kb_used_data":7496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337312,"statfs":{"total":96636764160,"available":96601407488,"internally_reserved":0,"allocated":7675904,"data_stored":9602514,"data_compressed":49143,"data_compressed_allocated":102400,"data_compressed_original":204800,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280521,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30944,"kb_used_data":3916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340896,"statfs":{"total":96636764160,"available":96605077504,"internally_reserved":0,"allocated":4009984,"data_stored":4892658,"data_compressed":56916,"data_compressed_allocated":135168,"data_compressed_original":716800,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,6,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280522,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30836,"kb_used_data":3804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341004,"statfs":{"total":96636764160,"available":96605188096,"internally_reserved":0,"allocated":3895296,"data_stored":4040146,"data_compressed":49143,"data_compressed_allocated":102400,"data_compressed_original":204800,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[2,5,6,7,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280520,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34840,"kb_used_data":7812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337000,"statfs":{"total":96636764160,"available":96601088000,"internally_reserved":0,"allocated":7999488,"data_stored":10557426,"data_compressed":56916,"data_compressed_allocated":135168,"data_compressed_original":716800,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,3,4,5,6,7,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280520,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30408,"kb_used_data":3376,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341432,"statfs":{"total":96636764160,"available":96605626368,"internally_reserved":0,"allocated":3457024,"data_stored":3679698,"data_compressed":49143,"data_compressed_allocated":102400,"data_compressed_original":204800,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280521,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29984,"kb_used_data":2952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341856,"statfs":{"total":96636764160,"available":96606060544,"internally_reserved":0,"allocated":3022848,"data_stored":3478994,"data_compressed":49143,"data_compressed_allocated":102400,"data_compressed_original":204800,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,4,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":24,"seq":103079215113,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35304,"kb_used_data":4116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336536,"statfs":{"total":96636764160,"available":96600612864,"internally_reserved":0,"allocated":4214784,"data_stored":4527302,"data_compressed":41151,"data_compressed_allocated":86016,"data_compressed_original":172032,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215114,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30484,"kb_used_data":3456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341356,"statfs":{"total":96636764160,"available":96605548544,"internally_reserved":0,"allocated":3538944,"data_stored":3554567,"data_compressed":43152,"data_compressed_allocated":90112,"data_compressed_original":180224,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":24,"seq":103079215113,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29280,"kb_used_data":2252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342560,"statfs":{"total":96636764160,"available":96606781440,"internally_reserved":0,"allocated":2306048,"data_stored":2178311,"data_compressed":43152,"data_compressed_allocated":90112,"data_compressed_original":180224,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3063808,"data_stored":3264512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6119424,"data_stored":6770688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3424256,"data_stored":3559424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2469888,"data_stored":2576384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":6610944,"data_stored":7983104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3588096,"data_stored":4050944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":6524928,"data_stored":8548352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1830912,"data_stored":1888256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":3112960,"data_stored":3391488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4575232,"data_stored":5636096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":5685248,"data_stored":8261632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":7401472,"data_stored":8871936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":2674688,"data_stored":3031040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":4288512,"data_stored":5378048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":4657152,"data_stored":5894144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":2760704,"data_stored":3272704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T01:04:21.898 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T01:04:21.915 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:21.915 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:21.930 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-04-19T01:04:21.987 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:21.987+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:22.047 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:22.047 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:22.061 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-04-19T01:04:22.088 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:04:22.087+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:04:22.184 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:22.184 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:22.189 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:04:22.187+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:04:22.198 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-19T01:04:22.200 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:04:22.200 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-04-19T01:04:22.220 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":53,"stamp":"2024-04-19T01:04:21.525165+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":39780896,"num_objects":68,"num_object_clones":18,"num_object_copies":466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":2,"num_read":316,"num_read_kb":20165,"num_write":524,"num_write_kb":44404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":552,"ondisk_log_size":552,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":524924,"kb_used_data":75820,"kb_used_omap":24,"kb_used_meta":448743,"kb_avail":1509424516,"statfs":{"total":1546188226560,"available":1545650704384,"internally_reserved":0,"allocated":77639680,"data_stored":91255739,"data_compressed":759887,"data_compressed_allocated":1617920,"data_compressed_original":4128768,"omap_allocated":25520,"internal_metadata":459512912},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4046848,"num_objects":1,"num_object_clones":2,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":5526,"num_write":119,"num_write_kb":6256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001607"},"pg_stats":[{"pgid":"3.f","version":"45'14","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-19T01:04:15.881381+0000","last_change":"2024-04-19T01:04:15.882117+0000","last_active":"2024-04-19T01:04:15.881381+0000","last_peered":"2024-04-19T01:04:15.881381+0000","last_clean":"2024-04-19T01:04:15.881381+0000","last_became_active":"2024-04-19T01:04:03.238651+0000","last_became_peered":"2024-04-19T01:04:03.238651+0000","last_unstale":"2024-04-19T01:04:15.881381+0000","last_undegraded":"2024-04-19T01:04:15.881381+0000","last_fullsized":"2024-04-19T01:04:15.881381+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:29.932463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01334835,"stat_sum":{"num_bytes":1900544,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":1397,"num_write":9,"num_write_kb":1856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,5,11,7,1,2],"acting":[0,13,5,11,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"5"}]},{"pgid":"3.e","version":"45'16","reported_seq":94,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-19T01:04:15.888767+0000","last_change":"2024-04-19T01:04:15.890755+0000","last_active":"2024-04-19T01:04:15.888767+0000","last_peered":"2024-04-19T01:04:15.888767+0000","last_clean":"2024-04-19T01:04:15.888767+0000","last_became_active":"2024-04-19T01:04:03.238386+0000","last_became_peered":"2024-04-19T01:04:03.238386+0000","last_unstale":"2024-04-19T01:04:15.888767+0000","last_undegraded":"2024-04-19T01:04:15.888767+0000","last_fullsized":"2024-04-19T01:04:15.888767+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:14.290256+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.01267603,"stat_sum":{"num_bytes":1703936,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":3700,"num_write":7,"num_write_kb":1664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11,5,4,2,9,15],"acting":[13,11,5,4,2,9,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"5"}]},{"pgid":"3.d","version":"49'43","reported_seq":133,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T01:04:19.547108+0000","last_change":"2024-04-19T01:04:19.548488+0000","last_active":"2024-04-19T01:04:19.547108+0000","last_peered":"2024-04-19T01:04:19.547108+0000","last_clean":"2024-04-19T01:04:19.547108+0000","last_became_active":"2024-04-19T01:04:03.025363+0000","last_became_peered":"2024-04-19T01:04:03.025363+0000","last_unstale":"2024-04-19T01:04:19.547108+0000","last_undegraded":"2024-04-19T01:04:19.547108+0000","last_fullsized":"2024-04-19T01:04:19.547108+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:02.599042+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.019013540999999998,"stat_sum":{"num_bytes":1540096,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":2560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,9,0,8,2,6],"acting":[12,4,9,0,8,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"},{"start":"8","length":"2"}]},{"pgid":"3.c","version":"45'29","reported_seq":103,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-19T01:04:15.893313+0000","last_change":"2024-04-19T01:04:15.893950+0000","last_active":"2024-04-19T01:04:15.893313+0000","last_peered":"2024-04-19T01:04:15.893313+0000","last_clean":"2024-04-19T01:04:15.893313+0000","last_became_active":"2024-04-19T01:04:03.025808+0000","last_became_peered":"2024-04-19T01:04:03.025808+0000","last_unstale":"2024-04-19T01:04:15.893313+0000","last_undegraded":"2024-04-19T01:04:15.893313+0000","last_fullsized":"2024-04-19T01:04:15.893313+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:09.514383+0000","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0.025445307,"stat_sum":{"num_bytes":1884160,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":13,"num_read_kb":2453,"num_write":32,"num_write_kb":1904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,12,1,9,2,6],"acting":[0,10,12,1,9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"5"}]},{"pgid":"3.b","version":"45'18","reported_seq":108,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-19T01:04:18.518324+0000","last_change":"2024-04-19T01:04:18.519279+0000","last_active":"2024-04-19T01:04:18.518324+0000","last_peered":"2024-04-19T01:04:18.518324+0000","last_clean":"2024-04-19T01:04:18.518324+0000","last_became_active":"2024-04-19T01:04:02.781569+0000","last_became_peered":"2024-04-19T01:04:02.781569+0000","last_unstale":"2024-04-19T01:04:18.518324+0000","last_undegraded":"2024-04-19T01:04:18.518324+0000","last_fullsized":"2024-04-19T01:04:18.518324+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:16.849719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00152271,"stat_sum":{"num_bytes":2129920,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":7,"num_read_kb":4,"num_write":9,"num_write_kb":2080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11,15,1,4,14],"acting":[10,6,11,15,1,4,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"},{"start":"9","length":"1"}]},{"pgid":"3.a","version":"49'12","reported_seq":124,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T01:04:19.557884+0000","last_change":"2024-04-19T01:04:19.558656+0000","last_active":"2024-04-19T01:04:19.557884+0000","last_peered":"2024-04-19T01:04:19.557884+0000","last_clean":"2024-04-19T01:04:19.557884+0000","last_became_active":"2024-04-19T01:04:03.323177+0000","last_became_peered":"2024-04-19T01:04:03.323177+0000","last_unstale":"2024-04-19T01:04:19.557884+0000","last_undegraded":"2024-04-19T01:04:19.557884+0000","last_fullsized":"2024-04-19T01:04:19.557884+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:14.771327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.029493378000000001,"stat_sum":{"num_bytes":3440640,"num_objects":5,"num_object_clones":1,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":5816,"num_write":9,"num_write_kb":3360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,7,3,13,1,8],"acting":[12,4,7,3,13,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"},{"start":"8","length":"2"}]},{"pgid":"3.9","version":"45'64","reported_seq":142,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-19T01:04:15.902226+0000","last_change":"2024-04-19T01:04:15.903771+0000","last_active":"2024-04-19T01:04:15.902226+0000","last_peered":"2024-04-19T01:04:15.902226+0000","last_clean":"2024-04-19T01:04:15.902226+0000","last_became_active":"2024-04-19T01:04:03.027372+0000","last_became_peered":"2024-04-19T01:04:03.027372+0000","last_unstale":"2024-04-19T01:04:15.902226+0000","last_undegraded":"2024-04-19T01:04:15.902226+0000","last_fullsized":"2024-04-19T01:04:15.902226+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:06.869418+0000","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0.025485052000000001,"stat_sum":{"num_bytes":9240576,"num_objects":8,"num_object_clones":2,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":50,"num_write_kb":9024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11,14,9,4,10,6],"acting":[13,11,14,9,4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"5"}]},{"pgid":"3.8","version":"49'25","reported_seq":116,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T01:04:19.534323+0000","last_change":"2024-04-19T01:04:19.549872+0000","last_active":"2024-04-19T01:04:19.534323+0000","last_peered":"2024-04-19T01:04:19.534323+0000","last_clean":"2024-04-19T01:04:19.534323+0000","last_became_active":"2024-04-19T01:04:03.237653+0000","last_became_peered":"2024-04-19T01:04:03.237653+0000","last_unstale":"2024-04-19T01:04:19.534323+0000","last_undegraded":"2024-04-19T01:04:19.534323+0000","last_fullsized":"2024-04-19T01:04:19.534323+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:16.852722+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.030904131000000001,"stat_sum":{"num_bytes":2195456,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":30,"num_write_kb":2144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,1,15,14,0,5],"acting":[11,10,1,15,14,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"},{"start":"8","length":"2"}]},{"pgid":"1.0","version":"25'157","reported_seq":256,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T01:04:19.518472+0000","last_change":"2024-04-19T01:03:50.776344+0000","last_active":"2024-04-19T01:04:19.518472+0000","last_peered":"2024-04-19T01:04:19.518472+0000","last_clean":"2024-04-19T01:04:19.518472+0000","last_became_active":"2024-04-19T01:03:50.774078+0000","last_became_peered":"2024-04-19T01:03:50.774078+0000","last_unstale":"2024-04-19T01:04:19.518472+0000","last_undegraded":"2024-04-19T01:04:19.518472+0000","last_fullsized":"2024-04-19T01:04:19.518472+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:49.580851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:49.580851+0000","last_clean_scrub_stamp":"2024-04-19T01:03:49.580851+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:07.495976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":54,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T01:04:19.518443+0000","last_change":"2024-04-19T01:03:53.766587+0000","last_active":"2024-04-19T01:04:19.518443+0000","last_peered":"2024-04-19T01:04:19.518443+0000","last_clean":"2024-04-19T01:04:19.518443+0000","last_became_active":"2024-04-19T01:03:53.765967+0000","last_became_peered":"2024-04-19T01:03:53.765967+0000","last_unstale":"2024-04-19T01:04:19.518443+0000","last_undegraded":"2024-04-19T01:04:19.518443+0000","last_fullsized":"2024-04-19T01:04:19.518443+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:18.833542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"45'14","reported_seq":108,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T01:04:19.519270+0000","last_change":"2024-04-19T01:04:19.520142+0000","last_active":"2024-04-19T01:04:19.519270+0000","last_peered":"2024-04-19T01:04:19.519270+0000","last_clean":"2024-04-19T01:04:19.519270+0000","last_became_active":"2024-04-19T01:04:03.322437+0000","last_became_peered":"2024-04-19T01:04:03.322437+0000","last_unstale":"2024-04-19T01:04:19.519270+0000","last_undegraded":"2024-04-19T01:04:19.519270+0000","last_fullsized":"2024-04-19T01:04:19.519270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:06.317438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001368743,"stat_sum":{"num_bytes":344064,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":6,"num_write_kb":336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0,4,8,3,7],"acting":[11,5,0,4,8,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"},{"start":"8","length":"2"}]},{"pgid":"2.0","version":"0'0","reported_seq":54,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T01:04:19.517419+0000","last_change":"2024-04-19T01:03:53.760075+0000","last_active":"2024-04-19T01:04:19.517419+0000","last_peered":"2024-04-19T01:04:19.517419+0000","last_clean":"2024-04-19T01:04:19.517419+0000","last_became_active":"2024-04-19T01:03:53.759326+0000","last_became_peered":"2024-04-19T01:03:53.759326+0000","last_unstale":"2024-04-19T01:04:19.517419+0000","last_undegraded":"2024-04-19T01:04:19.517419+0000","last_fullsized":"2024-04-19T01:04:19.517419+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:02.718012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"47'12","reported_seq":103,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-19T01:04:18.517899+0000","last_change":"2024-04-19T01:04:18.518427+0000","last_active":"2024-04-19T01:04:18.517899+0000","last_peered":"2024-04-19T01:04:18.517899+0000","last_clean":"2024-04-19T01:04:18.517899+0000","last_became_active":"2024-04-19T01:04:03.323727+0000","last_became_peered":"2024-04-19T01:04:03.323727+0000","last_unstale":"2024-04-19T01:04:18.517899+0000","last_undegraded":"2024-04-19T01:04:18.517899+0000","last_fullsized":"2024-04-19T01:04:18.517899+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:29.482112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011327349999999999,"stat_sum":{"num_bytes":1523712,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":2020,"num_write":27,"num_write_kb":1488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,11,9,1,7],"acting":[6,4,3,11,9,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"},{"start":"9","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":54,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T01:04:19.528833+0000","last_change":"2024-04-19T01:03:53.650434+0000","last_active":"2024-04-19T01:04:19.528833+0000","last_peered":"2024-04-19T01:04:19.528833+0000","last_clean":"2024-04-19T01:04:19.528833+0000","last_became_active":"2024-04-19T01:03:53.649606+0000","last_became_peered":"2024-04-19T01:03:53.649606+0000","last_unstale":"2024-04-19T01:04:19.528833+0000","last_undegraded":"2024-04-19T01:04:19.528833+0000","last_fullsized":"2024-04-19T01:04:19.528833+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:17.424729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"49'35","reported_seq":127,"reported_epoch":49,"state":"active+clean+snaptrim","last_fresh":"2024-04-19T01:04:19.565237+0000","last_change":"2024-04-19T01:04:19.532007+0000","last_active":"2024-04-19T01:04:19.565237+0000","last_peered":"2024-04-19T01:04:19.565237+0000","last_clean":"2024-04-19T01:04:19.565237+0000","last_became_active":"2024-04-19T01:04:03.322503+0000","last_became_peered":"2024-04-19T01:04:03.322503+0000","last_unstale":"2024-04-19T01:04:19.565237+0000","last_undegraded":"2024-04-19T01:04:19.565237+0000","last_fullsized":"2024-04-19T01:04:19.565237+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:13.454365+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.012386256,"stat_sum":{"num_bytes":3817472,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":15,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4,3,10,11,6],"acting":[1,9,4,3,10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"},{"start":"9","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":54,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T01:04:19.517767+0000","last_change":"2024-04-19T01:03:54.012138+0000","last_active":"2024-04-19T01:04:19.517767+0000","last_peered":"2024-04-19T01:04:19.517767+0000","last_clean":"2024-04-19T01:04:19.517767+0000","last_became_active":"2024-04-19T01:03:54.011460+0000","last_became_peered":"2024-04-19T01:03:54.011460+0000","last_unstale":"2024-04-19T01:04:19.517767+0000","last_undegraded":"2024-04-19T01:04:19.517767+0000","last_fullsized":"2024-04-19T01:04:19.517767+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:04:58.336612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"49'42","reported_seq":137,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T01:04:19.561330+0000","last_change":"2024-04-19T01:04:19.563217+0000","last_active":"2024-04-19T01:04:19.561330+0000","last_peered":"2024-04-19T01:04:19.561330+0000","last_clean":"2024-04-19T01:04:19.561330+0000","last_became_active":"2024-04-19T01:04:02.835716+0000","last_became_peered":"2024-04-19T01:04:02.835716+0000","last_unstale":"2024-04-19T01:04:19.561330+0000","last_undegraded":"2024-04-19T01:04:19.561330+0000","last_fullsized":"2024-04-19T01:04:19.561330+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:14.688145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032384638,"stat_sum":{"num_bytes":3833856,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":13,"num_read_kb":1110,"num_write":55,"num_write_kb":3744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2,0,1,4,12],"acting":[15,8,2,0,1,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"},{"start":"8","length":"2"}]},{"pgid":"3.5","version":"45'43","reported_seq":147,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T01:04:19.518177+0000","last_change":"2024-04-19T01:04:19.518641+0000","last_active":"2024-04-19T01:04:19.518177+0000","last_peered":"2024-04-19T01:04:19.518177+0000","last_clean":"2024-04-19T01:04:19.518177+0000","last_became_active":"2024-04-19T01:04:03.237886+0000","last_became_peered":"2024-04-19T01:04:03.237886+0000","last_unstale":"2024-04-19T01:04:19.518177+0000","last_undegraded":"2024-04-19T01:04:19.518177+0000","last_fullsized":"2024-04-19T01:04:19.518177+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:13.565446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068062999999999999,"stat_sum":{"num_bytes":3293184,"num_objects":9,"num_object_clones":3,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":30,"num_read_kb":2938,"num_write":39,"num_write_kb":4432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,14,11,10,2,5],"acting":[8,6,14,11,10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"},{"start":"8","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":52,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-19T01:04:18.517402+0000","last_change":"2024-04-19T01:03:53.716606+0000","last_active":"2024-04-19T01:04:18.517402+0000","last_peered":"2024-04-19T01:04:18.517402+0000","last_clean":"2024-04-19T01:04:18.517402+0000","last_became_active":"2024-04-19T01:03:53.716040+0000","last_became_peered":"2024-04-19T01:03:53.716040+0000","last_unstale":"2024-04-19T01:04:18.517402+0000","last_undegraded":"2024-04-19T01:04:18.517402+0000","last_fullsized":"2024-04-19T01:04:18.517402+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:01.975515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"48'16","reported_seq":111,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-19T01:04:18.536522+0000","last_change":"2024-04-19T01:04:18.536522+0000","last_active":"2024-04-19T01:04:18.536522+0000","last_peered":"2024-04-19T01:04:18.536522+0000","last_clean":"2024-04-19T01:04:18.536522+0000","last_became_active":"2024-04-19T01:04:03.322815+0000","last_became_peered":"2024-04-19T01:04:03.322815+0000","last_unstale":"2024-04-19T01:04:18.536522+0000","last_undegraded":"2024-04-19T01:04:18.536522+0000","last_fullsized":"2024-04-19T01:04:18.536522+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:12.411696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018823612,"stat_sum":{"num_bytes":753664,"num_objects":5,"num_object_clones":1,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":582,"num_write":12,"num_write_kb":1072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,13,14,3,15,1,5],"acting":[10,13,14,3,15,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"},{"start":"9","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":48,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-19T01:04:16.499015+0000","last_change":"2024-04-19T01:03:53.772709+0000","last_active":"2024-04-19T01:04:16.499015+0000","last_peered":"2024-04-19T01:04:16.499015+0000","last_clean":"2024-04-19T01:04:16.499015+0000","last_became_active":"2024-04-19T01:03:53.771887+0000","last_became_peered":"2024-04-19T01:03:53.771887+0000","last_unstale":"2024-04-19T01:04:16.499015+0000","last_undegraded":"2024-04-19T01:04:16.499015+0000","last_fullsized":"2024-04-19T01:04:16.499015+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:17.479766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"45'12","reported_seq":86,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-19T01:04:15.880263+0000","last_change":"2024-04-19T01:04:15.883168+0000","last_active":"2024-04-19T01:04:15.880263+0000","last_peered":"2024-04-19T01:04:15.880263+0000","last_clean":"2024-04-19T01:04:15.880263+0000","last_became_active":"2024-04-19T01:04:03.238517+0000","last_became_peered":"2024-04-19T01:04:03.238517+0000","last_unstale":"2024-04-19T01:04:15.880263+0000","last_undegraded":"2024-04-19T01:04:15.880263+0000","last_fullsized":"2024-04-19T01:04:15.880263+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:23.314696+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.016351998,"stat_sum":{"num_bytes":1589248,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":5,"num_write_kb":1552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,7,5,11,6,0],"acting":[14,13,7,5,11,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"5"}]},{"pgid":"2.6","version":"0'0","reported_seq":46,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-19T01:04:15.866554+0000","last_change":"2024-04-19T01:03:53.736290+0000","last_active":"2024-04-19T01:04:15.866554+0000","last_peered":"2024-04-19T01:04:15.866554+0000","last_clean":"2024-04-19T01:04:15.866554+0000","last_became_active":"2024-04-19T01:03:53.735549+0000","last_became_peered":"2024-04-19T01:03:53.735549+0000","last_unstale":"2024-04-19T01:04:15.866554+0000","last_undegraded":"2024-04-19T01:04:15.866554+0000","last_fullsized":"2024-04-19T01:04:15.866554+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:07.640918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13],"acting":[14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":73,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-19T01:04:15.866385+0000","last_change":"2024-04-19T01:04:15.867502+0000","last_active":"2024-04-19T01:04:15.866385+0000","last_peered":"2024-04-19T01:04:15.866385+0000","last_clean":"2024-04-19T01:04:15.866385+0000","last_became_active":"2024-04-19T01:04:02.889974+0000","last_became_peered":"2024-04-19T01:04:02.889974+0000","last_unstale":"2024-04-19T01:04:15.866385+0000","last_undegraded":"2024-04-19T01:04:15.866385+0000","last_fullsized":"2024-04-19T01:04:15.866385+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:17.036263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017315200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7,15,8,11,0,1],"acting":[14,7,15,8,11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"5"}]},{"pgid":"2.7","version":"0'0","reported_seq":54,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T01:04:19.517740+0000","last_change":"2024-04-19T01:03:53.939441+0000","last_active":"2024-04-19T01:04:19.517740+0000","last_peered":"2024-04-19T01:04:19.517740+0000","last_clean":"2024-04-19T01:04:19.517740+0000","last_became_active":"2024-04-19T01:03:53.938774+0000","last_became_peered":"2024-04-19T01:03:53.938774+0000","last_unstale":"2024-04-19T01:04:19.517740+0000","last_undegraded":"2024-04-19T01:04:19.517740+0000","last_fullsized":"2024-04-19T01:04:19.517740+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:14.081709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":39190528,"num_objects":66,"num_object_clones":18,"num_object_copies":462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":2,"num_read":180,"num_read_kb":20047,"num_write":321,"num_write_kb":40944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":68788224,"data_stored":82378752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":13},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":15546,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":14,"up_from":24,"seq":103079215113,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36192,"kb_used_data":5012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335648,"statfs":{"total":96636764160,"available":96599703552,"internally_reserved":0,"allocated":5132288,"data_stored":6184199,"data_compressed":43152,"data_compressed_allocated":90112,"data_compressed_original":180224,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,5,6,7,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":13,"up_from":24,"seq":103079215114,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35824,"kb_used_data":4640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336016,"statfs":{"total":96636764160,"available":96600080384,"internally_reserved":0,"allocated":4751360,"data_stored":6382790,"data_compressed":41151,"data_compressed_allocated":86016,"data_compressed_original":172032,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,4,5,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":9,"up_from":24,"seq":103079215115,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36120,"kb_used_data":4932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335720,"statfs":{"total":96636764160,"available":96599777280,"internally_reserved":0,"allocated":5050368,"data_stored":5926151,"data_compressed":43152,"data_compressed_allocated":90112,"data_compressed_original":180224,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313226,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33068,"kb_used_data":6040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338772,"statfs":{"total":96636764160,"available":96602902528,"internally_reserved":0,"allocated":6184960,"data_stored":7486833,"data_compressed":47144,"data_compressed_allocated":98304,"data_compressed_original":196608,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,3,4,5,6,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280520,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33904,"kb_used_data":6872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337936,"statfs":{"total":96636764160,"available":96602046464,"internally_reserved":0,"allocated":7036928,"data_stored":8869330,"data_compressed":49143,"data_compressed_allocated":102400,"data_compressed_original":204800,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280522,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33264,"kb_used_data":6232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338576,"statfs":{"total":96636764160,"available":96602701824,"internally_reserved":0,"allocated":6381568,"data_stored":6997458,"data_compressed":49143,"data_compressed_allocated":102400,"data_compressed_original":204800,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280521,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29944,"kb_used_data":2912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341896,"statfs":{"total":96636764160,"available":96606101504,"internally_reserved":0,"allocated":2981888,"data_stored":2897362,"data_compressed":49143,"data_compressed_allocated":102400,"data_compressed_original":204800,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[2,4,5,6,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280521,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34528,"kb_used_data":7496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337312,"statfs":{"total":96636764160,"available":96601407488,"internally_reserved":0,"allocated":7675904,"data_stored":9602514,"data_compressed":49143,"data_compressed_allocated":102400,"data_compressed_original":204800,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280521,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30944,"kb_used_data":3916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340896,"statfs":{"total":96636764160,"available":96605077504,"internally_reserved":0,"allocated":4009984,"data_stored":4892658,"data_compressed":56916,"data_compressed_allocated":135168,"data_compressed_original":716800,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,6,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280522,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30836,"kb_used_data":3804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341004,"statfs":{"total":96636764160,"available":96605188096,"internally_reserved":0,"allocated":3895296,"data_stored":4040146,"data_compressed":49143,"data_compressed_allocated":102400,"data_compressed_original":204800,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[2,5,6,7,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280520,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34840,"kb_used_data":7812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337000,"statfs":{"total":96636764160,"available":96601088000,"internally_reserved":0,"allocated":7999488,"data_stored":10557426,"data_compressed":56916,"data_compressed_allocated":135168,"data_compressed_original":716800,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,3,4,5,6,7,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280520,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30408,"kb_used_data":3376,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341432,"statfs":{"total":96636764160,"available":96605626368,"internally_reserved":0,"allocated":3457024,"data_stored":3679698,"data_compressed":49143,"data_compressed_allocated":102400,"data_compressed_original":204800,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280521,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29984,"kb_used_data":2952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341856,"statfs":{"total":96636764160,"available":96606060544,"internally_reserved":0,"allocated":3022848,"data_stored":3478994,"data_compressed":49143,"data_compressed_allocated":102400,"data_compressed_original":204800,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,4,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":24,"seq":103079215113,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35304,"kb_used_data":4116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336536,"statfs":{"total":96636764160,"available":96600612864,"internally_reserved":0,"allocated":4214784,"data_stored":4527302,"data_compressed":41151,"data_compressed_allocated":86016,"data_compressed_original":172032,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215114,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30484,"kb_used_data":3456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341356,"statfs":{"total":96636764160,"available":96605548544,"internally_reserved":0,"allocated":3538944,"data_stored":3554567,"data_compressed":43152,"data_compressed_allocated":90112,"data_compressed_original":180224,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":24,"seq":103079215113,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29280,"kb_used_data":2252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342560,"statfs":{"total":96636764160,"available":96606781440,"internally_reserved":0,"allocated":2306048,"data_stored":2178311,"data_compressed":43152,"data_compressed_allocated":90112,"data_compressed_original":180224,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3063808,"data_stored":3264512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6119424,"data_stored":6770688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3424256,"data_stored":3559424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2469888,"data_stored":2576384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":6610944,"data_stored":7983104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3588096,"data_stored":4050944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":6524928,"data_stored":8548352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1830912,"data_stored":1888256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":3112960,"data_stored":3391488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4575232,"data_stored":5636096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":5685248,"data_stored":8261632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":7401472,"data_stored":8871936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":2674688,"data_stored":3031040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":4288512,"data_stored":5378048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":4657152,"data_stored":5894144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":2760704,"data_stored":3272704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T01:04:22.221 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-19T01:04:22.222 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-19T01:04:22.222 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2024-04-19T01:04:22.222 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T01:04:22.309 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:04:22.307+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:04:22.322 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:22.322 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:22.333 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-19T01:04:22.396 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:04:22.392+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:04:22.474 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-19T01:04:22.491 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:22.491 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:22.493 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:04:22.495+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:04:22.513 INFO:tasks.rados.rados.0.smithi155.stdout:145: done (1 left) 2024-04-19T01:04:22.514 INFO:tasks.rados.rados.0.smithi155.stdout:146: rmattr oid 39 current snap is 10 2024-04-19T01:04:22.514 INFO:tasks.rados.rados.0.smithi155.stdout:146: done (1 left) 2024-04-19T01:04:22.514 INFO:tasks.rados.rados.0.smithi155.stdout:147: snap_remove snap 9 2024-04-19T01:04:22.514 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-19T01:04:22.545 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-19T01:04:22.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:04:22.583+0000 7fe0f3fff640 0 --2- 172.21.15.33:0/3001708561 >> v2:172.21.15.33:3300/0 conn(0x7fe0f414f050 0x7fe0f415c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-04-19T01:04:22.594 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:22.592+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:22.653 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-04-19T01:04:22.653 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T01:04:22.682 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:22.682 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2024-04-19T01:04:22.682 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2024-04-19T01:04:22.682 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-04-19T01:04:22.682 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.682 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:172 3.cs0 3:356739ec:::smithi15528762-4:head [create,setxattr aSrcO`NnS[qadSjpQT[PSmfh\\PfYagpctmbQ_kXi (1753) in=40b,setxattr hratdsb^VNONN^^acUkPeXNZohcdVkmXXhdTeofs (2805) in=40b,setxattr abOo``aaqamjUn`hc`Qde]XU\\lqYVrmVdcimNs_a (35) in=40b,setxattr ZheopalZpQPnebTbena`gaWXtmaSTdQkji`aZYOi (34) in=40b,setxattr katXYockjtYtieO]`jgpSrZZ\\mq`gOYplPOVal`Y (842) in=40b,setxattr [^]lbsa\\UX[na]akS\\hYcdqU^]aWe_[U[OlaQXUY (1894) in=40b,setxattr We][_Xheab\\Ylg`q^hsi`XTX_tQS`Ng\\]aqZTnV[ (1473) in=40b,setxattr TTRSo^W`XoqVWpSjRgc[adm`ROSSVYUNlsilh\\kS (911) in=40b,setxattr bqbUQROpngcT`ndhrZQP`iraoUgs[ZZatacnZoc_ (2876) in=40b,setxattr qPPSibdg_e\\Y_V`kTQO^\\`_jr^jt\\aiadkepqYUj (3576) in=40b,setxattr q`N\\bPOqejbpq`QZVQfWgjScjfahlgXWodnRReWS (1017) in=40b,setxattr `pdT]PSaeeZ^jhWYc]iO`^tRXWcjedgQdke^adpf (7928) in=40b,setxattr matSmfe^casVRfWbNqkt_mZNWbhhh`\\Waf\\djeca (7157) in=40b,setxattr ]SWX\\hrhracj[TeSRYP[bZ[[lSSRghmfarTYpp]U (735) in=40b,setxattr WrnQrVa (13964) in=7b,setxattr PcP_]XU]rhapTs\\X`ojNmm_Qsjkkjecf`VgrdTrR (1542) in=40b,setxattr o^RO^oeZlji]\\\\UhVsnb]t_WXomaRr^higjSUShQ (1016) in=40b,setxattr Z_ofVVcfZdUV[UaRnTV_Ni^_rRNl`\\kt`]aaaoed (701) in=40b,setxattr X[m^[`bVlbgi``^mWZ]cl[S_nn[Ok\\ldS\\_h]]qk (1999) in=40b,setxattr fpmQVVnVPSmZmt`OeaT`adhoNjhpOb[`\\aPWQ^Rd (700) in=40b] snapc 5=[5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-19T01:04:22.682 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:08.804031+0000", 2024-04-19T01:04:22.682 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 13.876911539, 2024-04-19T01:04:22.682 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014732528999999999, 2024-04-19T01:04:22.683 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:22.683 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:22.683 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:22.683 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:22.683 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:22.683 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:22.683 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 172 2024-04-19T01:04:22.683 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.683 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:22.683 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.684 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:22.684 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.804031+0000", 2024-04-19T01:04:22.684 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:22.684 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.684 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.684 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:22.684 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.804031+0000", 2024-04-19T01:04:22.684 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:22.684 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.684 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.684 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:22.685 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.804034+0000", 2024-04-19T01:04:22.685 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5389999999999999e-06 2024-04-19T01:04:22.685 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.685 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.685 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:22.686 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.804179+0000", 2024-04-19T01:04:22.686 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014485600000000001 2024-04-19T01:04:22.686 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.686 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.686 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:22.686 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.804208+0000", 2024-04-19T01:04:22.686 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8615999999999998e-05 2024-04-19T01:04:22.687 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.687 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.687 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:22.687 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.804223+0000", 2024-04-19T01:04:22.687 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5244e-05 2024-04-19T01:04:22.687 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.687 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.687 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:22.687 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.804362+0000", 2024-04-19T01:04:22.687 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013886000000000001 2024-04-19T01:04:22.687 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.688 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.688 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op must wait for map", 2024-04-19T01:04:22.688 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.804377+0000", 2024-04-19T01:04:22.688 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4832e-05 2024-04-19T01:04:22.688 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.688 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.688 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:22.688 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.809596+0000", 2024-04-19T01:04:22.688 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0052189250000000001 2024-04-19T01:04:22.688 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.688 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.688 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:22.689 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.809925+0000", 2024-04-19T01:04:22.689 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00032908900000000002 2024-04-19T01:04:22.689 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.689 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.689 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:22.689 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.813383+0000", 2024-04-19T01:04:22.689 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0034579929999999999 2024-04-19T01:04:22.689 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.689 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.689 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:22.689 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.815667+0000", 2024-04-19T01:04:22.689 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.002284632 2024-04-19T01:04:22.689 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.690 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.690 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:22.690 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.818639+0000", 2024-04-19T01:04:22.690 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.002972003 2024-04-19T01:04:22.690 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.690 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.690 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:22.690 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.818763+0000", 2024-04-19T01:04:22.690 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012394000000000001 2024-04-19T01:04:22.690 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:22.692 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:22.692 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:22.692 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.692 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.692 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:232 3.cs0 3:3233f066:::smithi15528762-50:2 [read 0~835584,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:22.692 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.838358+0000", 2024-04-19T01:04:22.693 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 8.8425848299999998, 2024-04-19T01:04:22.693 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011277769, 2024-04-19T01:04:22.693 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:22.693 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:22.693 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:22.693 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:22.693 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:22.693 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:22.693 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 232 2024-04-19T01:04:22.693 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.693 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:22.694 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.694 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:22.694 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838358+0000", 2024-04-19T01:04:22.694 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:22.694 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.694 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.694 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:22.694 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838358+0000", 2024-04-19T01:04:22.694 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:22.694 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.694 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.694 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:22.694 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838360+0000", 2024-04-19T01:04:22.694 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8770000000000002e-06 2024-04-19T01:04:22.695 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.695 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.695 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:22.695 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838388+0000", 2024-04-19T01:04:22.695 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7282000000000001e-05 2024-04-19T01:04:22.695 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.695 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.695 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:22.695 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838427+0000", 2024-04-19T01:04:22.695 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9242999999999997e-05 2024-04-19T01:04:22.695 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.695 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.695 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:22.695 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838444+0000", 2024-04-19T01:04:22.695 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7127000000000001e-05 2024-04-19T01:04:22.696 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.696 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.696 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:22.696 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838641+0000", 2024-04-19T01:04:22.696 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00019704499999999999 2024-04-19T01:04:22.696 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.696 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.696 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op must wait for map", 2024-04-19T01:04:22.696 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838653+0000", 2024-04-19T01:04:22.696 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2187999999999999e-05 2024-04-19T01:04:22.696 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.696 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.696 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:22.697 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.842741+0000", 2024-04-19T01:04:22.697 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0040879879999999999 2024-04-19T01:04:22.697 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.697 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.697 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:22.697 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.842898+0000", 2024-04-19T01:04:22.697 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015647299999999999 2024-04-19T01:04:22.697 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.697 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.697 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:22.698 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.849635+0000", 2024-04-19T01:04:22.698 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0067375459999999996 2024-04-19T01:04:22.698 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:22.698 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:22.698 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:22.698 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.698 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.698 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:233 3.cs0 3:3233f066:::smithi15528762-50:2 [read 0~835584,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:22.698 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.838487+0000", 2024-04-19T01:04:22.698 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 8.8424558159999993, 2024-04-19T01:04:22.698 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013873264, 2024-04-19T01:04:22.699 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:22.699 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:22.699 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:22.699 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:22.699 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:22.700 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:22.700 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 233 2024-04-19T01:04:22.700 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.700 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:22.700 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.700 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:22.700 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838487+0000", 2024-04-19T01:04:22.700 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:22.700 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.700 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.700 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:22.700 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838487+0000", 2024-04-19T01:04:22.700 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:22.700 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.700 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.701 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:22.701 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838488+0000", 2024-04-19T01:04:22.701 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0380000000000001e-06 2024-04-19T01:04:22.701 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.701 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.701 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:22.701 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838495+0000", 2024-04-19T01:04:22.701 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.8299999999999996e-06 2024-04-19T01:04:22.701 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.701 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.701 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:22.701 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838509+0000", 2024-04-19T01:04:22.701 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3954000000000001e-05 2024-04-19T01:04:22.701 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.702 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.702 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:22.702 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838519+0000", 2024-04-19T01:04:22.702 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.9729999999999992e-06 2024-04-19T01:04:22.702 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.702 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.702 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:22.702 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838755+0000", 2024-04-19T01:04:22.702 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00023589999999999999 2024-04-19T01:04:22.702 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.703 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.703 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:04:22.703 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838764+0000", 2024-04-19T01:04:22.703 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.0429999999999996e-06 2024-04-19T01:04:22.703 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.703 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.703 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:22.703 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.844563+0000", 2024-04-19T01:04:22.703 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0057983749999999997 2024-04-19T01:04:22.703 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.703 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.703 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:22.703 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.844897+0000", 2024-04-19T01:04:22.703 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00033450199999999998 2024-04-19T01:04:22.703 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.704 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.704 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:22.704 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.852360+0000", 2024-04-19T01:04:22.704 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.007462649 2024-04-19T01:04:22.704 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:22.704 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:22.704 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:22.704 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.704 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.704 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:234 3.cs0 3:3233f066:::smithi15528762-50:2 [read 0~835584,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:22.704 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.838536+0000", 2024-04-19T01:04:22.704 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 8.8424063299999993, 2024-04-19T01:04:22.704 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.016682111999999999, 2024-04-19T01:04:22.704 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:22.704 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:22.705 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:22.705 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:22.705 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:22.705 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:22.705 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 234 2024-04-19T01:04:22.705 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.705 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:22.705 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.705 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:22.705 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838536+0000", 2024-04-19T01:04:22.705 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:22.705 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.705 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.705 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:22.706 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838536+0000", 2024-04-19T01:04:22.706 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:22.706 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.706 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.706 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:22.706 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838537+0000", 2024-04-19T01:04:22.706 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1200000000000002e-07 2024-04-19T01:04:22.706 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.706 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.706 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:22.706 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838542+0000", 2024-04-19T01:04:22.706 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3789999999999997e-06 2024-04-19T01:04:22.706 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.706 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.707 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:22.707 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838553+0000", 2024-04-19T01:04:22.707 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0986e-05 2024-04-19T01:04:22.707 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.707 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.707 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:22.707 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838564+0000", 2024-04-19T01:04:22.707 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0895e-05 2024-04-19T01:04:22.707 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.707 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.707 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:22.707 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.840056+0000", 2024-04-19T01:04:22.707 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014917229999999999 2024-04-19T01:04:22.707 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.707 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.708 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:04:22.708 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.840066+0000", 2024-04-19T01:04:22.708 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.9329999999999996e-06 2024-04-19T01:04:22.708 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.708 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.708 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:22.708 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.846974+0000", 2024-04-19T01:04:22.708 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0069077770000000004 2024-04-19T01:04:22.708 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.708 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.708 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:22.708 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.847132+0000", 2024-04-19T01:04:22.708 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015876400000000001 2024-04-19T01:04:22.708 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.708 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.709 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:22.709 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.855218+0000", 2024-04-19T01:04:22.709 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0080858430000000005 2024-04-19T01:04:22.709 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:22.709 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:22.710 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:22.710 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.710 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.710 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:263 3.fs0 3:f8a60714:::smithi15528762-14:head [append 0~720896] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:22.710 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.990535+0000", 2024-04-19T01:04:22.710 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 8.6904070440000005, 2024-04-19T01:04:22.710 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.043978092000000003, 2024-04-19T01:04:22.710 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:22.710 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:22.710 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:22.710 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:22.710 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:22.710 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:22.711 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 263 2024-04-19T01:04:22.711 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.711 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:22.711 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.711 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:22.711 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.990535+0000", 2024-04-19T01:04:22.711 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:22.711 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.711 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.711 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:22.711 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.990535+0000", 2024-04-19T01:04:22.711 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:22.711 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.711 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.711 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:22.712 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.990537+0000", 2024-04-19T01:04:22.712 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4160000000000001e-06 2024-04-19T01:04:22.712 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.712 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.712 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:22.712 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991425+0000", 2024-04-19T01:04:22.712 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00088775099999999999 2024-04-19T01:04:22.712 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.712 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.712 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:22.712 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991440+0000", 2024-04-19T01:04:22.712 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5305999999999999e-05 2024-04-19T01:04:22.712 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.712 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.712 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:22.713 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991447+0000", 2024-04-19T01:04:22.713 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.4760000000000001e-06 2024-04-19T01:04:22.713 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.713 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.713 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:22.713 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991516+0000", 2024-04-19T01:04:22.713 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.9041999999999994e-05 2024-04-19T01:04:22.713 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.713 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.713 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:22.713 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991620+0000", 2024-04-19T01:04:22.713 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000103282 2024-04-19T01:04:22.713 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.713 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.714 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:22.714 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993018+0000", 2024-04-19T01:04:22.714 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001397976 2024-04-19T01:04:22.714 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.714 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.714 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:22.714 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.004440+0000", 2024-04-19T01:04:22.714 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011422261 2024-04-19T01:04:22.714 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.714 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.714 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:22.714 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.034454+0000", 2024-04-19T01:04:22.714 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.030013782999999999 2024-04-19T01:04:22.714 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.715 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.715 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:22.715 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.034513+0000", 2024-04-19T01:04:22.715 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.9799000000000001e-05 2024-04-19T01:04:22.715 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:22.715 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:22.715 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:22.715 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.715 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.715 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:264 3.fs0 3:f8a60714:::smithi15528762-14:head [append 0~376832] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:22.716 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.992666+0000", 2024-04-19T01:04:22.716 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 8.6882762719999995, 2024-04-19T01:04:22.716 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.041981560000000001, 2024-04-19T01:04:22.716 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:22.716 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:22.716 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:22.716 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:22.716 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:22.716 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:22.716 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 264 2024-04-19T01:04:22.716 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.716 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:22.716 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.716 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:22.716 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.992666+0000", 2024-04-19T01:04:22.717 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:22.717 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.717 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.717 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:22.717 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.992666+0000", 2024-04-19T01:04:22.717 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:22.717 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.717 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.717 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:22.717 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.992668+0000", 2024-04-19T01:04:22.717 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.483e-06 2024-04-19T01:04:22.717 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.717 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.717 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:22.718 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993021+0000", 2024-04-19T01:04:22.718 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000352958 2024-04-19T01:04:22.718 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.718 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.718 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:22.718 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993031+0000", 2024-04-19T01:04:22.718 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0723000000000001e-05 2024-04-19T01:04:22.718 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.718 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.718 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:22.718 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993040+0000", 2024-04-19T01:04:22.718 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.507e-06 2024-04-19T01:04:22.718 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.718 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.719 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:22.719 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.000363+0000", 2024-04-19T01:04:22.719 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0073236819999999998 2024-04-19T01:04:22.719 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.719 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.719 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:22.719 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.000482+0000", 2024-04-19T01:04:22.719 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011817299999999999 2024-04-19T01:04:22.719 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.719 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.720 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:22.720 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.001332+0000", 2024-04-19T01:04:22.720 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000849886 2024-04-19T01:04:22.720 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.720 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.720 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:22.720 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.006292+0000", 2024-04-19T01:04:22.720 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0049605090000000001 2024-04-19T01:04:22.720 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.720 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.721 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:22.721 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.034602+0000", 2024-04-19T01:04:22.721 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.028310076999999999 2024-04-19T01:04:22.721 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.721 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.721 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:22.721 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.034648+0000", 2024-04-19T01:04:22.721 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5562000000000002e-05 2024-04-19T01:04:22.721 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:22.721 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:22.721 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:22.721 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.721 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.721 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:265 3.fs0 3:f8a60714:::smithi15528762-14:head [setxattr _header (56) in=7b] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:22.722 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.993067+0000", 2024-04-19T01:04:22.722 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 8.6878750339999993, 2024-04-19T01:04:22.722 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.041773988999999997, 2024-04-19T01:04:22.722 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:22.722 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:22.722 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:22.722 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:22.722 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:22.722 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:22.722 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 265 2024-04-19T01:04:22.722 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.722 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:22.722 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.722 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:22.723 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993067+0000", 2024-04-19T01:04:22.723 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:22.723 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.723 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.723 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:22.723 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993067+0000", 2024-04-19T01:04:22.723 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:22.723 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.723 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.723 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:22.723 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993068+0000", 2024-04-19T01:04:22.723 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2699999999999999e-07 2024-04-19T01:04:22.723 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.723 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.723 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:22.724 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993072+0000", 2024-04-19T01:04:22.724 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0539999999999996e-06 2024-04-19T01:04:22.724 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.724 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.724 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:22.724 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993080+0000", 2024-04-19T01:04:22.724 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6140000000000003e-06 2024-04-19T01:04:22.724 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.724 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.724 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:22.724 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993119+0000", 2024-04-19T01:04:22.724 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8754000000000002e-05 2024-04-19T01:04:22.724 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.724 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.724 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:22.725 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.004460+0000", 2024-04-19T01:04:22.725 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011341188 2024-04-19T01:04:22.725 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.725 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.725 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:22.725 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.004518+0000", 2024-04-19T01:04:22.725 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8306e-05 2024-04-19T01:04:22.725 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.725 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.725 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:22.725 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.004774+0000", 2024-04-19T01:04:22.725 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00025565400000000002 2024-04-19T01:04:22.725 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.725 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.725 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:22.726 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.006318+0000", 2024-04-19T01:04:22.726 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0015444409999999999 2024-04-19T01:04:22.726 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.726 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.726 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:22.726 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.034787+0000", 2024-04-19T01:04:22.726 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.028468566000000001 2024-04-19T01:04:22.726 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.726 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.726 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:22.726 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.034841+0000", 2024-04-19T01:04:22.726 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4685000000000002e-05 2024-04-19T01:04:22.726 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:22.726 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:22.727 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:22.727 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.727 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.727 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:266 3.fs0 3:f8a60714:::smithi15528762-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:22.727 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.993144+0000", 2024-04-19T01:04:22.727 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 8.6877979809999992, 2024-04-19T01:04:22.727 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.042235719999999997, 2024-04-19T01:04:22.727 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:22.727 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:22.727 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:22.727 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:22.727 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:22.727 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:22.727 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 266 2024-04-19T01:04:22.728 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.728 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:22.728 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.728 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:22.728 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993144+0000", 2024-04-19T01:04:22.728 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:22.728 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.728 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.728 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:22.728 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993144+0000", 2024-04-19T01:04:22.728 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:22.728 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.728 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.728 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:22.729 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993145+0000", 2024-04-19T01:04:22.729 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5099999999999999e-07 2024-04-19T01:04:22.729 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.729 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.729 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:22.729 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993149+0000", 2024-04-19T01:04:22.729 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4539999999999999e-06 2024-04-19T01:04:22.729 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.729 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.729 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:22.729 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993157+0000", 2024-04-19T01:04:22.729 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.6349999999999995e-06 2024-04-19T01:04:22.729 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.729 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.729 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:22.730 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993166+0000", 2024-04-19T01:04:22.730 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2709999999999998e-06 2024-04-19T01:04:22.730 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.730 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.730 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:22.731 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.004924+0000", 2024-04-19T01:04:22.731 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011757834 2024-04-19T01:04:22.731 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.731 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.731 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:04:22.731 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.004971+0000", 2024-04-19T01:04:22.731 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6947000000000003e-05 2024-04-19T01:04:22.731 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.731 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.731 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:22.731 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.035366+0000", 2024-04-19T01:04:22.732 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.03039501 2024-04-19T01:04:22.732 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.732 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.732 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:22.732 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.035380+0000", 2024-04-19T01:04:22.732 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3918e-05 2024-04-19T01:04:22.732 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:22.732 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:22.732 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:22.732 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.732 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.732 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:317 3.fs0 3:f23900a8:::smithi15528762-6:head [append 0~475136] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:04:22.732 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.601883+0000", 2024-04-19T01:04:22.732 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.0790590030000002, 2024-04-19T01:04:22.733 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.029134103000000001, 2024-04-19T01:04:22.733 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:22.733 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:22.733 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:22.733 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:22.733 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:22.733 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:22.733 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 317 2024-04-19T01:04:22.733 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.733 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:22.733 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.733 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:22.733 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.601883+0000", 2024-04-19T01:04:22.733 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:22.733 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.734 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.734 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:22.734 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.601883+0000", 2024-04-19T01:04:22.734 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:22.734 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.734 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.734 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:22.734 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.601886+0000", 2024-04-19T01:04:22.734 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.2709999999999999e-06 2024-04-19T01:04:22.734 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.734 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.734 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:22.734 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.602696+0000", 2024-04-19T01:04:22.734 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00080995700000000004 2024-04-19T01:04:22.735 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.735 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.735 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:22.735 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.602714+0000", 2024-04-19T01:04:22.735 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8168999999999998e-05 2024-04-19T01:04:22.735 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.735 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.735 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:22.735 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.602721+0000", 2024-04-19T01:04:22.735 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.554e-06 2024-04-19T01:04:22.735 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.735 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.735 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:22.735 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.602799+0000", 2024-04-19T01:04:22.736 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7804000000000002e-05 2024-04-19T01:04:22.736 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.736 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.736 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:22.736 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.602920+0000", 2024-04-19T01:04:22.736 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012105299999999999 2024-04-19T01:04:22.736 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.736 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.736 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:22.736 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.604215+0000", 2024-04-19T01:04:22.736 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0012944129999999999 2024-04-19T01:04:22.736 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.737 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.737 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:22.737 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.609630+0000", 2024-04-19T01:04:22.737 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0054153789999999997 2024-04-19T01:04:22.737 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.737 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.737 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:22.737 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.630924+0000", 2024-04-19T01:04:22.737 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021294306999999998 2024-04-19T01:04:22.737 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.737 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.737 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:22.737 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.631018+0000", 2024-04-19T01:04:22.737 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.3195999999999993e-05 2024-04-19T01:04:22.737 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:22.738 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:22.738 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:22.738 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.738 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.738 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:318 3.fs0 3:f23900a8:::smithi15528762-6:head [append 0~622592] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:04:22.738 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.609539+0000", 2024-04-19T01:04:22.738 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.0714034090000002, 2024-04-19T01:04:22.738 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021603679000000001, 2024-04-19T01:04:22.738 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:22.738 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:22.738 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:22.738 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:22.738 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:22.738 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:22.739 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 318 2024-04-19T01:04:22.739 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.739 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:22.739 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.739 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:22.739 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.609539+0000", 2024-04-19T01:04:22.739 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:22.739 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.739 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.739 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:22.739 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.609539+0000", 2024-04-19T01:04:22.740 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:22.740 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.740 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.740 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:22.740 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.609541+0000", 2024-04-19T01:04:22.741 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.855e-06 2024-04-19T01:04:22.741 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.741 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.741 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:22.741 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610203+0000", 2024-04-19T01:04:22.741 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00066190200000000002 2024-04-19T01:04:22.741 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.741 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.741 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:22.741 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610215+0000", 2024-04-19T01:04:22.741 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2143e-05 2024-04-19T01:04:22.741 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.741 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.741 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:22.742 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610221+0000", 2024-04-19T01:04:22.742 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4590000000000003e-06 2024-04-19T01:04:22.742 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.742 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.742 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:22.742 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610283+0000", 2024-04-19T01:04:22.742 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1322999999999995e-05 2024-04-19T01:04:22.742 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.742 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.742 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:22.742 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610394+0000", 2024-04-19T01:04:22.742 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000111044 2024-04-19T01:04:22.742 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.742 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.743 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:22.743 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611327+0000", 2024-04-19T01:04:22.743 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00093369799999999995 2024-04-19T01:04:22.743 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.743 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.743 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:22.743 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.623095+0000", 2024-04-19T01:04:22.743 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011767454 2024-04-19T01:04:22.743 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.743 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.743 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:22.743 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.631101+0000", 2024-04-19T01:04:22.743 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0080059889999999998 2024-04-19T01:04:22.743 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.744 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.744 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:22.744 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.631143+0000", 2024-04-19T01:04:22.744 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1811999999999999e-05 2024-04-19T01:04:22.744 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:22.744 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:22.744 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:22.744 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.744 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.744 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 47/31 ECSubWriteReply(tid=61, last_complete=47'16, committed=1, applied=1))", 2024-04-19T01:04:22.744 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.610333+0000", 2024-04-19T01:04:22.744 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.0706091759999996, 2024-04-19T01:04:22.744 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0095713929999999992, 2024-04-19T01:04:22.744 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:22.744 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:22.745 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:22.745 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:22.745 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.745 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:22.745 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610333+0000", 2024-04-19T01:04:22.745 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:22.745 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.745 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.745 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:22.745 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610333+0000", 2024-04-19T01:04:22.745 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:22.745 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.745 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.745 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:22.746 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610335+0000", 2024-04-19T01:04:22.746 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.688e-06 2024-04-19T01:04:22.746 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.746 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.746 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:22.746 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610345+0000", 2024-04-19T01:04:22.746 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0505e-05 2024-04-19T01:04:22.746 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.746 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.746 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:22.746 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610372+0000", 2024-04-19T01:04:22.746 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7053e-05 2024-04-19T01:04:22.746 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.746 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.746 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:22.747 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610391+0000", 2024-04-19T01:04:22.747 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8694000000000001e-05 2024-04-19T01:04:22.747 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.747 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.747 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:22.747 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.619882+0000", 2024-04-19T01:04:22.747 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0094910199999999993 2024-04-19T01:04:22.747 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.747 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.747 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:22.747 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.619905+0000", 2024-04-19T01:04:22.747 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.2433e-05 2024-04-19T01:04:22.747 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:22.747 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:22.747 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:22.748 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.748 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.748 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 47/31 ECSubWriteReply(tid=61, last_complete=47'16, committed=1, applied=1))", 2024-04-19T01:04:22.748 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.610648+0000", 2024-04-19T01:04:22.748 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.070294584, 2024-04-19T01:04:22.748 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0093177699999999995, 2024-04-19T01:04:22.748 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:22.748 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:22.748 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:22.748 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:22.749 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.749 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:22.749 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610648+0000", 2024-04-19T01:04:22.749 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:22.749 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.749 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.749 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:22.749 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610648+0000", 2024-04-19T01:04:22.749 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:22.749 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.749 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.749 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:22.749 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610649+0000", 2024-04-19T01:04:22.749 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1100000000000005e-07 2024-04-19T01:04:22.750 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.750 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.750 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:22.750 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610657+0000", 2024-04-19T01:04:22.750 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.3839999999999997e-06 2024-04-19T01:04:22.750 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.750 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.750 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:22.751 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610668+0000", 2024-04-19T01:04:22.751 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1229e-05 2024-04-19T01:04:22.751 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.751 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.751 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:22.751 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610676+0000", 2024-04-19T01:04:22.752 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7700000000000001e-06 2024-04-19T01:04:22.752 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.752 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.752 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:22.752 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.619945+0000", 2024-04-19T01:04:22.752 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0092685019999999996 2024-04-19T01:04:22.752 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.752 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.752 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:22.752 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.619966+0000", 2024-04-19T01:04:22.753 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1073999999999998e-05 2024-04-19T01:04:22.753 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:22.753 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:22.753 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:22.753 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.753 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.753 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:319 3.fs0 3:f23900a8:::smithi15528762-6:head [append 0~147456] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:04:22.753 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.611402+0000", 2024-04-19T01:04:22.753 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.0695404460000004, 2024-04-19T01:04:22.753 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.020301099, 2024-04-19T01:04:22.753 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:22.753 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:22.753 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:22.753 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:22.754 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:22.754 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:22.754 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 319 2024-04-19T01:04:22.754 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.754 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:22.754 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.754 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:22.754 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611402+0000", 2024-04-19T01:04:22.754 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:22.754 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.754 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.754 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:22.754 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611402+0000", 2024-04-19T01:04:22.754 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:22.755 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.755 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.755 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:22.755 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611404+0000", 2024-04-19T01:04:22.755 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.52e-06 2024-04-19T01:04:22.755 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.755 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.755 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:22.755 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611632+0000", 2024-04-19T01:04:22.755 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000227526 2024-04-19T01:04:22.755 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.755 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.755 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:22.755 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611660+0000", 2024-04-19T01:04:22.755 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7841999999999999e-05 2024-04-19T01:04:22.756 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.756 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.756 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:22.756 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611678+0000", 2024-04-19T01:04:22.756 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8479e-05 2024-04-19T01:04:22.756 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.756 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.756 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:22.756 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.620224+0000", 2024-04-19T01:04:22.756 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0085460999999999992 2024-04-19T01:04:22.756 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.756 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.756 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:22.756 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.620334+0000", 2024-04-19T01:04:22.756 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010964 2024-04-19T01:04:22.757 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.757 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.757 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:22.757 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.620982+0000", 2024-04-19T01:04:22.757 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00064788699999999999 2024-04-19T01:04:22.757 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.757 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.757 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:22.757 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.623230+0000", 2024-04-19T01:04:22.757 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0022481950000000001 2024-04-19T01:04:22.757 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.757 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.757 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:22.757 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.631664+0000", 2024-04-19T01:04:22.757 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0084339810000000001 2024-04-19T01:04:22.758 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.758 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.758 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:22.758 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.631703+0000", 2024-04-19T01:04:22.758 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8928999999999999e-05 2024-04-19T01:04:22.758 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:22.758 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:22.758 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:22.758 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.758 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.758 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:320 3.fs0 3:f23900a8:::smithi15528762-6:head [setxattr _header (55) in=7b] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:04:22.758 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.611709+0000", 2024-04-19T01:04:22.758 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.0692329520000001, 2024-04-19T01:04:22.758 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.020427773, 2024-04-19T01:04:22.759 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:22.759 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:22.759 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:22.759 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:22.759 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:22.759 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:22.759 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 320 2024-04-19T01:04:22.759 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.759 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:22.759 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.759 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:22.759 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611709+0000", 2024-04-19T01:04:22.759 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:22.759 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.759 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.760 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:22.760 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611709+0000", 2024-04-19T01:04:22.760 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:22.760 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.760 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.760 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:22.760 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611711+0000", 2024-04-19T01:04:22.760 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.451e-06 2024-04-19T01:04:22.760 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.760 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.760 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:22.761 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611718+0000", 2024-04-19T01:04:22.761 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5800000000000003e-06 2024-04-19T01:04:22.761 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.761 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.761 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:22.761 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611736+0000", 2024-04-19T01:04:22.761 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7339000000000001e-05 2024-04-19T01:04:22.761 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.761 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.761 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:22.761 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611749+0000", 2024-04-19T01:04:22.761 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3557000000000001e-05 2024-04-19T01:04:22.762 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.762 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.762 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:22.762 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.622598+0000", 2024-04-19T01:04:22.762 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010848315000000001 2024-04-19T01:04:22.762 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.762 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.762 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:22.762 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.622654+0000", 2024-04-19T01:04:22.762 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6311999999999999e-05 2024-04-19T01:04:22.762 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.762 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.762 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:22.762 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.622895+0000", 2024-04-19T01:04:22.762 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00024134300000000001 2024-04-19T01:04:22.763 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.763 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.763 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:22.763 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.623782+0000", 2024-04-19T01:04:22.763 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00088635099999999996 2024-04-19T01:04:22.763 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.763 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.763 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:22.763 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.632093+0000", 2024-04-19T01:04:22.763 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.008311466 2024-04-19T01:04:22.763 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.763 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.763 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:22.763 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.632137+0000", 2024-04-19T01:04:22.764 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4058999999999999e-05 2024-04-19T01:04:22.764 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:22.764 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:22.764 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:22.764 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.764 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.764 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:321 3.fs0 3:f23900a8:::smithi15528762-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:04:22.764 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.611769+0000", 2024-04-19T01:04:22.764 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.0691734149999998, 2024-04-19T01:04:22.764 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021684730999999999, 2024-04-19T01:04:22.764 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:22.764 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:22.764 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:22.764 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:22.764 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:22.765 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:22.765 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 321 2024-04-19T01:04:22.765 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.765 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:22.765 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.765 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:22.765 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611769+0000", 2024-04-19T01:04:22.765 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:22.765 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.765 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.765 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:22.765 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611769+0000", 2024-04-19T01:04:22.765 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:22.766 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.766 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.766 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:22.766 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611770+0000", 2024-04-19T01:04:22.766 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.04e-06 2024-04-19T01:04:22.766 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.766 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.766 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:22.766 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611775+0000", 2024-04-19T01:04:22.766 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3310000000000003e-06 2024-04-19T01:04:22.766 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.766 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.766 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:22.766 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611793+0000", 2024-04-19T01:04:22.767 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7408e-05 2024-04-19T01:04:22.767 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.767 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.767 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:22.767 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611808+0000", 2024-04-19T01:04:22.767 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5259000000000001e-05 2024-04-19T01:04:22.767 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.767 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.767 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:22.767 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.623039+0000", 2024-04-19T01:04:22.767 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011230756 2024-04-19T01:04:22.767 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.767 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.767 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:04:22.767 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.623089+0000", 2024-04-19T01:04:22.768 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0078999999999999e-05 2024-04-19T01:04:22.768 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.768 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.768 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:22.768 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.632385+0000", 2024-04-19T01:04:22.768 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0092956649999999998 2024-04-19T01:04:22.768 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.768 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.768 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:22.768 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.632471+0000", 2024-04-19T01:04:22.768 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.6693999999999994e-05 2024-04-19T01:04:22.768 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.768 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.768 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:22.768 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.633454+0000", 2024-04-19T01:04:22.769 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00098249899999999996 2024-04-19T01:04:22.769 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:22.769 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:22.769 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:22.769 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.769 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.769 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:345 3.fs0 3:f23900a8:::smithi15528762-6:head [append 0~557056] snapc 9=[8,7] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:04:22.769 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.534632+0000", 2024-04-19T01:04:22.769 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 4.1463105909999998, 2024-04-19T01:04:22.769 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.044819185999999997, 2024-04-19T01:04:22.769 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:22.769 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:22.769 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:22.769 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:22.769 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:22.770 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:22.770 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 345 2024-04-19T01:04:22.770 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.770 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:22.770 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.770 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:22.770 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.534632+0000", 2024-04-19T01:04:22.770 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:22.770 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.770 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.770 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:22.770 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.534632+0000", 2024-04-19T01:04:22.770 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:22.770 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.771 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.771 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:22.771 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.534634+0000", 2024-04-19T01:04:22.771 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4099999999999998e-06 2024-04-19T01:04:22.771 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.771 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.771 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:22.771 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.535690+0000", 2024-04-19T01:04:22.771 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010557260000000001 2024-04-19T01:04:22.771 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.771 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.771 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:22.771 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.535710+0000", 2024-04-19T01:04:22.771 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0339e-05 2024-04-19T01:04:22.771 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.772 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.772 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:22.772 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.535720+0000", 2024-04-19T01:04:22.772 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7599999999999997e-06 2024-04-19T01:04:22.772 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.772 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.772 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:22.772 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.535817+0000", 2024-04-19T01:04:22.772 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7218000000000003e-05 2024-04-19T01:04:22.772 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.772 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.772 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:22.772 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.536037+0000", 2024-04-19T01:04:22.772 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00021956699999999999 2024-04-19T01:04:22.773 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.773 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.773 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:22.773 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.537439+0000", 2024-04-19T01:04:22.773 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014022240000000001 2024-04-19T01:04:22.773 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.773 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.773 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:22.773 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.557535+0000", 2024-04-19T01:04:22.773 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.020095874999999999 2024-04-19T01:04:22.773 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.773 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.773 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:22.773 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.579388+0000", 2024-04-19T01:04:22.773 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021852772999999999 2024-04-19T01:04:22.774 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.774 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.774 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:22.774 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.579451+0000", 2024-04-19T01:04:22.774 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3293999999999994e-05 2024-04-19T01:04:22.774 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:22.774 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:22.774 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:22.774 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.774 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.774 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:346 3.fs0 3:f23900a8:::smithi15528762-6:head [append 0~671744] snapc 9=[8,7] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:04:22.774 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.540969+0000", 2024-04-19T01:04:22.774 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 4.1399730630000002, 2024-04-19T01:04:22.774 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.039266644000000003, 2024-04-19T01:04:22.775 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:22.775 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:22.775 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:22.775 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:22.775 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:22.775 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:22.775 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 346 2024-04-19T01:04:22.775 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.775 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:22.775 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.775 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:22.775 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.540969+0000", 2024-04-19T01:04:22.775 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:22.775 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.775 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.776 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:22.776 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.540969+0000", 2024-04-19T01:04:22.776 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:22.776 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.776 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.776 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:22.776 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.540971+0000", 2024-04-19T01:04:22.776 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9039999999999999e-06 2024-04-19T01:04:22.776 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.776 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.776 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:22.776 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542173+0000", 2024-04-19T01:04:22.776 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00120192 2024-04-19T01:04:22.776 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.777 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.777 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:22.777 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542196+0000", 2024-04-19T01:04:22.777 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3159e-05 2024-04-19T01:04:22.777 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.777 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.777 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:22.777 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542210+0000", 2024-04-19T01:04:22.777 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3899e-05 2024-04-19T01:04:22.777 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.777 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.777 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:22.777 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.546462+0000", 2024-04-19T01:04:22.777 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0042514629999999996 2024-04-19T01:04:22.777 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.778 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.778 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:22.778 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.546596+0000", 2024-04-19T01:04:22.778 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013463900000000001 2024-04-19T01:04:22.778 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.778 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.778 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:22.778 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.547898+0000", 2024-04-19T01:04:22.778 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0013016239999999999 2024-04-19T01:04:22.778 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.778 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.778 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:22.778 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.558500+0000", 2024-04-19T01:04:22.778 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010601928 2024-04-19T01:04:22.779 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.779 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.779 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:22.779 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.580189+0000", 2024-04-19T01:04:22.779 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021688796 2024-04-19T01:04:22.779 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.779 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.779 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:22.779 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.580236+0000", 2024-04-19T01:04:22.779 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7311999999999997e-05 2024-04-19T01:04:22.779 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:22.779 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:22.779 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:22.779 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.779 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.780 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:347 3.fs0 3:f23900a8:::smithi15528762-6:head [append 0~98304] snapc 9=[8,7] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:04:22.780 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.542237+0000", 2024-04-19T01:04:22.780 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 4.138705506, 2024-04-19T01:04:22.780 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.038983274999999998, 2024-04-19T01:04:22.780 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:22.780 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:22.780 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:22.780 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:22.780 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:22.780 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:22.780 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 347 2024-04-19T01:04:22.780 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.780 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:22.780 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.780 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:22.781 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542237+0000", 2024-04-19T01:04:22.781 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:22.781 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.781 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.781 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:22.781 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542237+0000", 2024-04-19T01:04:22.781 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:22.781 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.781 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.781 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:22.781 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542238+0000", 2024-04-19T01:04:22.781 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3659999999999999e-06 2024-04-19T01:04:22.781 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.781 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.782 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:22.782 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542318+0000", 2024-04-19T01:04:22.782 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.0158000000000001e-05 2024-04-19T01:04:22.782 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.782 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.782 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:22.782 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542336+0000", 2024-04-19T01:04:22.782 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7836999999999999e-05 2024-04-19T01:04:22.782 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.782 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.782 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:22.782 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542348+0000", 2024-04-19T01:04:22.782 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1646e-05 2024-04-19T01:04:22.783 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.783 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.783 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:22.783 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.553869+0000", 2024-04-19T01:04:22.783 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011521164 2024-04-19T01:04:22.783 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.783 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.783 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:22.783 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.554020+0000", 2024-04-19T01:04:22.783 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015064200000000001 2024-04-19T01:04:22.783 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.783 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.783 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:22.783 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.554913+0000", 2024-04-19T01:04:22.783 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00089314099999999997 2024-04-19T01:04:22.783 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.784 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.784 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:22.784 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.558556+0000", 2024-04-19T01:04:22.784 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003642803 2024-04-19T01:04:22.784 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.784 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.784 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:22.784 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.581179+0000", 2024-04-19T01:04:22.784 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.022623306999999999 2024-04-19T01:04:22.784 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.784 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.784 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:22.784 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.581220+0000", 2024-04-19T01:04:22.784 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1211000000000002e-05 2024-04-19T01:04:22.785 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:22.785 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:22.785 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:22.785 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.785 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.785 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:348 3.fs0 3:f23900a8:::smithi15528762-6:head [setxattr _header (55) in=7b] snapc 9=[8,7] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:04:22.785 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.542369+0000", 2024-04-19T01:04:22.786 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 4.1385736460000002, 2024-04-19T01:04:22.786 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.039646576000000003, 2024-04-19T01:04:22.786 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:22.786 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:22.786 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:22.786 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:22.786 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:22.786 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:22.786 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 348 2024-04-19T01:04:22.786 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.786 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:22.786 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.786 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:22.786 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542369+0000", 2024-04-19T01:04:22.787 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:22.787 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.787 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.787 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:22.787 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542369+0000", 2024-04-19T01:04:22.787 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:22.787 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.787 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.787 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:22.787 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542370+0000", 2024-04-19T01:04:22.787 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.384e-06 2024-04-19T01:04:22.787 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.787 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.787 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:22.787 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542376+0000", 2024-04-19T01:04:22.788 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.806e-06 2024-04-19T01:04:22.788 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.788 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.788 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:22.788 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542400+0000", 2024-04-19T01:04:22.788 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4218000000000001e-05 2024-04-19T01:04:22.788 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.788 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.788 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:22.788 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542411+0000", 2024-04-19T01:04:22.788 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0801999999999999e-05 2024-04-19T01:04:22.789 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.789 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.789 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:22.789 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.557611+0000", 2024-04-19T01:04:22.789 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015199724 2024-04-19T01:04:22.789 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.789 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.789 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:22.789 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.557731+0000", 2024-04-19T01:04:22.789 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012045900000000001 2024-04-19T01:04:22.789 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.789 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.789 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:22.789 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.558229+0000", 2024-04-19T01:04:22.790 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00049771099999999997 2024-04-19T01:04:22.790 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.790 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.790 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:22.790 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.559080+0000", 2024-04-19T01:04:22.790 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00085117400000000001 2024-04-19T01:04:22.790 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.790 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.790 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:22.790 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.581912+0000", 2024-04-19T01:04:22.790 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.022832086000000001 2024-04-19T01:04:22.791 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.791 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.791 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:22.791 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.582015+0000", 2024-04-19T01:04:22.791 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000103212 2024-04-19T01:04:22.791 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:22.791 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:22.791 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:22.791 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.791 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.791 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:349 3.fs0 3:f23900a8:::smithi15528762-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:04:22.791 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.542427+0000", 2024-04-19T01:04:22.791 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 4.1385154750000002, 2024-04-19T01:04:22.791 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.041128310000000001, 2024-04-19T01:04:22.792 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:22.792 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:22.792 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:22.792 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:22.792 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:22.792 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:22.792 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 349 2024-04-19T01:04:22.792 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.792 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:22.792 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.792 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:22.792 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542427+0000", 2024-04-19T01:04:22.792 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:22.792 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.792 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.793 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:22.793 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542427+0000", 2024-04-19T01:04:22.793 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:22.793 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.793 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.793 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:22.793 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542428+0000", 2024-04-19T01:04:22.793 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.4e-07 2024-04-19T01:04:22.793 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.793 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.793 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:22.793 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542432+0000", 2024-04-19T01:04:22.793 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5499999999999996e-06 2024-04-19T01:04:22.793 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.793 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.794 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:22.794 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542446+0000", 2024-04-19T01:04:22.794 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3365e-05 2024-04-19T01:04:22.794 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.794 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.794 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:22.794 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542456+0000", 2024-04-19T01:04:22.794 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.029e-05 2024-04-19T01:04:22.794 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.794 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.794 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:22.794 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.558587+0000", 2024-04-19T01:04:22.794 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.016131106999999999 2024-04-19T01:04:22.794 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.794 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.795 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:04:22.795 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.558676+0000", 2024-04-19T01:04:22.795 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.8823e-05 2024-04-19T01:04:22.795 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.795 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.795 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:22.795 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.582441+0000", 2024-04-19T01:04:22.795 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.023764864 2024-04-19T01:04:22.795 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.795 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.795 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:22.796 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.582493+0000", 2024-04-19T01:04:22.796 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2130999999999999e-05 2024-04-19T01:04:22.796 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:22.796 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:22.796 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:22.796 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.583555+0000", 2024-04-19T01:04:22.796 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010623399999999999 2024-04-19T01:04:22.796 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:22.796 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:22.796 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:22.796 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:22.796 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:22.796 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:22.797 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-04-19T01:04:22.798 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T01:04:22.799 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-19T01:04:22.799 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-04-19T01:04:22.799 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T01:04:22.800 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-04-19T01:04:22.800 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T01:04:22.800 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-04-19T01:04:22.800 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T01:04:22.801 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-04-19T01:04:22.801 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T01:04:22.801 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-04-19T01:04:22.801 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T01:04:22.802 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:04:22.797+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:04:22.818 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:04:22.819 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-04-19T01:04:22.819 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-04-19T01:04:22.820 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T01:04:22.836 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2024-04-19T01:04:22.837 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T01:04:22.838 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":54,"stamp":"2024-04-19T01:04:22.505369+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":41222688,"num_objects":73,"num_object_clones":23,"num_object_copies":501,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":3,"num_read":319,"num_read_kb":20168,"num_write":564,"num_write_kb":48116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":604,"ondisk_log_size":604,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":532732,"kb_used_data":79488,"kb_used_omap":24,"kb_used_meta":452903,"kb_avail":1509416708,"statfs":{"total":1546188226560,"available":1545642708992,"internally_reserved":0,"allocated":81395712,"data_stored":95271969,"data_compressed":819815,"data_compressed_allocated":1740800,"data_compressed_original":4374528,"omap_allocated":25520,"internal_metadata":463772752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5488640,"num_objects":6,"num_object_clones":7,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":25,"num_read_kb":5529,"num_write":159,"num_write_kb":9968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.998966"},"pg_stats":[{"pgid":"3.f","version":"49'27","reported_seq":131,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-19T01:04:21.518293+0000","last_change":"2024-04-19T01:04:19.544581+0000","last_active":"2024-04-19T01:04:21.518293+0000","last_peered":"2024-04-19T01:04:21.518293+0000","last_clean":"2024-04-19T01:04:21.518293+0000","last_became_active":"2024-04-19T01:04:03.238651+0000","last_became_peered":"2024-04-19T01:04:03.238651+0000","last_unstale":"2024-04-19T01:04:21.518293+0000","last_undegraded":"2024-04-19T01:04:21.518293+0000","last_fullsized":"2024-04-19T01:04:21.518293+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:09.633104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.020118509,"stat_sum":{"num_bytes":4472832,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":13,"num_read_kb":1399,"num_write":17,"num_write_kb":4368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,5,11,7,1,2],"acting":[0,13,5,11,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"},{"start":"8","length":"2"}]},{"pgid":"3.e","version":"49'28","reported_seq":132,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-19T01:04:21.535518+0000","last_change":"2024-04-19T01:04:19.553986+0000","last_active":"2024-04-19T01:04:21.535518+0000","last_peered":"2024-04-19T01:04:21.535518+0000","last_clean":"2024-04-19T01:04:21.535518+0000","last_became_active":"2024-04-19T01:04:03.238386+0000","last_became_peered":"2024-04-19T01:04:03.238386+0000","last_unstale":"2024-04-19T01:04:21.535518+0000","last_undegraded":"2024-04-19T01:04:21.535518+0000","last_fullsized":"2024-04-19T01:04:21.535518+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:28.023622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.019943764999999999,"stat_sum":{"num_bytes":1703936,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":3701,"num_write":13,"num_write_kb":2864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11,5,4,2,9,15],"acting":[13,11,5,4,2,9,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"},{"start":"8","length":"2"}]},{"pgid":"3.d","version":"49'43","reported_seq":133,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T01:04:19.547108+0000","last_change":"2024-04-19T01:04:19.548488+0000","last_active":"2024-04-19T01:04:19.547108+0000","last_peered":"2024-04-19T01:04:19.547108+0000","last_clean":"2024-04-19T01:04:19.547108+0000","last_became_active":"2024-04-19T01:04:03.025363+0000","last_became_peered":"2024-04-19T01:04:03.025363+0000","last_unstale":"2024-04-19T01:04:19.547108+0000","last_undegraded":"2024-04-19T01:04:19.547108+0000","last_fullsized":"2024-04-19T01:04:19.547108+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:02.599042+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.019013540999999998,"stat_sum":{"num_bytes":1540096,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":2560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,9,0,8,2,6],"acting":[12,4,9,0,8,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"},{"start":"8","length":"2"}]},{"pgid":"3.c","version":"49'43","reported_seq":139,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-19T01:04:21.518119+0000","last_change":"2024-04-19T01:04:19.546964+0000","last_active":"2024-04-19T01:04:21.518119+0000","last_peered":"2024-04-19T01:04:21.518119+0000","last_clean":"2024-04-19T01:04:21.518119+0000","last_became_active":"2024-04-19T01:04:03.025808+0000","last_became_peered":"2024-04-19T01:04:03.025808+0000","last_unstale":"2024-04-19T01:04:21.518119+0000","last_undegraded":"2024-04-19T01:04:21.518119+0000","last_fullsized":"2024-04-19T01:04:21.518119+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:19.695598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022536225999999999,"stat_sum":{"num_bytes":1884160,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":13,"num_read_kb":2453,"num_write":35,"num_write_kb":1904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,12,1,9,2,6],"acting":[0,10,12,1,9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"},{"start":"8","length":"2"}]},{"pgid":"3.b","version":"45'18","reported_seq":108,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-19T01:04:18.518324+0000","last_change":"2024-04-19T01:04:18.519279+0000","last_active":"2024-04-19T01:04:18.518324+0000","last_peered":"2024-04-19T01:04:18.518324+0000","last_clean":"2024-04-19T01:04:18.518324+0000","last_became_active":"2024-04-19T01:04:02.781569+0000","last_became_peered":"2024-04-19T01:04:02.781569+0000","last_unstale":"2024-04-19T01:04:18.518324+0000","last_undegraded":"2024-04-19T01:04:18.518324+0000","last_fullsized":"2024-04-19T01:04:18.518324+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:16.849719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00152271,"stat_sum":{"num_bytes":2129920,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":7,"num_read_kb":4,"num_write":9,"num_write_kb":2080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11,15,1,4,14],"acting":[10,6,11,15,1,4,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"},{"start":"9","length":"1"}]},{"pgid":"3.a","version":"49'12","reported_seq":124,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T01:04:19.557884+0000","last_change":"2024-04-19T01:04:19.558656+0000","last_active":"2024-04-19T01:04:19.557884+0000","last_peered":"2024-04-19T01:04:19.557884+0000","last_clean":"2024-04-19T01:04:19.557884+0000","last_became_active":"2024-04-19T01:04:03.323177+0000","last_became_peered":"2024-04-19T01:04:03.323177+0000","last_unstale":"2024-04-19T01:04:19.557884+0000","last_undegraded":"2024-04-19T01:04:19.557884+0000","last_fullsized":"2024-04-19T01:04:19.557884+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:14.771327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.029493378000000001,"stat_sum":{"num_bytes":3440640,"num_objects":5,"num_object_clones":1,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":5816,"num_write":9,"num_write_kb":3360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,7,3,13,1,8],"acting":[12,4,7,3,13,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"},{"start":"8","length":"2"}]},{"pgid":"3.9","version":"49'77","reported_seq":178,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-19T01:04:21.534426+0000","last_change":"2024-04-19T01:04:19.557504+0000","last_active":"2024-04-19T01:04:21.534426+0000","last_peered":"2024-04-19T01:04:21.534426+0000","last_clean":"2024-04-19T01:04:21.534426+0000","last_became_active":"2024-04-19T01:04:03.027372+0000","last_became_peered":"2024-04-19T01:04:03.027372+0000","last_unstale":"2024-04-19T01:04:21.534426+0000","last_undegraded":"2024-04-19T01:04:21.534426+0000","last_fullsized":"2024-04-19T01:04:21.534426+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:12.554321+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.023685806,"stat_sum":{"num_bytes":8110080,"num_objects":9,"num_object_clones":3,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":73,"num_write_kb":9024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11,14,9,4,10,6],"acting":[13,11,14,9,4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"},{"start":"8","length":"2"}]},{"pgid":"3.8","version":"49'25","reported_seq":116,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T01:04:19.534323+0000","last_change":"2024-04-19T01:04:19.549872+0000","last_active":"2024-04-19T01:04:19.534323+0000","last_peered":"2024-04-19T01:04:19.534323+0000","last_clean":"2024-04-19T01:04:19.534323+0000","last_became_active":"2024-04-19T01:04:03.237653+0000","last_became_peered":"2024-04-19T01:04:03.237653+0000","last_unstale":"2024-04-19T01:04:19.534323+0000","last_undegraded":"2024-04-19T01:04:19.534323+0000","last_fullsized":"2024-04-19T01:04:19.534323+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:16.852722+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.030904131000000001,"stat_sum":{"num_bytes":2195456,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":30,"num_write_kb":2144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,1,15,14,0,5],"acting":[11,10,1,15,14,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"},{"start":"8","length":"2"}]},{"pgid":"1.0","version":"25'157","reported_seq":256,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T01:04:19.518472+0000","last_change":"2024-04-19T01:03:50.776344+0000","last_active":"2024-04-19T01:04:19.518472+0000","last_peered":"2024-04-19T01:04:19.518472+0000","last_clean":"2024-04-19T01:04:19.518472+0000","last_became_active":"2024-04-19T01:03:50.774078+0000","last_became_peered":"2024-04-19T01:03:50.774078+0000","last_unstale":"2024-04-19T01:04:19.518472+0000","last_undegraded":"2024-04-19T01:04:19.518472+0000","last_fullsized":"2024-04-19T01:04:19.518472+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:49.580851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:49.580851+0000","last_clean_scrub_stamp":"2024-04-19T01:03:49.580851+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:07.495976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":54,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T01:04:19.518443+0000","last_change":"2024-04-19T01:03:53.766587+0000","last_active":"2024-04-19T01:04:19.518443+0000","last_peered":"2024-04-19T01:04:19.518443+0000","last_clean":"2024-04-19T01:04:19.518443+0000","last_became_active":"2024-04-19T01:03:53.765967+0000","last_became_peered":"2024-04-19T01:03:53.765967+0000","last_unstale":"2024-04-19T01:04:19.518443+0000","last_undegraded":"2024-04-19T01:04:19.518443+0000","last_fullsized":"2024-04-19T01:04:19.518443+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:18.833542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"45'14","reported_seq":108,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T01:04:19.519270+0000","last_change":"2024-04-19T01:04:19.520142+0000","last_active":"2024-04-19T01:04:19.519270+0000","last_peered":"2024-04-19T01:04:19.519270+0000","last_clean":"2024-04-19T01:04:19.519270+0000","last_became_active":"2024-04-19T01:04:03.322437+0000","last_became_peered":"2024-04-19T01:04:03.322437+0000","last_unstale":"2024-04-19T01:04:19.519270+0000","last_undegraded":"2024-04-19T01:04:19.519270+0000","last_fullsized":"2024-04-19T01:04:19.519270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:06.317438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001368743,"stat_sum":{"num_bytes":344064,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":6,"num_write_kb":336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0,4,8,3,7],"acting":[11,5,0,4,8,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"},{"start":"8","length":"2"}]},{"pgid":"2.0","version":"0'0","reported_seq":54,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T01:04:19.517419+0000","last_change":"2024-04-19T01:03:53.760075+0000","last_active":"2024-04-19T01:04:19.517419+0000","last_peered":"2024-04-19T01:04:19.517419+0000","last_clean":"2024-04-19T01:04:19.517419+0000","last_became_active":"2024-04-19T01:03:53.759326+0000","last_became_peered":"2024-04-19T01:03:53.759326+0000","last_unstale":"2024-04-19T01:04:19.517419+0000","last_undegraded":"2024-04-19T01:04:19.517419+0000","last_fullsized":"2024-04-19T01:04:19.517419+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:02.718012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"47'12","reported_seq":103,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-19T01:04:18.517899+0000","last_change":"2024-04-19T01:04:18.518427+0000","last_active":"2024-04-19T01:04:18.517899+0000","last_peered":"2024-04-19T01:04:18.517899+0000","last_clean":"2024-04-19T01:04:18.517899+0000","last_became_active":"2024-04-19T01:04:03.323727+0000","last_became_peered":"2024-04-19T01:04:03.323727+0000","last_unstale":"2024-04-19T01:04:18.517899+0000","last_undegraded":"2024-04-19T01:04:18.517899+0000","last_fullsized":"2024-04-19T01:04:18.517899+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:29.482112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011327349999999999,"stat_sum":{"num_bytes":1523712,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":2020,"num_write":27,"num_write_kb":1488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,11,9,1,7],"acting":[6,4,3,11,9,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"},{"start":"9","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":54,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T01:04:19.528833+0000","last_change":"2024-04-19T01:03:53.650434+0000","last_active":"2024-04-19T01:04:19.528833+0000","last_peered":"2024-04-19T01:04:19.528833+0000","last_clean":"2024-04-19T01:04:19.528833+0000","last_became_active":"2024-04-19T01:03:53.649606+0000","last_became_peered":"2024-04-19T01:03:53.649606+0000","last_unstale":"2024-04-19T01:04:19.528833+0000","last_undegraded":"2024-04-19T01:04:19.528833+0000","last_fullsized":"2024-04-19T01:04:19.528833+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:17.424729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"49'35","reported_seq":127,"reported_epoch":49,"state":"active+clean+snaptrim","last_fresh":"2024-04-19T01:04:19.565237+0000","last_change":"2024-04-19T01:04:19.532007+0000","last_active":"2024-04-19T01:04:19.565237+0000","last_peered":"2024-04-19T01:04:19.565237+0000","last_clean":"2024-04-19T01:04:19.565237+0000","last_became_active":"2024-04-19T01:04:03.322503+0000","last_became_peered":"2024-04-19T01:04:03.322503+0000","last_unstale":"2024-04-19T01:04:19.565237+0000","last_undegraded":"2024-04-19T01:04:19.565237+0000","last_fullsized":"2024-04-19T01:04:19.565237+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:13.454365+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.012386256,"stat_sum":{"num_bytes":3817472,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":15,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4,3,10,11,6],"acting":[1,9,4,3,10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"},{"start":"9","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":54,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T01:04:19.517767+0000","last_change":"2024-04-19T01:03:54.012138+0000","last_active":"2024-04-19T01:04:19.517767+0000","last_peered":"2024-04-19T01:04:19.517767+0000","last_clean":"2024-04-19T01:04:19.517767+0000","last_became_active":"2024-04-19T01:03:54.011460+0000","last_became_peered":"2024-04-19T01:03:54.011460+0000","last_unstale":"2024-04-19T01:04:19.517767+0000","last_undegraded":"2024-04-19T01:04:19.517767+0000","last_fullsized":"2024-04-19T01:04:19.517767+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:04:58.336612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"49'42","reported_seq":137,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T01:04:19.561330+0000","last_change":"2024-04-19T01:04:19.563217+0000","last_active":"2024-04-19T01:04:19.561330+0000","last_peered":"2024-04-19T01:04:19.561330+0000","last_clean":"2024-04-19T01:04:19.561330+0000","last_became_active":"2024-04-19T01:04:02.835716+0000","last_became_peered":"2024-04-19T01:04:02.835716+0000","last_unstale":"2024-04-19T01:04:19.561330+0000","last_undegraded":"2024-04-19T01:04:19.561330+0000","last_fullsized":"2024-04-19T01:04:19.561330+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:14.688145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032384638,"stat_sum":{"num_bytes":3833856,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":13,"num_read_kb":1110,"num_write":55,"num_write_kb":3744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2,0,1,4,12],"acting":[15,8,2,0,1,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"},{"start":"8","length":"2"}]},{"pgid":"3.5","version":"45'43","reported_seq":147,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T01:04:19.518177+0000","last_change":"2024-04-19T01:04:19.518641+0000","last_active":"2024-04-19T01:04:19.518177+0000","last_peered":"2024-04-19T01:04:19.518177+0000","last_clean":"2024-04-19T01:04:19.518177+0000","last_became_active":"2024-04-19T01:04:03.237886+0000","last_became_peered":"2024-04-19T01:04:03.237886+0000","last_unstale":"2024-04-19T01:04:19.518177+0000","last_undegraded":"2024-04-19T01:04:19.518177+0000","last_fullsized":"2024-04-19T01:04:19.518177+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:13.565446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068062999999999999,"stat_sum":{"num_bytes":3293184,"num_objects":9,"num_object_clones":3,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":30,"num_read_kb":2938,"num_write":39,"num_write_kb":4432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,14,11,10,2,5],"acting":[8,6,14,11,10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"},{"start":"8","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":52,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-19T01:04:18.517402+0000","last_change":"2024-04-19T01:03:53.716606+0000","last_active":"2024-04-19T01:04:18.517402+0000","last_peered":"2024-04-19T01:04:18.517402+0000","last_clean":"2024-04-19T01:04:18.517402+0000","last_became_active":"2024-04-19T01:03:53.716040+0000","last_became_peered":"2024-04-19T01:03:53.716040+0000","last_unstale":"2024-04-19T01:04:18.517402+0000","last_undegraded":"2024-04-19T01:04:18.517402+0000","last_fullsized":"2024-04-19T01:04:18.517402+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:01.975515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"48'16","reported_seq":111,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-19T01:04:18.536522+0000","last_change":"2024-04-19T01:04:18.536522+0000","last_active":"2024-04-19T01:04:18.536522+0000","last_peered":"2024-04-19T01:04:18.536522+0000","last_clean":"2024-04-19T01:04:18.536522+0000","last_became_active":"2024-04-19T01:04:03.322815+0000","last_became_peered":"2024-04-19T01:04:03.322815+0000","last_unstale":"2024-04-19T01:04:18.536522+0000","last_undegraded":"2024-04-19T01:04:18.536522+0000","last_fullsized":"2024-04-19T01:04:18.536522+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:12.411696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018823612,"stat_sum":{"num_bytes":753664,"num_objects":5,"num_object_clones":1,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":582,"num_write":12,"num_write_kb":1072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,13,14,3,15,1,5],"acting":[10,13,14,3,15,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"3"},{"start":"9","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":58,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-19T01:04:21.520459+0000","last_change":"2024-04-19T01:03:53.772709+0000","last_active":"2024-04-19T01:04:21.520459+0000","last_peered":"2024-04-19T01:04:21.520459+0000","last_clean":"2024-04-19T01:04:21.520459+0000","last_became_active":"2024-04-19T01:03:53.771887+0000","last_became_peered":"2024-04-19T01:03:53.771887+0000","last_unstale":"2024-04-19T01:04:21.520459+0000","last_undegraded":"2024-04-19T01:04:21.520459+0000","last_fullsized":"2024-04-19T01:04:21.520459+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:17.479766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"45'12","reported_seq":112,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-19T01:04:21.504008+0000","last_change":"2024-04-19T01:04:19.536554+0000","last_active":"2024-04-19T01:04:21.504008+0000","last_peered":"2024-04-19T01:04:21.504008+0000","last_clean":"2024-04-19T01:04:21.504008+0000","last_became_active":"2024-04-19T01:04:03.238517+0000","last_became_peered":"2024-04-19T01:04:03.238517+0000","last_unstale":"2024-04-19T01:04:21.504008+0000","last_undegraded":"2024-04-19T01:04:21.504008+0000","last_fullsized":"2024-04-19T01:04:21.504008+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:05.503369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011797730000000001,"stat_sum":{"num_bytes":1589248,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":5,"num_write_kb":1552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,7,5,11,6,0],"acting":[14,13,7,5,11,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"},{"start":"8","length":"2"}]},{"pgid":"2.6","version":"0'0","reported_seq":58,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-19T01:04:21.503636+0000","last_change":"2024-04-19T01:03:53.736290+0000","last_active":"2024-04-19T01:04:21.503636+0000","last_peered":"2024-04-19T01:04:21.503636+0000","last_clean":"2024-04-19T01:04:21.503636+0000","last_became_active":"2024-04-19T01:03:53.735549+0000","last_became_peered":"2024-04-19T01:03:53.735549+0000","last_unstale":"2024-04-19T01:04:21.503636+0000","last_undegraded":"2024-04-19T01:04:21.503636+0000","last_fullsized":"2024-04-19T01:04:21.503636+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:07.640918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13],"acting":[14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":99,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-19T01:04:21.503914+0000","last_change":"2024-04-19T01:04:19.534772+0000","last_active":"2024-04-19T01:04:21.503914+0000","last_peered":"2024-04-19T01:04:21.503914+0000","last_clean":"2024-04-19T01:04:21.503914+0000","last_became_active":"2024-04-19T01:04:02.889974+0000","last_became_peered":"2024-04-19T01:04:02.889974+0000","last_unstale":"2024-04-19T01:04:21.503914+0000","last_undegraded":"2024-04-19T01:04:21.503914+0000","last_fullsized":"2024-04-19T01:04:21.503914+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:16.353148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090651100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7,15,8,11,0,1],"acting":[14,7,15,8,11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"},{"start":"8","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":54,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T01:04:19.517740+0000","last_change":"2024-04-19T01:03:53.939441+0000","last_active":"2024-04-19T01:04:19.517740+0000","last_peered":"2024-04-19T01:04:19.517740+0000","last_clean":"2024-04-19T01:04:19.517740+0000","last_became_active":"2024-04-19T01:03:53.938774+0000","last_became_peered":"2024-04-19T01:03:53.938774+0000","last_unstale":"2024-04-19T01:04:19.517740+0000","last_undegraded":"2024-04-19T01:04:19.517740+0000","last_fullsized":"2024-04-19T01:04:19.517740+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:14.081709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":40632320,"num_objects":71,"num_object_clones":23,"num_object_copies":497,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":3,"num_read":183,"num_read_kb":20050,"num_write":361,"num_write_kb":44656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":72511488,"data_stored":87506944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":447,"ondisk_log_size":447,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":13},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":15546,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":14,"up_from":24,"seq":103079215114,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36324,"kb_used_data":5136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335516,"statfs":{"total":96636764160,"available":96599568384,"internally_reserved":0,"allocated":5259264,"data_stored":6364861,"data_compressed":55137,"data_compressed_allocated":114688,"data_compressed_original":229376,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,5,6,7,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":13,"up_from":24,"seq":103079215115,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36592,"kb_used_data":5412,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335248,"statfs":{"total":96636764160,"available":96599293952,"internally_reserved":0,"allocated":5541888,"data_stored":6639155,"data_compressed":51143,"data_compressed_allocated":106496,"data_compressed_original":212992,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,4,5,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":9,"up_from":24,"seq":103079215116,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36244,"kb_used_data":5060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335596,"statfs":{"total":96636764160,"available":96599650304,"internally_reserved":0,"allocated":5181440,"data_stored":6305372,"data_compressed":53136,"data_compressed_allocated":110592,"data_compressed_original":221184,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313226,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33068,"kb_used_data":6040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338772,"statfs":{"total":96636764160,"available":96602902528,"internally_reserved":0,"allocated":6184960,"data_stored":7486833,"data_compressed":47144,"data_compressed_allocated":98304,"data_compressed_original":196608,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,3,4,5,6,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280520,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33904,"kb_used_data":6872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337936,"statfs":{"total":96636764160,"available":96602046464,"internally_reserved":0,"allocated":7036928,"data_stored":8869330,"data_compressed":49143,"data_compressed_allocated":102400,"data_compressed_original":204800,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280522,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33264,"kb_used_data":6232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338576,"statfs":{"total":96636764160,"available":96602701824,"internally_reserved":0,"allocated":6381568,"data_stored":6997458,"data_compressed":49143,"data_compressed_allocated":102400,"data_compressed_original":204800,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280521,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29944,"kb_used_data":2912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341896,"statfs":{"total":96636764160,"available":96606101504,"internally_reserved":0,"allocated":2981888,"data_stored":2897362,"data_compressed":49143,"data_compressed_allocated":102400,"data_compressed_original":204800,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[2,4,5,6,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280521,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34528,"kb_used_data":7496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337312,"statfs":{"total":96636764160,"available":96601407488,"internally_reserved":0,"allocated":7675904,"data_stored":9602514,"data_compressed":49143,"data_compressed_allocated":102400,"data_compressed_original":204800,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280521,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30944,"kb_used_data":3916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340896,"statfs":{"total":96636764160,"available":96605077504,"internally_reserved":0,"allocated":4009984,"data_stored":4892658,"data_compressed":56916,"data_compressed_allocated":135168,"data_compressed_original":716800,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,6,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280522,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30836,"kb_used_data":3804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341004,"statfs":{"total":96636764160,"available":96605188096,"internally_reserved":0,"allocated":3895296,"data_stored":4040146,"data_compressed":49143,"data_compressed_allocated":102400,"data_compressed_original":204800,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[2,5,6,7,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280520,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34840,"kb_used_data":7812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337000,"statfs":{"total":96636764160,"available":96601088000,"internally_reserved":0,"allocated":7999488,"data_stored":10557426,"data_compressed":56916,"data_compressed_allocated":135168,"data_compressed_original":716800,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,3,4,5,6,7,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280520,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30408,"kb_used_data":3376,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341432,"statfs":{"total":96636764160,"available":96605626368,"internally_reserved":0,"allocated":3457024,"data_stored":3679698,"data_compressed":49143,"data_compressed_allocated":102400,"data_compressed_original":204800,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280521,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29984,"kb_used_data":2952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341856,"statfs":{"total":96636764160,"available":96606060544,"internally_reserved":0,"allocated":3022848,"data_stored":3478994,"data_compressed":49143,"data_compressed_allocated":102400,"data_compressed_original":204800,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,4,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":24,"seq":103079215114,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35776,"kb_used_data":4596,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336064,"statfs":{"total":96636764160,"available":96600129536,"internally_reserved":0,"allocated":4706304,"data_stored":5131827,"data_compressed":51143,"data_compressed_allocated":106496,"data_compressed_original":212992,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215115,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35752,"kb_used_data":4572,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336088,"statfs":{"total":96636764160,"available":96600154112,"internally_reserved":0,"allocated":4681728,"data_stored":5062195,"data_compressed":51143,"data_compressed_allocated":106496,"data_compressed_original":212992,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":24,"seq":103079215114,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30324,"kb_used_data":3300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341516,"statfs":{"total":96636764160,"available":96605712384,"internally_reserved":0,"allocated":3379200,"data_stored":3266140,"data_compressed":53136,"data_compressed_allocated":110592,"data_compressed_original":221184,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4730880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6119424,"data_stored":6770688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4182016,"data_stored":4800512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2469888,"data_stored":2576384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":6610944,"data_stored":7983104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3588096,"data_stored":4050944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":6524928,"data_stored":8548352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2842624,"data_stored":2924544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":3112960,"data_stored":3391488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4653056,"data_stored":5971968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":5685248,"data_stored":8261632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":7401472,"data_stored":8871936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":2674688,"data_stored":3031040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":5017600,"data_stored":6307840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":4710400,"data_stored":6012928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":2760704,"data_stored":3272704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T01:04:22.839 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pgp_num 16 2024-04-19T01:04:22.857 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2024-04-19T01:04:22.857 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T01:04:22.877 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2024-04-19T01:04:22.878 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T01:04:22.898 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2024-04-19T01:04:22.898 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T01:04:22.899 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:04:22.895+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:04:22.915 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2024-04-19T01:04:22.915 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-19T01:04:22.918 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:22.918 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:04:22.918 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2024-04-19T01:04:22.918 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:22.932 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-19T01:04:22.935 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2024-04-19T01:04:22.936 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-19T01:04:22.957 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2024-04-19T01:04:22.957 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-19T01:04:22.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:04:22.955+0000 7f4aee502640 0 -- 172.21.15.33:0/267208225 >> [v2:172.21.15.155:6807/741132602,v1:172.21.15.155:6810/741132602] conn(0x7f4ae82213b0 msgr2=0x7f4ae82237a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-04-19T01:04:22.979 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2024-04-19T01:04:22.979 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-19T01:04:23.041 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:04:23.039+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:04:23.046 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:23.046 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:04:23.046 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2024-04-19T01:04:23.047 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2024-04-19T01:04:23.047 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:23.060 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-19T01:04:23.101 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:04:23.101+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:04:23.201 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:04:23.204+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:04:23.221 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:23.221 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2024-04-19T01:04:23.221 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2024-04-19T01:04:23.221 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-04-19T01:04:23.221 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.221 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.3s5 38/31 ECSubWrite(tid=13, reqid=client.4604.0:173, at_version=38'9, trim_to=0'0, roll_forward_to=34'7))", 2024-04-19T01:04:23.221 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:08.824535+0000", 2024-04-19T01:04:23.221 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 14.39497933, 2024-04-19T01:04:23.221 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0061315839999999998, 2024-04-19T01:04:23.221 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:23.222 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:23.222 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:23.222 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:23.222 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.222 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:23.222 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.824535+0000", 2024-04-19T01:04:23.222 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.222 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.222 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.222 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:23.222 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.824535+0000", 2024-04-19T01:04:23.222 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.222 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.222 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.223 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:23.223 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.824535+0000", 2024-04-19T01:04:23.223 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9099999999999999e-07 2024-04-19T01:04:23.223 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.223 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.223 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:23.223 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.824893+0000", 2024-04-19T01:04:23.223 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00035774800000000002 2024-04-19T01:04:23.223 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.223 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.223 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:23.223 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.824919+0000", 2024-04-19T01:04:23.224 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6097000000000001e-05 2024-04-19T01:04:23.224 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.224 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.224 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:23.224 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.824933+0000", 2024-04-19T01:04:23.224 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4102e-05 2024-04-19T01:04:23.224 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.224 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.224 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.224 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.825006+0000", 2024-04-19T01:04:23.224 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.3485999999999994e-05 2024-04-19T01:04:23.224 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.224 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.224 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:23.225 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.825017+0000", 2024-04-19T01:04:23.225 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0236e-05 2024-04-19T01:04:23.225 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.225 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.225 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:23.225 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.830631+0000", 2024-04-19T01:04:23.225 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0056146809999999998 2024-04-19T01:04:23.226 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.226 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.226 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:23.226 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.830666+0000", 2024-04-19T01:04:23.226 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4842999999999998e-05 2024-04-19T01:04:23.226 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.226 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.226 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.226 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.226 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.226 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.9s4 43/31 ECSubWrite(tid=70, reqid=client.4604.0:235, at_version=43'43, trim_to=0'0, roll_forward_to=41'37))", 2024-04-19T01:04:23.226 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.869289+0000", 2024-04-19T01:04:23.226 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 9.3502252509999995, 2024-04-19T01:04:23.226 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0068565980000000002, 2024-04-19T01:04:23.227 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:23.227 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:23.227 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:23.228 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:23.228 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.228 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:23.228 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.869289+0000", 2024-04-19T01:04:23.228 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.228 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.228 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.228 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:23.229 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.869289+0000", 2024-04-19T01:04:23.229 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.229 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.229 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.229 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:23.229 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.869290+0000", 2024-04-19T01:04:23.229 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2419999999999999e-06 2024-04-19T01:04:23.230 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.230 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.230 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:23.230 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.869823+0000", 2024-04-19T01:04:23.230 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00053351500000000003 2024-04-19T01:04:23.230 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.230 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.230 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:23.230 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.869851+0000", 2024-04-19T01:04:23.230 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7634999999999999e-05 2024-04-19T01:04:23.230 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.231 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.231 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:23.234 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.869862+0000", 2024-04-19T01:04:23.234 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0677e-05 2024-04-19T01:04:23.234 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.234 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.234 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.235 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.870063+0000", 2024-04-19T01:04:23.235 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00020101900000000001 2024-04-19T01:04:23.235 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.235 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.235 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:23.235 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.870075+0000", 2024-04-19T01:04:23.235 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1855000000000001e-05 2024-04-19T01:04:23.235 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.235 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.235 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:23.235 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.876107+0000", 2024-04-19T01:04:23.235 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0060328990000000004 2024-04-19T01:04:23.235 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.236 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.236 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:23.236 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.876145+0000", 2024-04-19T01:04:23.236 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7756000000000002e-05 2024-04-19T01:04:23.236 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.236 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.236 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.236 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.236 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.236 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds1 43/31 ECSubWrite(tid=38, reqid=client.4604.0:253, at_version=43'14, trim_to=0'0, roll_forward_to=43'12))", 2024-04-19T01:04:23.236 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.986285+0000", 2024-04-19T01:04:23.236 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 9.2332283969999995, 2024-04-19T01:04:23.236 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0090548699999999996, 2024-04-19T01:04:23.236 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:23.237 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:23.237 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:23.237 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:23.237 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.237 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:23.237 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.986285+0000", 2024-04-19T01:04:23.237 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.237 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.237 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.237 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:23.237 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.986285+0000", 2024-04-19T01:04:23.237 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.237 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.238 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.238 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:23.238 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.986287+0000", 2024-04-19T01:04:23.238 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.421e-06 2024-04-19T01:04:23.238 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.238 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.238 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:23.238 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.986389+0000", 2024-04-19T01:04:23.238 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010215100000000001 2024-04-19T01:04:23.238 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.238 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.238 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:23.238 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.986414+0000", 2024-04-19T01:04:23.239 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4447000000000002e-05 2024-04-19T01:04:23.239 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.239 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.239 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:23.239 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.986435+0000", 2024-04-19T01:04:23.239 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1069000000000001e-05 2024-04-19T01:04:23.239 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.239 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.239 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.239 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.986566+0000", 2024-04-19T01:04:23.239 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013169899999999999 2024-04-19T01:04:23.240 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.240 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.240 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:23.240 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.986604+0000", 2024-04-19T01:04:23.240 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7857e-05 2024-04-19T01:04:23.240 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.240 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.240 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:23.240 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.995257+0000", 2024-04-19T01:04:23.240 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0086525209999999998 2024-04-19T01:04:23.240 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.240 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.240 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:23.240 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.995340+0000", 2024-04-19T01:04:23.241 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.3704999999999999e-05 2024-04-19T01:04:23.241 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.241 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.241 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.241 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.241 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.241 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds1 43/31 ECSubWrite(tid=39, reqid=client.4604.0:254, at_version=43'15, trim_to=0'0, roll_forward_to=43'12))", 2024-04-19T01:04:23.241 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.996134+0000", 2024-04-19T01:04:23.241 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 9.2233799449999996, 2024-04-19T01:04:23.241 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0070946079999999996, 2024-04-19T01:04:23.241 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:23.241 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:23.241 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:23.242 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:23.242 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.242 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:23.242 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.996134+0000", 2024-04-19T01:04:23.242 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.242 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.242 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.242 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:23.242 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.996134+0000", 2024-04-19T01:04:23.242 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.242 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.242 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.242 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:23.243 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.996135+0000", 2024-04-19T01:04:23.243 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2839999999999999e-06 2024-04-19T01:04:23.243 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.243 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.243 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:23.243 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.996186+0000", 2024-04-19T01:04:23.243 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0572999999999998e-05 2024-04-19T01:04:23.243 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.243 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.243 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:23.243 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.996212+0000", 2024-04-19T01:04:23.244 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6658000000000001e-05 2024-04-19T01:04:23.244 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.244 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.244 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:23.244 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.996220+0000", 2024-04-19T01:04:23.245 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5939999999999996e-06 2024-04-19T01:04:23.245 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.245 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.245 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.245 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.996318+0000", 2024-04-19T01:04:23.245 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7595000000000003e-05 2024-04-19T01:04:23.245 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.245 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.245 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:23.245 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.996326+0000", 2024-04-19T01:04:23.245 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.8650000000000001e-06 2024-04-19T01:04:23.245 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.245 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.245 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:23.246 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.003178+0000", 2024-04-19T01:04:23.246 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0068528449999999998 2024-04-19T01:04:23.246 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.246 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.246 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:23.246 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.003229+0000", 2024-04-19T01:04:23.246 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0194000000000001e-05 2024-04-19T01:04:23.246 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.246 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.246 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.246 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.247 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.247 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.9s4 43/31 ECSubWrite(tid=76, reqid=client.4604.0:267, at_version=43'46, trim_to=0'0, roll_forward_to=43'45))", 2024-04-19T01:04:23.247 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:14.025386+0000", 2024-04-19T01:04:23.247 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 9.1941274229999994, 2024-04-19T01:04:23.247 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0060441590000000003, 2024-04-19T01:04:23.247 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:23.247 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:23.247 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:23.247 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:23.247 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.247 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:23.247 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.025386+0000", 2024-04-19T01:04:23.248 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.248 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.248 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.248 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:23.248 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.025386+0000", 2024-04-19T01:04:23.248 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.248 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.248 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.248 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:23.248 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.025388+0000", 2024-04-19T01:04:23.248 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1000000000000001e-06 2024-04-19T01:04:23.248 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.248 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.248 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:23.249 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.025554+0000", 2024-04-19T01:04:23.249 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00016603000000000001 2024-04-19T01:04:23.249 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.249 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.249 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:23.249 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.025569+0000", 2024-04-19T01:04:23.249 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5701000000000001e-05 2024-04-19T01:04:23.249 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.249 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.249 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:23.249 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.025578+0000", 2024-04-19T01:04:23.249 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.7430000000000008e-06 2024-04-19T01:04:23.249 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.249 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.250 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.250 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.025645+0000", 2024-04-19T01:04:23.250 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.6625000000000007e-05 2024-04-19T01:04:23.250 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.250 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.250 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:23.250 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.025650+0000", 2024-04-19T01:04:23.250 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6860000000000003e-06 2024-04-19T01:04:23.250 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.250 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.250 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:23.250 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.031390+0000", 2024-04-19T01:04:23.250 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0057393100000000001 2024-04-19T01:04:23.251 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.251 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.251 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:23.251 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.031431+0000", 2024-04-19T01:04:23.251 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0964000000000003e-05 2024-04-19T01:04:23.251 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.251 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.251 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.252 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.252 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.252 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.9s4 44/31 ECSubWrite(tid=82, reqid=client.4604.0:280, at_version=44'50, trim_to=0'0, roll_forward_to=43'48))", 2024-04-19T01:04:23.252 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:14.880621+0000", 2024-04-19T01:04:23.252 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 8.3388931900000003, 2024-04-19T01:04:23.252 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.009454835, 2024-04-19T01:04:23.252 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:23.252 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:23.252 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:23.252 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:23.252 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.252 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:23.252 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.880621+0000", 2024-04-19T01:04:23.252 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.253 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.253 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.253 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:23.253 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.880621+0000", 2024-04-19T01:04:23.253 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.253 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.253 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.253 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:23.253 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.880622+0000", 2024-04-19T01:04:23.253 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.6200000000000006e-07 2024-04-19T01:04:23.254 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.254 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.254 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:23.254 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.881098+0000", 2024-04-19T01:04:23.255 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00047665400000000002 2024-04-19T01:04:23.255 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.255 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.255 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:23.255 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.881115+0000", 2024-04-19T01:04:23.255 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6612999999999998e-05 2024-04-19T01:04:23.255 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.255 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.255 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:23.255 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.881124+0000", 2024-04-19T01:04:23.255 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.3959999999999995e-06 2024-04-19T01:04:23.255 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.256 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.256 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.256 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.881176+0000", 2024-04-19T01:04:23.256 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.1965000000000001e-05 2024-04-19T01:04:23.256 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.256 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.256 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:23.256 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.881182+0000", 2024-04-19T01:04:23.256 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.806e-06 2024-04-19T01:04:23.256 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.257 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.257 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:23.257 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.890038+0000", 2024-04-19T01:04:23.257 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0088557170000000008 2024-04-19T01:04:23.258 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.258 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.258 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:23.258 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.890076+0000", 2024-04-19T01:04:23.258 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7722000000000002e-05 2024-04-19T01:04:23.258 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.258 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.258 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.258 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.258 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.258 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.0s2 44/31 ECSubWrite(tid=25, reqid=client.4604.0:284, at_version=44'22, trim_to=0'0, roll_forward_to=44'20))", 2024-04-19T01:04:23.258 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:14.920303+0000", 2024-04-19T01:04:23.258 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 8.2992109169999999, 2024-04-19T01:04:23.258 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011517777999999999, 2024-04-19T01:04:23.259 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:23.259 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:23.259 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:23.259 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:23.259 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.259 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:23.259 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.920303+0000", 2024-04-19T01:04:23.259 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.259 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.259 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.259 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:23.259 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.920303+0000", 2024-04-19T01:04:23.259 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.260 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.260 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.260 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:23.260 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.920304+0000", 2024-04-19T01:04:23.260 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2900000000000003e-07 2024-04-19T01:04:23.260 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.260 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.260 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:23.260 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.921382+0000", 2024-04-19T01:04:23.260 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010785510000000001 2024-04-19T01:04:23.260 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.260 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.260 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:23.260 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.921394+0000", 2024-04-19T01:04:23.260 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2183e-05 2024-04-19T01:04:23.261 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.261 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.261 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:23.261 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.921403+0000", 2024-04-19T01:04:23.261 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9330000000000007e-06 2024-04-19T01:04:23.261 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.261 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.261 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.261 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.921468+0000", 2024-04-19T01:04:23.261 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4191999999999998e-05 2024-04-19T01:04:23.261 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.261 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.261 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:23.261 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.921479+0000", 2024-04-19T01:04:23.262 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1763000000000001e-05 2024-04-19T01:04:23.262 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.262 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.262 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:23.262 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.931782+0000", 2024-04-19T01:04:23.263 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010302891999999999 2024-04-19T01:04:23.263 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.263 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.263 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:23.263 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.931821+0000", 2024-04-19T01:04:23.263 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8535000000000003e-05 2024-04-19T01:04:23.263 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.263 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.263 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.264 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.264 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.264 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds1 47/31 ECSubWrite(tid=61, reqid=client.4604.0:330, at_version=47'32, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:04:23.264 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.676507+0000", 2024-04-19T01:04:23.264 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.5430068070000003, 2024-04-19T01:04:23.264 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010458914, 2024-04-19T01:04:23.264 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:23.264 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:23.265 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:23.265 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:23.265 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.265 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:23.265 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.676507+0000", 2024-04-19T01:04:23.265 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.265 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.265 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.265 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:23.265 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.676507+0000", 2024-04-19T01:04:23.265 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.265 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.265 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.265 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:23.266 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.676508+0000", 2024-04-19T01:04:23.266 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2399999999999996e-07 2024-04-19T01:04:23.266 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.266 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.266 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:23.266 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.676672+0000", 2024-04-19T01:04:23.266 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000164258 2024-04-19T01:04:23.266 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.266 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.266 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:23.267 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.676688+0000", 2024-04-19T01:04:23.267 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5443000000000001e-05 2024-04-19T01:04:23.268 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.268 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.268 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:23.268 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.676696+0000", 2024-04-19T01:04:23.268 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1550000000000007e-06 2024-04-19T01:04:23.268 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.268 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.268 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.268 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.676799+0000", 2024-04-19T01:04:23.269 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000102786 2024-04-19T01:04:23.269 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.269 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.269 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:23.269 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.676814+0000", 2024-04-19T01:04:23.269 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4951e-05 2024-04-19T01:04:23.269 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.269 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.269 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:23.269 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.686911+0000", 2024-04-19T01:04:23.269 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010097415 2024-04-19T01:04:23.269 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.269 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.269 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:23.270 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.686966+0000", 2024-04-19T01:04:23.270 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4982000000000003e-05 2024-04-19T01:04:23.270 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.270 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.270 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.270 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.270 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.270 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds1 47/31 ECSubWrite(tid=62, reqid=client.4604.0:331, at_version=47'33, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:04:23.270 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.681491+0000", 2024-04-19T01:04:23.270 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.5380227399999997, 2024-04-19T01:04:23.270 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0059223949999999996, 2024-04-19T01:04:23.270 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:23.271 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:23.271 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:23.271 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:23.271 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.271 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:23.271 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.681491+0000", 2024-04-19T01:04:23.271 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.271 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.271 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.271 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:23.271 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.681491+0000", 2024-04-19T01:04:23.271 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.271 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.272 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.272 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:23.272 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.681492+0000", 2024-04-19T01:04:23.272 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1370000000000001e-06 2024-04-19T01:04:23.272 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.272 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.272 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:23.272 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.681640+0000", 2024-04-19T01:04:23.272 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014797700000000001 2024-04-19T01:04:23.272 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.272 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.272 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:23.273 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.681660+0000", 2024-04-19T01:04:23.273 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9630999999999999e-05 2024-04-19T01:04:23.273 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.273 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.273 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:23.273 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.681670+0000", 2024-04-19T01:04:23.273 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.6970000000000005e-06 2024-04-19T01:04:23.273 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.273 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.273 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.273 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682082+0000", 2024-04-19T01:04:23.273 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000412879 2024-04-19T01:04:23.273 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.273 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.274 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:23.274 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682094+0000", 2024-04-19T01:04:23.274 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1539000000000001e-05 2024-04-19T01:04:23.274 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.274 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.274 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:23.274 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.687364+0000", 2024-04-19T01:04:23.274 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0052699119999999999 2024-04-19T01:04:23.274 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.274 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.274 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:23.274 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.687414+0000", 2024-04-19T01:04:23.275 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.9623000000000002e-05 2024-04-19T01:04:23.275 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.275 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.275 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.275 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.275 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.275 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds1 48/31 ECSubWrite(tid=67, reqid=osd.12.0:67, at_version=48'37, trim_to=0'0, roll_forward_to=47'35))", 2024-04-19T01:04:23.275 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.511815+0000", 2024-04-19T01:04:23.275 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 4.7076984340000001, 2024-04-19T01:04:23.275 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0069408539999999998, 2024-04-19T01:04:23.275 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:23.275 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:23.275 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:23.275 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:23.276 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.276 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:23.276 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.511815+0000", 2024-04-19T01:04:23.276 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.276 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.276 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.276 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:23.276 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.511815+0000", 2024-04-19T01:04:23.276 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.276 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.276 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.276 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:23.276 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.511816+0000", 2024-04-19T01:04:23.277 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6000000000000004e-07 2024-04-19T01:04:23.277 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.277 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.277 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:23.277 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.511877+0000", 2024-04-19T01:04:23.277 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0924000000000001e-05 2024-04-19T01:04:23.277 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.278 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.278 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:23.278 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.511908+0000", 2024-04-19T01:04:23.278 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0635000000000001e-05 2024-04-19T01:04:23.279 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.279 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.279 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:23.279 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.511931+0000", 2024-04-19T01:04:23.279 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3462e-05 2024-04-19T01:04:23.279 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.279 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.279 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.279 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.512031+0000", 2024-04-19T01:04:23.279 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000100202 2024-04-19T01:04:23.279 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.279 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.279 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:23.280 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.512049+0000", 2024-04-19T01:04:23.280 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7623000000000001e-05 2024-04-19T01:04:23.280 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.280 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.280 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:23.280 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.518712+0000", 2024-04-19T01:04:23.280 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0066631950000000002 2024-04-19T01:04:23.280 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.280 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.280 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:23.280 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.518756+0000", 2024-04-19T01:04:23.280 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4252999999999998e-05 2024-04-19T01:04:23.280 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.281 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.281 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.281 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.281 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.281 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds1 48/31 ECSubWrite(tid=68, reqid=osd.12.0:68, at_version=48'39, trim_to=0'0, roll_forward_to=47'35))", 2024-04-19T01:04:23.281 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.513367+0000", 2024-04-19T01:04:23.282 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 4.7061466469999997, 2024-04-19T01:04:23.282 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0073005730000000003, 2024-04-19T01:04:23.282 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:23.282 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:23.283 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:23.283 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:23.283 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.283 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:23.283 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.513367+0000", 2024-04-19T01:04:23.283 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.283 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.283 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.284 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:23.284 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.513367+0000", 2024-04-19T01:04:23.284 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.284 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.285 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.286 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:23.286 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.513369+0000", 2024-04-19T01:04:23.286 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6670000000000001e-06 2024-04-19T01:04:23.286 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.286 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.286 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:23.286 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.513417+0000", 2024-04-19T01:04:23.286 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8534e-05 2024-04-19T01:04:23.287 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.287 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.287 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:23.287 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.513461+0000", 2024-04-19T01:04:23.287 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3151000000000001e-05 2024-04-19T01:04:23.287 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.287 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.287 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:23.287 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.513484+0000", 2024-04-19T01:04:23.287 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3088999999999999e-05 2024-04-19T01:04:23.287 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.287 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.287 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.287 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.514630+0000", 2024-04-19T01:04:23.288 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001145868 2024-04-19T01:04:23.288 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.288 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.288 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:23.289 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.514638+0000", 2024-04-19T01:04:23.289 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.8769999999999999e-06 2024-04-19T01:04:23.289 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.289 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.289 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:23.289 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.520622+0000", 2024-04-19T01:04:23.289 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0059831240000000003 2024-04-19T01:04:23.290 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.290 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.290 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:23.290 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.520668+0000", 2024-04-19T01:04:23.290 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6263000000000001e-05 2024-04-19T01:04:23.290 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.290 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.290 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.290 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.290 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.290 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.3s5 48/31 ECSubWrite(tid=46, reqid=client.4604.0:343, at_version=48'36, trim_to=0'0, roll_forward_to=48'34))", 2024-04-19T01:04:23.290 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.541349+0000", 2024-04-19T01:04:23.291 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 4.6781643959999997, 2024-04-19T01:04:23.291 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010550992, 2024-04-19T01:04:23.291 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:23.291 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:23.291 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:23.291 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:23.291 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.291 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:23.291 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.541349+0000", 2024-04-19T01:04:23.291 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.291 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.291 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.292 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:23.292 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.541349+0000", 2024-04-19T01:04:23.292 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.292 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.292 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.292 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:23.292 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.541351+0000", 2024-04-19T01:04:23.292 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.248e-06 2024-04-19T01:04:23.292 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.292 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.292 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:23.292 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.541675+0000", 2024-04-19T01:04:23.293 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00032395500000000001 2024-04-19T01:04:23.293 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.293 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.293 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:23.293 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.541707+0000", 2024-04-19T01:04:23.293 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2042000000000003e-05 2024-04-19T01:04:23.293 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.293 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.293 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:23.293 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.541720+0000", 2024-04-19T01:04:23.293 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3101999999999999e-05 2024-04-19T01:04:23.293 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.293 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.294 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.294 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.541873+0000", 2024-04-19T01:04:23.294 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015282199999999999 2024-04-19T01:04:23.294 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.294 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.294 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:23.294 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.541899+0000", 2024-04-19T01:04:23.294 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6381999999999999e-05 2024-04-19T01:04:23.294 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.294 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.294 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:23.294 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.551828+0000", 2024-04-19T01:04:23.295 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0099288589999999999 2024-04-19T01:04:23.295 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.295 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.295 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:23.295 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.551900+0000", 2024-04-19T01:04:23.295 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2582000000000004e-05 2024-04-19T01:04:23.295 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.295 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.295 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.295 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.296 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.296 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.as1 48/31 ECSubWrite(tid=71, reqid=client.4604.0:354, at_version=48'10, trim_to=0'0, roll_forward_to=34'8))", 2024-04-19T01:04:23.296 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.556705+0000", 2024-04-19T01:04:23.296 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 4.662808622, 2024-04-19T01:04:23.296 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013008369000000001, 2024-04-19T01:04:23.296 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:23.296 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:23.296 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:23.296 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:23.296 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.296 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:23.296 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.556705+0000", 2024-04-19T01:04:23.297 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.297 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.297 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.297 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:23.297 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.556705+0000", 2024-04-19T01:04:23.297 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.297 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.297 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.297 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:23.297 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.556706+0000", 2024-04-19T01:04:23.297 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2800000000000005e-07 2024-04-19T01:04:23.297 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.297 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.297 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:23.297 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.557214+0000", 2024-04-19T01:04:23.298 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00050754200000000004 2024-04-19T01:04:23.298 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.298 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.298 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:23.298 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.557239+0000", 2024-04-19T01:04:23.298 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.567e-05 2024-04-19T01:04:23.298 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.299 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.299 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:23.299 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.557268+0000", 2024-04-19T01:04:23.299 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8496e-05 2024-04-19T01:04:23.299 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.300 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.300 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.300 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.557438+0000", 2024-04-19T01:04:23.300 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000169963 2024-04-19T01:04:23.300 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.300 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.300 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:23.300 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.557491+0000", 2024-04-19T01:04:23.300 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2689999999999999e-05 2024-04-19T01:04:23.300 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.300 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.300 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:23.300 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.569674+0000", 2024-04-19T01:04:23.300 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012183727 2024-04-19T01:04:23.300 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.301 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.302 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:23.302 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.569714+0000", 2024-04-19T01:04:23.302 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9353000000000001e-05 2024-04-19T01:04:23.302 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.302 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.302 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.302 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.302 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.303 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.3s5 49/31 ECSubWrite(tid=48, reqid=osd.15.0:48, at_version=49'38, trim_to=0'0, roll_forward_to=48'36))", 2024-04-19T01:04:23.303 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:19.533518+0000", 2024-04-19T01:04:23.303 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 3.685995637, 2024-04-19T01:04:23.303 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011056513, 2024-04-19T01:04:23.303 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:23.303 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:23.303 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:23.303 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:23.303 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.303 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:23.303 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.533518+0000", 2024-04-19T01:04:23.303 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.303 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.303 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.303 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:23.304 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.533518+0000", 2024-04-19T01:04:23.304 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.304 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.304 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.304 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:23.304 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.533519+0000", 2024-04-19T01:04:23.304 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4000000000000002e-07 2024-04-19T01:04:23.304 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.304 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.304 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:23.304 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.533543+0000", 2024-04-19T01:04:23.304 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4494999999999998e-05 2024-04-19T01:04:23.304 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.304 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.305 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:23.305 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.533556+0000", 2024-04-19T01:04:23.305 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3145e-05 2024-04-19T01:04:23.305 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.305 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.305 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:23.305 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.533565+0000", 2024-04-19T01:04:23.305 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.4330000000000004e-06 2024-04-19T01:04:23.305 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.305 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.305 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.306 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.533697+0000", 2024-04-19T01:04:23.306 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013253399999999999 2024-04-19T01:04:23.306 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.306 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.306 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:23.306 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.533714+0000", 2024-04-19T01:04:23.306 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7163000000000002e-05 2024-04-19T01:04:23.306 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.306 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.306 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:23.306 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.544518+0000", 2024-04-19T01:04:23.306 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010803063999999999 2024-04-19T01:04:23.306 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.307 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.307 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:23.307 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.544575+0000", 2024-04-19T01:04:23.307 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.7238999999999999e-05 2024-04-19T01:04:23.307 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.307 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.307 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.307 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.307 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.307 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.3s5 49/31 ECSubWrite(tid=49, reqid=osd.15.0:49, at_version=49'40, trim_to=0'0, roll_forward_to=48'36))", 2024-04-19T01:04:23.307 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:19.535950+0000", 2024-04-19T01:04:23.307 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 3.683563736, 2024-04-19T01:04:23.307 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0096398579999999994, 2024-04-19T01:04:23.307 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:23.308 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:23.308 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:23.308 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:23.309 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.309 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:23.310 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.535950+0000", 2024-04-19T01:04:23.310 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.310 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.310 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.310 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:23.310 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.535950+0000", 2024-04-19T01:04:23.310 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.310 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.310 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.310 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:23.311 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.535952+0000", 2024-04-19T01:04:23.311 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4249999999999999e-06 2024-04-19T01:04:23.311 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.311 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.311 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:23.311 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.535984+0000", 2024-04-19T01:04:23.311 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2076000000000003e-05 2024-04-19T01:04:23.311 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.311 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.311 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:23.311 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.536002+0000", 2024-04-19T01:04:23.311 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8252000000000001e-05 2024-04-19T01:04:23.311 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.311 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.312 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:23.312 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.536015+0000", 2024-04-19T01:04:23.312 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3358e-05 2024-04-19T01:04:23.312 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.312 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.312 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.312 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.536116+0000", 2024-04-19T01:04:23.312 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000101105 2024-04-19T01:04:23.312 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.312 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.312 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:23.312 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.536132+0000", 2024-04-19T01:04:23.312 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5495999999999999e-05 2024-04-19T01:04:23.312 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.313 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.313 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:23.313 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.545543+0000", 2024-04-19T01:04:23.313 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0094109139999999994 2024-04-19T01:04:23.313 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.313 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.313 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:23.313 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.545590+0000", 2024-04-19T01:04:23.313 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7231999999999998e-05 2024-04-19T01:04:23.313 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.313 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.313 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.313 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.313 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.314 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.es3 49/31 ECSubWrite(tid=125, reqid=osd.13.0:125, at_version=49'28, trim_to=0'0, roll_forward_to=48'26))", 2024-04-19T01:04:23.314 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:19.538199+0000", 2024-04-19T01:04:23.314 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 3.681314441, 2024-04-19T01:04:23.314 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0081672089999999999, 2024-04-19T01:04:23.314 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:23.314 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:23.314 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:23.314 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:23.314 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.314 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:23.314 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538199+0000", 2024-04-19T01:04:23.315 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.315 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.315 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.315 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:23.315 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538199+0000", 2024-04-19T01:04:23.315 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.315 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.315 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.315 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:23.315 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538200+0000", 2024-04-19T01:04:23.315 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8799999999999998e-07 2024-04-19T01:04:23.315 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.315 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.315 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:23.315 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538236+0000", 2024-04-19T01:04:23.316 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.6575999999999997e-05 2024-04-19T01:04:23.316 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.316 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.316 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:23.316 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538257+0000", 2024-04-19T01:04:23.316 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0874999999999999e-05 2024-04-19T01:04:23.316 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.316 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.316 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:23.316 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538275+0000", 2024-04-19T01:04:23.316 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7745999999999999e-05 2024-04-19T01:04:23.316 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.316 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.316 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.317 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538356+0000", 2024-04-19T01:04:23.317 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1361999999999998e-05 2024-04-19T01:04:23.317 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.317 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.317 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:23.317 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538366+0000", 2024-04-19T01:04:23.317 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.9809999999999998e-06 2024-04-19T01:04:23.317 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.317 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.317 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:23.317 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.546323+0000", 2024-04-19T01:04:23.317 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0079562420000000005 2024-04-19T01:04:23.318 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.318 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.318 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:23.318 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.546367+0000", 2024-04-19T01:04:23.318 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4039000000000003e-05 2024-04-19T01:04:23.318 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.318 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.318 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.318 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.318 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.318 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.9s4 49/31 ECSubWrite(tid=124, reqid=osd.13.0:124, at_version=49'75, trim_to=0'0, roll_forward_to=48'73))", 2024-04-19T01:04:23.318 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:19.538317+0000", 2024-04-19T01:04:23.319 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 3.6811971749999999, 2024-04-19T01:04:23.319 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0082727589999999993, 2024-04-19T01:04:23.319 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:23.319 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:23.319 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:23.319 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:23.319 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.319 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:23.319 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538317+0000", 2024-04-19T01:04:23.320 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.321 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.321 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.321 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:23.321 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538317+0000", 2024-04-19T01:04:23.321 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.321 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.321 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.321 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:23.321 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538317+0000", 2024-04-19T01:04:23.321 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1200000000000003e-07 2024-04-19T01:04:23.321 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.321 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.321 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:23.322 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538344+0000", 2024-04-19T01:04:23.322 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6293000000000001e-05 2024-04-19T01:04:23.322 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.322 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.322 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:23.322 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538360+0000", 2024-04-19T01:04:23.322 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6761000000000001e-05 2024-04-19T01:04:23.322 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.322 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.322 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:23.322 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538374+0000", 2024-04-19T01:04:23.322 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4017e-05 2024-04-19T01:04:23.322 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.323 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.323 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.323 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538500+0000", 2024-04-19T01:04:23.323 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000125587 2024-04-19T01:04:23.323 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.323 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.323 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:23.323 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538510+0000", 2024-04-19T01:04:23.323 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.046e-05 2024-04-19T01:04:23.323 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.323 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.323 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:23.323 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.546545+0000", 2024-04-19T01:04:23.323 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0080348729999999997 2024-04-19T01:04:23.324 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.324 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.324 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:23.324 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.546589+0000", 2024-04-19T01:04:23.324 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4156000000000002e-05 2024-04-19T01:04:23.324 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.324 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.324 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.324 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.324 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.324 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.0s2 49/31 ECSubWrite(tid=39, reqid=osd.1.0:39, at_version=49'35, trim_to=0'0, roll_forward_to=48'33))", 2024-04-19T01:04:23.324 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:19.539477+0000", 2024-04-19T01:04:23.324 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 3.680036689, 2024-04-19T01:04:23.325 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0068473759999999996, 2024-04-19T01:04:23.325 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:23.325 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:23.325 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:23.325 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:23.325 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.325 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:23.325 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.539477+0000", 2024-04-19T01:04:23.325 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.325 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.325 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.325 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:23.325 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.539477+0000", 2024-04-19T01:04:23.326 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.326 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.326 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.326 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:23.326 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.539478+0000", 2024-04-19T01:04:23.326 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9099999999999999e-07 2024-04-19T01:04:23.326 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.326 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.327 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:23.327 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.539511+0000", 2024-04-19T01:04:23.327 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.3732000000000002e-05 2024-04-19T01:04:23.327 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.327 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.327 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:23.327 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.539530+0000", 2024-04-19T01:04:23.327 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8765000000000001e-05 2024-04-19T01:04:23.327 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.327 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.327 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:23.327 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.539566+0000", 2024-04-19T01:04:23.327 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5482999999999999e-05 2024-04-19T01:04:23.328 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.328 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.328 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.328 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.539713+0000", 2024-04-19T01:04:23.328 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000147193 2024-04-19T01:04:23.328 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.328 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.328 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:23.328 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.539729+0000", 2024-04-19T01:04:23.328 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.666e-05 2024-04-19T01:04:23.328 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.328 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.328 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:23.328 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.546279+0000", 2024-04-19T01:04:23.329 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0065497539999999996 2024-04-19T01:04:23.329 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.329 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.329 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:23.329 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.546325+0000", 2024-04-19T01:04:23.329 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5398000000000001e-05 2024-04-19T01:04:23.329 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.329 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.329 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.329 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.329 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.329 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.9s4 49/31 ECSubWrite(tid=126, reqid=osd.13.0:126, at_version=49'77, trim_to=0'0, roll_forward_to=48'73))", 2024-04-19T01:04:23.329 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:19.539763+0000", 2024-04-19T01:04:23.329 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 3.679751327, 2024-04-19T01:04:23.329 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0076408589999999998, 2024-04-19T01:04:23.330 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:23.330 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:23.330 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:23.330 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:23.330 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.330 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:23.330 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.539763+0000", 2024-04-19T01:04:23.331 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.331 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.331 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.331 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:23.331 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.539763+0000", 2024-04-19T01:04:23.331 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.331 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.331 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.331 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:23.332 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.539764+0000", 2024-04-19T01:04:23.332 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.3900000000000003e-07 2024-04-19T01:04:23.332 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.332 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.332 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:23.332 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.539791+0000", 2024-04-19T01:04:23.332 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7328999999999999e-05 2024-04-19T01:04:23.332 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.332 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.332 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:23.332 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.539808+0000", 2024-04-19T01:04:23.332 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6960999999999999e-05 2024-04-19T01:04:23.332 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.333 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.333 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:23.333 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.539817+0000", 2024-04-19T01:04:23.333 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.1789999999999997e-06 2024-04-19T01:04:23.333 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.333 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.333 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.333 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.544922+0000", 2024-04-19T01:04:23.333 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0051049770000000001 2024-04-19T01:04:23.333 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.333 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.333 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:23.333 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.544936+0000", 2024-04-19T01:04:23.334 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3712e-05 2024-04-19T01:04:23.334 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.334 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.334 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:23.334 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.547361+0000", 2024-04-19T01:04:23.334 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0024257039999999999 2024-04-19T01:04:23.334 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.334 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.334 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:23.334 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.547403+0000", 2024-04-19T01:04:23.334 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2058000000000003e-05 2024-04-19T01:04:23.334 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.334 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.335 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.335 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.335 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.335 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.9s4 49/31 ECSubWrite(tid=128, reqid=osd.13.0:126, at_version=0'0, trim_to=0'0, roll_forward_to=49'77))", 2024-04-19T01:04:23.335 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:19.553758+0000", 2024-04-19T01:04:23.335 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 3.6657561240000001, 2024-04-19T01:04:23.335 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0062653450000000003, 2024-04-19T01:04:23.335 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:23.335 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:23.335 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:23.335 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:23.335 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.335 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:23.335 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.553758+0000", 2024-04-19T01:04:23.336 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.336 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.336 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.336 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:23.336 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.553758+0000", 2024-04-19T01:04:23.336 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.336 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.336 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.336 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:23.336 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.553759+0000", 2024-04-19T01:04:23.336 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.8700000000000005e-07 2024-04-19T01:04:23.336 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.336 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.336 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:23.336 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.553770+0000", 2024-04-19T01:04:23.337 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1158999999999999e-05 2024-04-19T01:04:23.337 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.337 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.337 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:23.337 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.553785+0000", 2024-04-19T01:04:23.337 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5693e-05 2024-04-19T01:04:23.337 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.337 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.337 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:23.337 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.553796+0000", 2024-04-19T01:04:23.337 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.096e-05 2024-04-19T01:04:23.337 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.337 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.337 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.337 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.553861+0000", 2024-04-19T01:04:23.338 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4727000000000006e-05 2024-04-19T01:04:23.338 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.338 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.338 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:23.338 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.553871+0000", 2024-04-19T01:04:23.338 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4469999999999995e-06 2024-04-19T01:04:23.338 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.338 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.338 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:23.338 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.559982+0000", 2024-04-19T01:04:23.338 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0061114170000000001 2024-04-19T01:04:23.338 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.339 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.339 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:23.339 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.560023+0000", 2024-04-19T01:04:23.339 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1155000000000002e-05 2024-04-19T01:04:23.339 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.339 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.339 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.339 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.339 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.339 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:23.341 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-19T01:04:23.342 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:04:23.340+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:04:23.441 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:23.444+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:23.471 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:23.471 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:04:23.472 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2024-04-19T01:04:23.472 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:23.486 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-19T01:04:23.505 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2024-04-19T01:04:23.505 INFO:teuthology.orchestra.run.smithi033.stderr:set pool 3 pgp_num to 16 2024-04-19T01:04:23.511 INFO:tasks.rados.rados.0.smithi155.stdout:147: done (1 left) 2024-04-19T01:04:23.511 INFO:tasks.rados.rados.0.smithi155.stdout:148: snap_create 2024-04-19T01:04:23.511 INFO:tasks.rados.rados.0.smithi155.stdout:144: finishing copy_from racing read to smithi15528762-26 2024-04-19T01:04:23.512 INFO:tasks.rados.rados.0.smithi155.stdout:144: finishing copy_from to smithi15528762-26 2024-04-19T01:04:23.512 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 26 v 30 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-19T01:04:23.542 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:04:23.544+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:04:23.609 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:23.610 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:04:23.610 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2024-04-19T01:04:23.610 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2024-04-19T01:04:23.610 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:23.624 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-19T01:04:23.644 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:04:23.643+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:04:23.745 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:04:23.743+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:04:23.792 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:23.792 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2024-04-19T01:04:23.792 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2024-04-19T01:04:23.793 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-04-19T01:04:23.793 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.793 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:230 3.5s0 3:a069ec9a:::smithi15528762-39:head [stat,delete] snapc 7=[7,6,4,2] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-19T01:04:23.793 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:12.831192+0000", 2024-04-19T01:04:23.793 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 10.959744283999999, 2024-04-19T01:04:23.793 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0097994629999999996, 2024-04-19T01:04:23.793 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:23.793 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:23.793 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:23.793 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:23.793 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:23.793 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:23.794 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 230 2024-04-19T01:04:23.794 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.794 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:23.794 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.794 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:23.794 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:12.831192+0000", 2024-04-19T01:04:23.794 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.794 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.794 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.794 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:23.794 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:12.831192+0000", 2024-04-19T01:04:23.794 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.795 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.795 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.795 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:23.795 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:12.831196+0000", 2024-04-19T01:04:23.795 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5449999999999999e-06 2024-04-19T01:04:23.795 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.795 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.795 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:23.795 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:12.831227+0000", 2024-04-19T01:04:23.795 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.1189999999999998e-05 2024-04-19T01:04:23.795 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.795 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.795 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:23.795 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:12.831267+0000", 2024-04-19T01:04:23.796 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9873999999999998e-05 2024-04-19T01:04:23.796 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.796 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.796 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:23.796 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:12.831282+0000", 2024-04-19T01:04:23.796 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5233e-05 2024-04-19T01:04:23.796 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.796 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.796 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.796 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:12.831374+0000", 2024-04-19T01:04:23.796 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2134999999999998e-05 2024-04-19T01:04:23.796 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.797 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.797 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op must wait for map", 2024-04-19T01:04:23.797 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:12.831382+0000", 2024-04-19T01:04:23.797 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.3579999999999997e-06 2024-04-19T01:04:23.797 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.797 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.797 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.797 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:12.834147+0000", 2024-04-19T01:04:23.797 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0027649049999999998 2024-04-19T01:04:23.799 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.799 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.800 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:23.800 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:12.834409+0000", 2024-04-19T01:04:23.800 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00026196500000000002 2024-04-19T01:04:23.800 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.800 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.800 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:23.800 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:12.836737+0000", 2024-04-19T01:04:23.800 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0023288940000000002 2024-04-19T01:04:23.800 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.801 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.801 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:23.801 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:12.838724+0000", 2024-04-19T01:04:23.801 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0019870080000000002 2024-04-19T01:04:23.801 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.801 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.802 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:23.805 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:12.840917+0000", 2024-04-19T01:04:23.805 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0021923329999999999 2024-04-19T01:04:23.805 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.805 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.805 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:23.805 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:12.840992+0000", 2024-04-19T01:04:23.805 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5023000000000003e-05 2024-04-19T01:04:23.805 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.805 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.805 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.805 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.806 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.806 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:251 3.5s0 3:a8ddfdee:::smithi15528762-22:head [copy-from ver 6] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:23.806 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.978778+0000", 2024-04-19T01:04:23.806 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 9.8121589979999992, 2024-04-19T01:04:23.806 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.036510621, 2024-04-19T01:04:23.806 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:23.806 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:23.806 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:23.806 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:23.806 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:23.806 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:23.806 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 251 2024-04-19T01:04:23.806 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.806 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:23.807 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.807 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:23.807 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.978778+0000", 2024-04-19T01:04:23.807 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.807 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.807 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.807 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:23.807 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.978778+0000", 2024-04-19T01:04:23.807 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.807 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.807 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.807 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:23.807 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.978780+0000", 2024-04-19T01:04:23.807 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4729999999999999e-06 2024-04-19T01:04:23.808 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.808 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.808 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:23.809 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.978790+0000", 2024-04-19T01:04:23.810 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0319999999999999e-05 2024-04-19T01:04:23.810 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.810 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.810 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:23.810 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.978814+0000", 2024-04-19T01:04:23.810 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3805999999999999e-05 2024-04-19T01:04:23.810 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.810 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.810 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:23.810 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.978828+0000", 2024-04-19T01:04:23.810 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3594000000000001e-05 2024-04-19T01:04:23.810 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.811 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.811 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.811 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.978916+0000", 2024-04-19T01:04:23.811 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.8331000000000005e-05 2024-04-19T01:04:23.811 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.811 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.811 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:23.811 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.979042+0000", 2024-04-19T01:04:23.811 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000125592 2024-04-19T01:04:23.811 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.811 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.812 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:23.812 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.984647+0000", 2024-04-19T01:04:23.812 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0056049150000000002 2024-04-19T01:04:23.812 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.812 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.812 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:23.812 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.011228+0000", 2024-04-19T01:04:23.812 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.026581707 2024-04-19T01:04:23.812 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.812 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.812 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:23.812 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.015199+0000", 2024-04-19T01:04:23.813 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0039706569999999998 2024-04-19T01:04:23.813 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.813 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.813 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:23.813 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.015288+0000", 2024-04-19T01:04:23.813 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9226000000000003e-05 2024-04-19T01:04:23.813 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.813 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.813 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.813 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.813 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.814 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:252 3.5s0 3:a8ddfdee:::smithi15528762-22:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:23.815 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.978852+0000", 2024-04-19T01:04:23.815 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 9.8120851289999997, 2024-04-19T01:04:23.815 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.024670299, 2024-04-19T01:04:23.815 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:23.815 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:23.815 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:23.815 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:23.815 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:23.816 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:23.816 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 252 2024-04-19T01:04:23.816 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.816 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:23.816 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.816 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:23.816 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.978852+0000", 2024-04-19T01:04:23.816 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.816 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.816 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.816 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:23.816 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.978852+0000", 2024-04-19T01:04:23.816 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.816 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.816 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.817 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:23.817 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.978852+0000", 2024-04-19T01:04:23.817 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8899999999999999e-07 2024-04-19T01:04:23.817 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.817 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.817 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:23.818 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.978855+0000", 2024-04-19T01:04:23.818 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.1310000000000002e-06 2024-04-19T01:04:23.818 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.818 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.818 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:23.818 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.978870+0000", 2024-04-19T01:04:23.819 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4994000000000001e-05 2024-04-19T01:04:23.819 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.820 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.820 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:23.820 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.978875+0000", 2024-04-19T01:04:23.820 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6179999999999997e-06 2024-04-19T01:04:23.820 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.820 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.820 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.821 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.979272+0000", 2024-04-19T01:04:23.821 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00039709000000000001 2024-04-19T01:04:23.821 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.821 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.821 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for blocked object", 2024-04-19T01:04:23.821 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.979339+0000", 2024-04-19T01:04:23.821 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.7161000000000003e-05 2024-04-19T01:04:23.821 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.821 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.821 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.821 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.003307+0000", 2024-04-19T01:04:23.821 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.023967677999999999 2024-04-19T01:04:23.821 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.821 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.821 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:23.822 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.003402+0000", 2024-04-19T01:04:23.822 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.5583999999999998e-05 2024-04-19T01:04:23.822 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.822 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.822 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:23.822 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.003522+0000", 2024-04-19T01:04:23.822 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000119454 2024-04-19T01:04:23.822 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.822 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.822 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.822 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.822 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.822 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:257 3.5s0 3:a03c5c89:::smithi15528762-37:head [read 0~0,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:23.822 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.985133+0000", 2024-04-19T01:04:23.823 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 9.8058039509999997, 2024-04-19T01:04:23.823 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018534069, 2024-04-19T01:04:23.823 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:23.823 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:23.823 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:23.823 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:23.823 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:23.823 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:23.823 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 257 2024-04-19T01:04:23.824 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.824 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:23.824 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.824 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:23.824 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985133+0000", 2024-04-19T01:04:23.824 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.824 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.824 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.824 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:23.824 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985133+0000", 2024-04-19T01:04:23.824 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.824 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.824 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.825 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:23.825 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985135+0000", 2024-04-19T01:04:23.825 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8840000000000001e-06 2024-04-19T01:04:23.825 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.825 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.825 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:23.825 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985143+0000", 2024-04-19T01:04:23.825 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.8140000000000007e-06 2024-04-19T01:04:23.825 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.826 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.826 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:23.826 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985162+0000", 2024-04-19T01:04:23.826 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8139999999999999e-05 2024-04-19T01:04:23.826 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.826 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.826 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:23.827 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985174+0000", 2024-04-19T01:04:23.827 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2208000000000001e-05 2024-04-19T01:04:23.827 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.827 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.827 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.827 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.003571+0000", 2024-04-19T01:04:23.827 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018396843999999999 2024-04-19T01:04:23.827 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.827 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.827 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:23.827 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.003667+0000", 2024-04-19T01:04:23.827 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.6179000000000002e-05 2024-04-19T01:04:23.827 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.827 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.828 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.828 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.828 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.828 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:258 3.5s0 3:a03c5c89:::smithi15528762-37:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:23.828 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.985209+0000", 2024-04-19T01:04:23.828 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 9.8057281590000009, 2024-04-19T01:04:23.828 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.026491132000000001, 2024-04-19T01:04:23.828 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:23.828 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:23.828 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:23.828 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:23.828 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:23.828 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:23.828 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 258 2024-04-19T01:04:23.828 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.829 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:23.829 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.829 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:23.829 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985209+0000", 2024-04-19T01:04:23.829 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.829 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.829 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.829 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:23.829 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985209+0000", 2024-04-19T01:04:23.829 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.830 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.830 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.830 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:23.830 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985209+0000", 2024-04-19T01:04:23.830 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2600000000000002e-07 2024-04-19T01:04:23.830 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.830 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.830 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:23.830 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985214+0000", 2024-04-19T01:04:23.830 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7110000000000003e-06 2024-04-19T01:04:23.830 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.830 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.830 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:23.830 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985225+0000", 2024-04-19T01:04:23.830 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1382e-05 2024-04-19T01:04:23.831 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.831 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.831 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:23.831 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985233+0000", 2024-04-19T01:04:23.831 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.418e-06 2024-04-19T01:04:23.831 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.831 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.831 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.831 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.011520+0000", 2024-04-19T01:04:23.831 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.026287015 2024-04-19T01:04:23.831 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.831 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.831 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:23.831 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.011700+0000", 2024-04-19T01:04:23.831 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00017987999999999999 2024-04-19T01:04:23.832 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.832 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.832 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.832 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.832 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.832 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:259 3.5s0 3:a03c5c89:::smithi15528762-37:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:23.832 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.985245+0000", 2024-04-19T01:04:23.832 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 9.8056920390000002, 2024-04-19T01:04:23.832 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.026550810000000001, 2024-04-19T01:04:23.832 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:23.832 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:23.832 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:23.832 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:23.832 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:23.832 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:23.833 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 259 2024-04-19T01:04:23.833 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.833 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:23.833 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.833 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:23.833 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985245+0000", 2024-04-19T01:04:23.833 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.833 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.833 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.833 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:23.833 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985245+0000", 2024-04-19T01:04:23.833 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.833 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.833 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.834 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:23.834 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985245+0000", 2024-04-19T01:04:23.834 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6000000000000004e-07 2024-04-19T01:04:23.834 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.834 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.834 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:23.834 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985249+0000", 2024-04-19T01:04:23.834 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.036e-06 2024-04-19T01:04:23.834 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.834 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.834 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:23.834 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985258+0000", 2024-04-19T01:04:23.834 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.0569999999999999e-06 2024-04-19T01:04:23.835 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.835 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.835 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:23.835 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985266+0000", 2024-04-19T01:04:23.835 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.0339999999999997e-06 2024-04-19T01:04:23.835 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.835 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.835 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.835 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.011729+0000", 2024-04-19T01:04:23.835 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.02646308 2024-04-19T01:04:23.835 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.835 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.835 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:23.835 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.011795+0000", 2024-04-19T01:04:23.836 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.6043000000000004e-05 2024-04-19T01:04:23.836 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.836 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.836 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.836 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.836 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.836 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:260 3.5s0 3:a626fd0d:::smithi15528762-17:head [read 0~278528,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:23.836 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.985277+0000", 2024-04-19T01:04:23.836 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 9.8056600730000003, 2024-04-19T01:04:23.836 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.029721959999999999, 2024-04-19T01:04:23.836 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:23.836 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:23.836 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:23.836 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:23.836 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:23.837 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:23.837 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 260 2024-04-19T01:04:23.837 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.837 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:23.837 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.837 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:23.837 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985277+0000", 2024-04-19T01:04:23.837 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.837 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.837 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.837 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:23.837 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985277+0000", 2024-04-19T01:04:23.837 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.837 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.837 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.838 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:23.838 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985277+0000", 2024-04-19T01:04:23.838 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8800000000000002e-07 2024-04-19T01:04:23.838 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.838 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.838 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:23.838 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985286+0000", 2024-04-19T01:04:23.838 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.8009999999999993e-06 2024-04-19T01:04:23.838 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.838 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.838 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:23.838 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985297+0000", 2024-04-19T01:04:23.838 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1321000000000001e-05 2024-04-19T01:04:23.838 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.839 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.839 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:23.839 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985305+0000", 2024-04-19T01:04:23.839 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.3390000000000004e-06 2024-04-19T01:04:23.839 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.839 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.839 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.839 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.011821+0000", 2024-04-19T01:04:23.839 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.026516310000000001 2024-04-19T01:04:23.839 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.839 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.839 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:23.839 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.011905+0000", 2024-04-19T01:04:23.839 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.4538999999999998e-05 2024-04-19T01:04:23.839 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.840 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.840 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:23.840 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.014999+0000", 2024-04-19T01:04:23.840 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0030930620000000002 2024-04-19T01:04:23.840 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.840 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.840 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.840 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.840 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.840 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:261 3.5s0 3:a626fd0d:::smithi15528762-17:head [read 0~278528,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:23.840 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.985317+0000", 2024-04-19T01:04:23.840 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 9.8056196349999993, 2024-04-19T01:04:23.840 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.030889950999999999, 2024-04-19T01:04:23.840 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:23.841 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:23.841 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:23.841 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:23.841 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:23.841 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:23.841 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 261 2024-04-19T01:04:23.841 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.841 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:23.841 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.841 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:23.841 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985317+0000", 2024-04-19T01:04:23.841 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.841 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.841 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.841 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:23.842 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985317+0000", 2024-04-19T01:04:23.842 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.842 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.842 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.842 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:23.842 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985318+0000", 2024-04-19T01:04:23.842 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6100000000000001e-07 2024-04-19T01:04:23.842 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.842 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.842 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:23.842 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985321+0000", 2024-04-19T01:04:23.842 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5930000000000002e-06 2024-04-19T01:04:23.842 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.843 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.843 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:23.843 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985329+0000", 2024-04-19T01:04:23.843 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.9699999999999999e-06 2024-04-19T01:04:23.843 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.843 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.843 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:23.843 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985336+0000", 2024-04-19T01:04:23.843 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5300000000000002e-06 2024-04-19T01:04:23.843 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.843 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.843 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.843 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.012184+0000", 2024-04-19T01:04:23.843 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.026848627 2024-04-19T01:04:23.843 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.844 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.844 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:23.844 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.012257+0000", 2024-04-19T01:04:23.844 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2534999999999996e-05 2024-04-19T01:04:23.844 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.844 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.845 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:23.845 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.016207+0000", 2024-04-19T01:04:23.845 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0039501349999999996 2024-04-19T01:04:23.845 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.845 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.845 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.845 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.846 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.846 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:262 3.5s0 3:a626fd0d:::smithi15528762-17:head [read 0~278528,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:23.846 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.985345+0000", 2024-04-19T01:04:23.846 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 9.8055919869999997, 2024-04-19T01:04:23.846 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.031344419999999998, 2024-04-19T01:04:23.846 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:23.846 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:23.846 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:23.846 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:23.846 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:23.846 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:23.846 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 262 2024-04-19T01:04:23.846 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.846 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:23.847 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.847 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:23.847 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985345+0000", 2024-04-19T01:04:23.847 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.847 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.847 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.847 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:23.847 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985345+0000", 2024-04-19T01:04:23.847 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.847 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.847 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.847 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:23.847 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985345+0000", 2024-04-19T01:04:23.847 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3799999999999997e-07 2024-04-19T01:04:23.847 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.848 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.848 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:23.848 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985358+0000", 2024-04-19T01:04:23.848 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2969999999999999e-05 2024-04-19T01:04:23.848 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.848 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.848 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:23.848 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985369+0000", 2024-04-19T01:04:23.848 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0472000000000001e-05 2024-04-19T01:04:23.848 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.848 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.848 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:23.848 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985378+0000", 2024-04-19T01:04:23.848 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.8019999999999998e-06 2024-04-19T01:04:23.848 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.849 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.849 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.849 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.012481+0000", 2024-04-19T01:04:23.849 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.027103136 2024-04-19T01:04:23.849 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.849 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.849 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:23.849 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.012531+0000", 2024-04-19T01:04:23.849 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0189999999999999e-05 2024-04-19T01:04:23.849 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.849 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.849 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:23.849 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.016689+0000", 2024-04-19T01:04:23.849 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0041582119999999997 2024-04-19T01:04:23.850 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.850 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.850 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.850 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.850 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.850 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds4 43/31 ECSubWrite(tid=38, reqid=client.4604.0:253, at_version=43'14, trim_to=0'0, roll_forward_to=43'12))", 2024-04-19T01:04:23.850 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.991581+0000", 2024-04-19T01:04:23.850 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 9.7993559179999998, 2024-04-19T01:04:23.850 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.019891641000000002, 2024-04-19T01:04:23.850 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:23.850 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:23.850 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:23.851 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:23.851 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.851 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:23.851 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991581+0000", 2024-04-19T01:04:23.851 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.851 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.851 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.851 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:23.851 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991581+0000", 2024-04-19T01:04:23.851 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.851 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.851 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.851 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:23.851 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991582+0000", 2024-04-19T01:04:23.851 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7430000000000001e-06 2024-04-19T01:04:23.852 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.852 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.852 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:23.852 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991739+0000", 2024-04-19T01:04:23.852 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015636 2024-04-19T01:04:23.852 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.852 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.852 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:23.852 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991764+0000", 2024-04-19T01:04:23.852 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5626000000000002e-05 2024-04-19T01:04:23.852 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.852 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.852 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:23.852 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991776+0000", 2024-04-19T01:04:23.853 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1154e-05 2024-04-19T01:04:23.853 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.853 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.853 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.853 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.003746+0000", 2024-04-19T01:04:23.853 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011970404 2024-04-19T01:04:23.853 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.853 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.853 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:23.853 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.003754+0000", 2024-04-19T01:04:23.853 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.9270000000000005e-06 2024-04-19T01:04:23.853 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.853 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.853 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:23.853 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.011430+0000", 2024-04-19T01:04:23.854 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0076758590000000002 2024-04-19T01:04:23.854 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.854 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.854 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:23.854 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.011472+0000", 2024-04-19T01:04:23.854 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2568000000000003e-05 2024-04-19T01:04:23.854 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.854 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.854 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.854 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.854 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.854 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds4 43/31 ECSubWrite(tid=39, reqid=client.4604.0:254, at_version=43'15, trim_to=0'0, roll_forward_to=43'12))", 2024-04-19T01:04:23.854 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.996147+0000", 2024-04-19T01:04:23.854 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 9.7947894689999995, 2024-04-19T01:04:23.854 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017030894000000001, 2024-04-19T01:04:23.855 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:23.855 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:23.855 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:23.855 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:23.855 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.855 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:23.855 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.996147+0000", 2024-04-19T01:04:23.855 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.855 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.855 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.855 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:23.855 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.996147+0000", 2024-04-19T01:04:23.855 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.855 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.855 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.856 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:23.856 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.996149+0000", 2024-04-19T01:04:23.856 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5319999999999999e-06 2024-04-19T01:04:23.856 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.856 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.856 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:23.856 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.996208+0000", 2024-04-19T01:04:23.856 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8924e-05 2024-04-19T01:04:23.856 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.856 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.856 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:23.856 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.996230+0000", 2024-04-19T01:04:23.856 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.2208000000000002e-05 2024-04-19T01:04:23.856 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.856 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.857 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:23.857 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.996242+0000", 2024-04-19T01:04:23.857 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1871e-05 2024-04-19T01:04:23.857 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.857 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.857 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.857 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.008733+0000", 2024-04-19T01:04:23.857 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012491189999999999 2024-04-19T01:04:23.857 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.858 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.858 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:23.858 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.008743+0000", 2024-04-19T01:04:23.858 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0448000000000001e-05 2024-04-19T01:04:23.858 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.858 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.858 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:23.858 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.013128+0000", 2024-04-19T01:04:23.858 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0043847499999999998 2024-04-19T01:04:23.858 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.858 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.858 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:23.858 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.013178+0000", 2024-04-19T01:04:23.858 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.9971e-05 2024-04-19T01:04:23.859 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.859 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.859 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.859 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.859 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.859 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds4 43/31 ECSubWrite(tid=40, reqid=client.4604.0:255, at_version=43'16, trim_to=0'0, roll_forward_to=43'12))", 2024-04-19T01:04:23.859 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:14.005080+0000", 2024-04-19T01:04:23.859 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 9.7858568869999996, 2024-04-19T01:04:23.859 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0081245250000000005, 2024-04-19T01:04:23.859 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:23.859 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:23.859 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:23.859 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:23.859 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.860 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:23.860 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.005080+0000", 2024-04-19T01:04:23.860 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.860 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.860 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.860 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:23.860 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.005080+0000", 2024-04-19T01:04:23.860 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.860 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.860 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.860 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:23.860 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.005081+0000", 2024-04-19T01:04:23.860 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.198e-06 2024-04-19T01:04:23.860 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.861 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.861 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:23.861 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.005100+0000", 2024-04-19T01:04:23.861 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9374000000000001e-05 2024-04-19T01:04:23.861 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.861 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.861 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:23.861 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.005113+0000", 2024-04-19T01:04:23.861 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2503999999999999e-05 2024-04-19T01:04:23.861 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.861 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.861 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:23.861 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.005121+0000", 2024-04-19T01:04:23.861 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.4249999999999998e-06 2024-04-19T01:04:23.861 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.862 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.862 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.862 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.011220+0000", 2024-04-19T01:04:23.862 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0060991760000000004 2024-04-19T01:04:23.862 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.862 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.863 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:23.863 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.011260+0000", 2024-04-19T01:04:23.863 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9255000000000003e-05 2024-04-19T01:04:23.863 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.863 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.863 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:23.863 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.013182+0000", 2024-04-19T01:04:23.863 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001921797 2024-04-19T01:04:23.863 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.863 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.863 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:23.863 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.013204+0000", 2024-04-19T01:04:23.863 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.2796e-05 2024-04-19T01:04:23.863 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.864 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.864 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.864 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.864 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.864 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:290 3.5s0 3:a626fd0d:::smithi15528762-17:head [create,setxattr Zk\\P^SegWYU[`WTPeUNnQmiTaa_jWqlbUlOOo_`Q (807) in=40b,setxattr WQnfRYk\\Wee]jkWpPVU\\b\\^O`d_aqhhsrNa\\q]` (1683) in=39b,setxattr isjYpf]SpaTmbRX[lPfZYd`iRg^qpPtXgta\\YSeP (1578) in=40b,setxattr Rj`[raddO^W[eaYZe\\p]n (1683) in=21b,setxattr \\b (1683) in=2b,setxattr SUU_aQmekRQTrtncbjnO]YVtnYUiSmRYkZdfVWjn (9612) in=40b,setxattr htY\\NVO]cOXO_S_laj[Tb^UlYeRVeTdgtRihsQNV (1717) in=40b,setxattr cNlYV\\rZSe_tflYeOk_TblX`eVOcfgno_ioWgrZY (875) in=40b,setxattr qS]Oimh`cRqmTYb[qlU[TqXhleWb\\ (1683) in=29b,setxattr NUkmOR`_Ui`rla]jt_Ql[Tfra[NmmgYhj_PTghpt (911) in=40b,setxattr e[kalNoWQnaXoX_kOZap_kXQahW]d[gPb_OgoTtr (1542) in=40b,setxattr rXW]_VkTh]^hcoaOsfUSdi_i[q\\oYfihbhS\\ooV] (210) in=40b,setxattr Na\\ch (1683) in=5b,setxattr f`WaS^^g\\Zc]]\\fgOeYOa_cN_^[lUa[YT`aNlZep (876) in=40b,setxattr ][ebaQ\\NOOfnlZQmROWj_mXld\\XTdfe^bZ (1683) in=34b,setxattr [\\hq^][jRsSfi\\SoVV\\VfOrrdbanYccVh`b`Pmhh (1789) in=40b,setxattr _Zkp`Xt_VWpQVOPk[tlZaRs`[rbSgWjNXP[k\\_O] (15648) in=40b,setxattr \\PnkdXaP]^o_orX[hlrOSrRRPVfReZparfiseVeR (13964) in=40b,setxattr PS]d`lagRXOkQsXaRYVtOYmbonZdcZtmTbXtThNa (1367) in=40b,setxattr VijqchiV`trXRPoVjQV^^XR\\qPPRNWct[RN]sgse (491) in=40b] snapc 7=[7,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-19T01:04:23.864 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:14.917998+0000", 2024-04-19T01:04:23.864 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 8.8729388759999992, 2024-04-19T01:04:23.864 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.01439083, 2024-04-19T01:04:23.864 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:23.864 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:23.864 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:23.864 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:23.864 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:23.864 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:23.865 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 290 2024-04-19T01:04:23.865 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.865 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:23.865 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.865 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:23.865 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.917998+0000", 2024-04-19T01:04:23.865 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.865 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.865 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.865 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:23.865 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.917998+0000", 2024-04-19T01:04:23.865 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.865 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.865 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.865 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:23.866 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.917999+0000", 2024-04-19T01:04:23.866 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5999999999999999e-06 2024-04-19T01:04:23.866 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.866 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.866 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:23.866 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.918166+0000", 2024-04-19T01:04:23.866 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00016676500000000001 2024-04-19T01:04:23.866 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.866 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.866 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:23.866 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.918179+0000", 2024-04-19T01:04:23.866 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2903e-05 2024-04-19T01:04:23.866 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.866 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.867 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:23.867 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.918185+0000", 2024-04-19T01:04:23.867 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8669999999999999e-06 2024-04-19T01:04:23.867 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.867 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.867 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.867 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.918238+0000", 2024-04-19T01:04:23.867 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2833000000000001e-05 2024-04-19T01:04:23.867 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.867 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.867 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:23.867 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.918317+0000", 2024-04-19T01:04:23.867 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.8838000000000006e-05 2024-04-19T01:04:23.867 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.868 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.868 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:23.868 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.918930+0000", 2024-04-19T01:04:23.868 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00061318799999999997 2024-04-19T01:04:23.868 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.868 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.868 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:23.868 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.919877+0000", 2024-04-19T01:04:23.868 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00094760500000000002 2024-04-19T01:04:23.868 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.868 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.868 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:23.868 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.932324+0000", 2024-04-19T01:04:23.868 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012446521 2024-04-19T01:04:23.868 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.869 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.869 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:23.869 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.932389+0000", 2024-04-19T01:04:23.869 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4709999999999995e-05 2024-04-19T01:04:23.869 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.869 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.869 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.869 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.869 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.869 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:342 3.5s0 3:a03c5c89:::smithi15528762-37:head [rollback 7] snapc 9=[8,7] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:04:23.869 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.511159+0000", 2024-04-19T01:04:23.869 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.2797776110000001, 2024-04-19T01:04:23.869 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011730943000000001, 2024-04-19T01:04:23.869 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:23.869 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:23.870 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:23.870 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:23.870 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:23.870 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:23.870 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 342 2024-04-19T01:04:23.870 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.870 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:23.870 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.870 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:23.870 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.511159+0000", 2024-04-19T01:04:23.870 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.870 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.870 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.870 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:23.870 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.511159+0000", 2024-04-19T01:04:23.871 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.871 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.871 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.871 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:23.871 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.511162+0000", 2024-04-19T01:04:23.871 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7650000000000002e-06 2024-04-19T01:04:23.871 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.871 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.871 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:23.871 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.511172+0000", 2024-04-19T01:04:23.871 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.8740000000000007e-06 2024-04-19T01:04:23.871 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.871 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.871 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:23.871 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.511196+0000", 2024-04-19T01:04:23.872 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4298999999999999e-05 2024-04-19T01:04:23.872 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.872 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.872 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:23.872 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.511217+0000", 2024-04-19T01:04:23.872 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1457000000000001e-05 2024-04-19T01:04:23.872 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.872 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.872 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.872 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.511328+0000", 2024-04-19T01:04:23.872 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000110351 2024-04-19T01:04:23.872 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.872 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.872 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op must wait for map", 2024-04-19T01:04:23.873 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.511340+0000", 2024-04-19T01:04:23.873 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2618e-05 2024-04-19T01:04:23.873 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.873 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.873 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.873 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.522524+0000", 2024-04-19T01:04:23.873 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011183571 2024-04-19T01:04:23.873 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.873 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.873 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:23.873 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.522662+0000", 2024-04-19T01:04:23.873 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013795800000000001 2024-04-19T01:04:23.873 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.873 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.873 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:23.874 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.522890+0000", 2024-04-19T01:04:23.874 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00022804999999999999 2024-04-19T01:04:23.874 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.874 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.874 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.874 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.874 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.874 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds4 48/31 ECSubWrite(tid=68, reqid=osd.12.0:68, at_version=48'39, trim_to=0'0, roll_forward_to=47'35))", 2024-04-19T01:04:23.874 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.514395+0000", 2024-04-19T01:04:23.874 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.2765413649999999, 2024-04-19T01:04:23.874 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0091589570000000006, 2024-04-19T01:04:23.874 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:23.874 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:23.874 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:23.874 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:23.875 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.875 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:23.875 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.514395+0000", 2024-04-19T01:04:23.875 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.875 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.875 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.875 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:23.875 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.514395+0000", 2024-04-19T01:04:23.875 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.875 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.875 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.875 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:23.875 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.514396+0000", 2024-04-19T01:04:23.875 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.1299999999999998e-07 2024-04-19T01:04:23.876 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.876 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.876 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:23.876 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.514435+0000", 2024-04-19T01:04:23.876 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8358000000000001e-05 2024-04-19T01:04:23.876 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.876 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.876 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:23.876 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.514454+0000", 2024-04-19T01:04:23.876 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9485000000000001e-05 2024-04-19T01:04:23.876 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.876 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.876 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:23.877 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.514467+0000", 2024-04-19T01:04:23.877 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3383e-05 2024-04-19T01:04:23.877 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.877 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.877 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.877 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.515285+0000", 2024-04-19T01:04:23.877 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00081705899999999997 2024-04-19T01:04:23.877 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.877 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.877 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:23.877 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.515299+0000", 2024-04-19T01:04:23.877 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4861000000000001e-05 2024-04-19T01:04:23.877 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.877 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.877 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:23.878 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.523472+0000", 2024-04-19T01:04:23.878 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0081727320000000003 2024-04-19T01:04:23.878 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.878 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.878 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:23.878 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.523554+0000", 2024-04-19T01:04:23.878 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.2165999999999999e-05 2024-04-19T01:04:23.878 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.878 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.878 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.878 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.878 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.878 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.as6 48/31 ECSubWrite(tid=71, reqid=client.4604.0:354, at_version=48'10, trim_to=0'0, roll_forward_to=34'8))", 2024-04-19T01:04:23.878 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.560650+0000", 2024-04-19T01:04:23.879 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.2302868220000001, 2024-04-19T01:04:23.879 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.020025294999999999, 2024-04-19T01:04:23.879 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:23.879 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:23.879 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:23.879 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:23.879 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.879 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:23.879 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.560650+0000", 2024-04-19T01:04:23.879 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.879 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.879 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.880 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:23.880 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.560650+0000", 2024-04-19T01:04:23.880 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.880 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.880 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.880 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:23.880 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.560651+0000", 2024-04-19T01:04:23.880 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3990000000000001e-06 2024-04-19T01:04:23.880 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.881 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.881 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:23.881 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.561126+0000", 2024-04-19T01:04:23.881 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00047482999999999998 2024-04-19T01:04:23.881 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.881 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.881 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:23.881 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.561147+0000", 2024-04-19T01:04:23.881 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1036e-05 2024-04-19T01:04:23.881 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.881 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.881 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:23.881 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.561163+0000", 2024-04-19T01:04:23.881 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6059e-05 2024-04-19T01:04:23.882 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.882 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.882 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.882 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.561254+0000", 2024-04-19T01:04:23.882 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.0653000000000001e-05 2024-04-19T01:04:23.882 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.882 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.882 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:23.882 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.561268+0000", 2024-04-19T01:04:23.882 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4242e-05 2024-04-19T01:04:23.882 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.882 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.882 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:23.882 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.580610+0000", 2024-04-19T01:04:23.883 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.019342301999999999 2024-04-19T01:04:23.883 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.883 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.883 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:23.883 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.580675+0000", 2024-04-19T01:04:23.883 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4774e-05 2024-04-19T01:04:23.883 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.883 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.883 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.883 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.883 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.883 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.3s1 49/31 ECSubWrite(tid=48, reqid=osd.15.0:48, at_version=49'38, trim_to=0'0, roll_forward_to=48'36))", 2024-04-19T01:04:23.883 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:19.536349+0000", 2024-04-19T01:04:23.884 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 4.2545878300000002, 2024-04-19T01:04:23.884 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0094613549999999994, 2024-04-19T01:04:23.884 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:23.884 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:23.884 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:23.884 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:23.884 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.884 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:23.884 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.536349+0000", 2024-04-19T01:04:23.884 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.884 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.884 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.885 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:23.885 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.536349+0000", 2024-04-19T01:04:23.885 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.885 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.885 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.885 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:23.885 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.536349+0000", 2024-04-19T01:04:23.885 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7599999999999998e-07 2024-04-19T01:04:23.885 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.885 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.885 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:23.885 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.536387+0000", 2024-04-19T01:04:23.885 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7596999999999999e-05 2024-04-19T01:04:23.885 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.885 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.886 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:23.886 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.536411+0000", 2024-04-19T01:04:23.886 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4094e-05 2024-04-19T01:04:23.886 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.886 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.886 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:23.886 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.536425+0000", 2024-04-19T01:04:23.886 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4609e-05 2024-04-19T01:04:23.886 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.886 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.886 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.886 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.536644+0000", 2024-04-19T01:04:23.886 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00021823099999999999 2024-04-19T01:04:23.886 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.887 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.887 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:23.887 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.536660+0000", 2024-04-19T01:04:23.887 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6099999999999998e-05 2024-04-19T01:04:23.887 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.887 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.887 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:23.887 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.545769+0000", 2024-04-19T01:04:23.887 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0091092310000000006 2024-04-19T01:04:23.887 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.887 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.887 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:23.887 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.545810+0000", 2024-04-19T01:04:23.888 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1217000000000002e-05 2024-04-19T01:04:23.888 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.888 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.888 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.888 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.888 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.888 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.3s1 49/31 ECSubWrite(tid=49, reqid=osd.15.0:49, at_version=49'40, trim_to=0'0, roll_forward_to=48'36))", 2024-04-19T01:04:23.888 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:19.536462+0000", 2024-04-19T01:04:23.888 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 4.2544747010000004, 2024-04-19T01:04:23.888 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010056377999999999, 2024-04-19T01:04:23.888 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:23.888 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:23.889 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:23.889 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:23.889 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.889 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:23.889 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.536462+0000", 2024-04-19T01:04:23.889 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.889 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.889 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.889 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:23.889 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.536462+0000", 2024-04-19T01:04:23.889 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.889 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.889 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.889 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:23.889 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.536463+0000", 2024-04-19T01:04:23.890 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3499999999999996e-07 2024-04-19T01:04:23.890 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.890 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.890 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:23.890 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.536487+0000", 2024-04-19T01:04:23.890 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4189000000000002e-05 2024-04-19T01:04:23.890 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.890 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.890 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:23.890 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.536502+0000", 2024-04-19T01:04:23.890 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4914e-05 2024-04-19T01:04:23.890 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.890 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.890 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:23.891 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.536515+0000", 2024-04-19T01:04:23.891 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3597e-05 2024-04-19T01:04:23.891 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.891 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.891 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.891 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.537366+0000", 2024-04-19T01:04:23.891 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00085066500000000001 2024-04-19T01:04:23.891 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.891 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.891 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:23.891 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.537421+0000", 2024-04-19T01:04:23.891 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4633999999999999e-05 2024-04-19T01:04:23.891 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.891 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.891 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:23.892 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.546475+0000", 2024-04-19T01:04:23.892 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0090545409999999993 2024-04-19T01:04:23.892 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.892 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.892 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:23.892 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.546518+0000", 2024-04-19T01:04:23.892 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3303000000000002e-05 2024-04-19T01:04:23.892 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.892 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.892 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.893 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.893 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.893 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.as6 49/31 ECSubWrite(tid=73, reqid=osd.12.0:73, at_version=49'12, trim_to=0'0, roll_forward_to=48'10))", 2024-04-19T01:04:23.893 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:19.537803+0000", 2024-04-19T01:04:23.893 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 4.253133418, 2024-04-19T01:04:23.893 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0086916029999999991, 2024-04-19T01:04:23.893 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:23.893 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:23.893 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:23.893 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:23.893 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.893 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:23.893 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.537803+0000", 2024-04-19T01:04:23.894 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.894 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.894 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.894 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:23.894 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.537803+0000", 2024-04-19T01:04:23.894 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.894 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.894 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.894 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:23.894 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.537804+0000", 2024-04-19T01:04:23.894 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.9000000000000005e-07 2024-04-19T01:04:23.894 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.894 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.894 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:23.894 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.537829+0000", 2024-04-19T01:04:23.895 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4604999999999999e-05 2024-04-19T01:04:23.895 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.895 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.895 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:23.895 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.537844+0000", 2024-04-19T01:04:23.895 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5566999999999999e-05 2024-04-19T01:04:23.895 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.895 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.895 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:23.895 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.537854+0000", 2024-04-19T01:04:23.895 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2469999999999997e-06 2024-04-19T01:04:23.895 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.895 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.895 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.895 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538039+0000", 2024-04-19T01:04:23.896 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00018537800000000001 2024-04-19T01:04:23.896 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.896 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.896 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:23.896 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538053+0000", 2024-04-19T01:04:23.896 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3468e-05 2024-04-19T01:04:23.896 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.896 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.896 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:23.896 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.546456+0000", 2024-04-19T01:04:23.896 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0084033400000000005 2024-04-19T01:04:23.896 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.896 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.896 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:23.896 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.546495+0000", 2024-04-19T01:04:23.897 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9008000000000003e-05 2024-04-19T01:04:23.897 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.897 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.897 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.897 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.897 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.897 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds4 49/31 ECSubWrite(tid=75, reqid=osd.12.0:75, at_version=49'43, trim_to=0'0, roll_forward_to=48'39))", 2024-04-19T01:04:23.897 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:19.538168+0000", 2024-04-19T01:04:23.897 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 4.2527683319999996, 2024-04-19T01:04:23.897 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0083652829999999994, 2024-04-19T01:04:23.897 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:23.897 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:23.897 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:23.897 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:23.897 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.898 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:23.898 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538168+0000", 2024-04-19T01:04:23.898 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.898 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.898 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.898 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:23.898 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538168+0000", 2024-04-19T01:04:23.898 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:23.898 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.898 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.898 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:23.898 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538169+0000", 2024-04-19T01:04:23.898 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.9199999999999999e-07 2024-04-19T01:04:23.898 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.898 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.899 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:23.899 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538193+0000", 2024-04-19T01:04:23.899 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3723e-05 2024-04-19T01:04:23.899 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.899 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.899 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:23.899 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538205+0000", 2024-04-19T01:04:23.899 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1912999999999999e-05 2024-04-19T01:04:23.899 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.899 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.899 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:23.899 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538214+0000", 2024-04-19T01:04:23.899 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.0499999999999997e-06 2024-04-19T01:04:23.899 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.899 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.900 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:23.900 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538332+0000", 2024-04-19T01:04:23.900 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000117878 2024-04-19T01:04:23.900 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.900 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.900 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:23.900 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538341+0000", 2024-04-19T01:04:23.900 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.516e-06 2024-04-19T01:04:23.900 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.900 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.900 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:23.900 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.546508+0000", 2024-04-19T01:04:23.900 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0081663020000000003 2024-04-19T01:04:23.900 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:23.900 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:23.901 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:23.901 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.546534+0000", 2024-04-19T01:04:23.901 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5908999999999999e-05 2024-04-19T01:04:23.901 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.901 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.901 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.901 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:23.901 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:23.901 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:23.903 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-04-19T01:04:23.903 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:04:23.901+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:04:24.003 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:24.004+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:24.076 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:24.076 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:04:24.077 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2024-04-19T01:04:24.077 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:24.090 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-04-19T01:04:24.104 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:04:24.105+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:04:24.205 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:04:24.207+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:04:24.216 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:24.216 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:04:24.217 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2024-04-19T01:04:24.217 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2024-04-19T01:04:24.217 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:24.232 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-04-19T01:04:24.306 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:24.304+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:24.365 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:24.365 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2024-04-19T01:04:24.365 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2024-04-19T01:04:24.365 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-04-19T01:04:24.365 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.365 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.3s6 38/31 ECSubWrite(tid=13, reqid=client.4604.0:173, at_version=38'9, trim_to=0'0, roll_forward_to=34'7))", 2024-04-19T01:04:24.365 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:08.824420+0000", 2024-04-19T01:04:24.366 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 15.538791657999999, 2024-04-19T01:04:24.366 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0060518400000000002, 2024-04-19T01:04:24.366 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:24.366 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:24.366 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:24.366 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:24.366 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.366 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:24.366 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.824420+0000", 2024-04-19T01:04:24.366 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:24.366 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.366 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.367 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:24.367 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.824420+0000", 2024-04-19T01:04:24.367 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:24.367 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.367 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.367 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:24.367 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.824421+0000", 2024-04-19T01:04:24.367 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4499999999999998e-07 2024-04-19T01:04:24.367 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.367 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.367 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:24.367 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.824862+0000", 2024-04-19T01:04:24.367 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00044151800000000001 2024-04-19T01:04:24.367 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.368 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.368 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:24.368 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.824877+0000", 2024-04-19T01:04:24.368 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4914e-05 2024-04-19T01:04:24.368 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.368 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.368 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:24.368 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.824887+0000", 2024-04-19T01:04:24.369 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.6239999999999997e-06 2024-04-19T01:04:24.369 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.369 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.369 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:24.369 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.824994+0000", 2024-04-19T01:04:24.369 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000107435 2024-04-19T01:04:24.369 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.369 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.369 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:24.369 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.825007+0000", 2024-04-19T01:04:24.369 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2490000000000001e-05 2024-04-19T01:04:24.370 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.370 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.370 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:24.370 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.830422+0000", 2024-04-19T01:04:24.370 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0054158670000000004 2024-04-19T01:04:24.370 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.370 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.370 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:24.371 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.830472+0000", 2024-04-19T01:04:24.371 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.9647000000000001e-05 2024-04-19T01:04:24.371 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:24.371 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:24.371 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:24.371 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.371 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.371 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.3s6 38/31 ECSubWrite(tid=14, reqid=client.4604.0:174, at_version=38'10, trim_to=0'0, roll_forward_to=34'7))", 2024-04-19T01:04:24.371 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:08.832936+0000", 2024-04-19T01:04:24.371 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 15.530275514, 2024-04-19T01:04:24.371 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0075292629999999996, 2024-04-19T01:04:24.371 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:24.371 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:24.372 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:24.372 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:24.372 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.372 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:24.372 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.832936+0000", 2024-04-19T01:04:24.372 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:24.372 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.372 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.372 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:24.372 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.832936+0000", 2024-04-19T01:04:24.372 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:24.372 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.372 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.373 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:24.373 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.832937+0000", 2024-04-19T01:04:24.373 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.3200000000000003e-07 2024-04-19T01:04:24.373 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.373 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.373 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:24.373 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.833130+0000", 2024-04-19T01:04:24.373 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000192871 2024-04-19T01:04:24.373 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.373 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.373 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:24.373 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.833159+0000", 2024-04-19T01:04:24.373 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8986999999999999e-05 2024-04-19T01:04:24.373 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.373 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.374 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:24.374 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.833201+0000", 2024-04-19T01:04:24.374 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1637999999999997e-05 2024-04-19T01:04:24.374 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.374 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.374 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:24.374 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.833325+0000", 2024-04-19T01:04:24.374 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012459400000000001 2024-04-19T01:04:24.374 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.374 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.375 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:24.375 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.833356+0000", 2024-04-19T01:04:24.375 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0715e-05 2024-04-19T01:04:24.375 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.375 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.375 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:24.375 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.840406+0000", 2024-04-19T01:04:24.375 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.007049563 2024-04-19T01:04:24.375 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.375 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.375 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:24.375 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.840466+0000", 2024-04-19T01:04:24.375 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.9963000000000002e-05 2024-04-19T01:04:24.375 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:24.376 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:24.376 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:24.376 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.376 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.376 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.cs2 43/31 ECSubWrite(tid=45, reqid=osd.0.0:45, at_version=43'23, trim_to=0'0, roll_forward_to=41'21))", 2024-04-19T01:04:24.376 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.851044+0000", 2024-04-19T01:04:24.376 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 10.512167953000001, 2024-04-19T01:04:24.376 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0059246899999999998, 2024-04-19T01:04:24.376 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:24.376 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:24.376 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:24.376 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:24.376 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.376 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:24.376 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.851044+0000", 2024-04-19T01:04:24.377 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:24.377 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.377 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.377 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:24.377 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.851044+0000", 2024-04-19T01:04:24.377 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:24.377 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.377 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.377 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:24.377 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.851045+0000", 2024-04-19T01:04:24.377 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.6899999999999996e-07 2024-04-19T01:04:24.378 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.378 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.378 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:24.378 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.851073+0000", 2024-04-19T01:04:24.378 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7725e-05 2024-04-19T01:04:24.378 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.378 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.378 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:24.378 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.851086+0000", 2024-04-19T01:04:24.378 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3628000000000001e-05 2024-04-19T01:04:24.378 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.378 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.378 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:24.379 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.851098+0000", 2024-04-19T01:04:24.379 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1783999999999999e-05 2024-04-19T01:04:24.379 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.379 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.379 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:24.379 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.851206+0000", 2024-04-19T01:04:24.379 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000107759 2024-04-19T01:04:24.379 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.379 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.379 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:24.379 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.851221+0000", 2024-04-19T01:04:24.379 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5449999999999999e-05 2024-04-19T01:04:24.379 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.380 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.380 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:24.381 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.856908+0000", 2024-04-19T01:04:24.386 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0056870749999999998 2024-04-19T01:04:24.387 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.387 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.387 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:24.387 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.856969+0000", 2024-04-19T01:04:24.387 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0300000000000002e-05 2024-04-19T01:04:24.387 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:24.387 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:24.387 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:24.387 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.387 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.387 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:253 3.ds0 3:b9317ffa:::smithi15528762-1:head [append 0~704512] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:24.387 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.983022+0000", 2024-04-19T01:04:24.387 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 10.380189977000001, 2024-04-19T01:04:24.388 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.034825673000000001, 2024-04-19T01:04:24.388 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:24.388 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:24.388 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:24.388 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:24.388 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:24.388 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:24.388 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 253 2024-04-19T01:04:24.388 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.388 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:24.388 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.388 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:24.389 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.983022+0000", 2024-04-19T01:04:24.389 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:24.389 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.389 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.389 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:24.389 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.983022+0000", 2024-04-19T01:04:24.389 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:24.389 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.389 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.390 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:24.390 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.983023+0000", 2024-04-19T01:04:24.390 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.6500000000000008e-07 2024-04-19T01:04:24.390 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.390 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.390 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:24.395 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.983925+0000", 2024-04-19T01:04:24.395 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000902439 2024-04-19T01:04:24.395 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.395 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.395 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:24.395 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.983943+0000", 2024-04-19T01:04:24.395 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7975e-05 2024-04-19T01:04:24.395 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.395 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.395 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:24.395 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.983952+0000", 2024-04-19T01:04:24.396 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.4270000000000008e-06 2024-04-19T01:04:24.396 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.396 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.396 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:24.396 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.984029+0000", 2024-04-19T01:04:24.396 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7631000000000003e-05 2024-04-19T01:04:24.396 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.396 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.396 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:24.396 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.984150+0000", 2024-04-19T01:04:24.396 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000120548 2024-04-19T01:04:24.396 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.396 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.396 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:24.397 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.986166+0000", 2024-04-19T01:04:24.397 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0020163410000000001 2024-04-19T01:04:24.397 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.397 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.397 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:24.397 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.001084+0000", 2024-04-19T01:04:24.397 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014917872 2024-04-19T01:04:24.397 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.397 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.397 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:24.397 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.017779+0000", 2024-04-19T01:04:24.397 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.016694817000000001 2024-04-19T01:04:24.397 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.397 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.398 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:24.398 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.017848+0000", 2024-04-19T01:04:24.398 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.8658000000000006e-05 2024-04-19T01:04:24.398 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:24.398 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:24.398 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:24.398 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.398 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.398 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:254 3.ds0 3:b9317ffa:::smithi15528762-1:head [append 0~376832] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:24.398 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.985096+0000", 2024-04-19T01:04:24.398 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 10.378116362, 2024-04-19T01:04:24.399 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.043912868000000001, 2024-04-19T01:04:24.399 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:24.399 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:24.399 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:24.399 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:24.399 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:24.399 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:24.399 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 254 2024-04-19T01:04:24.399 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.399 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:24.399 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.399 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:24.399 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985096+0000", 2024-04-19T01:04:24.400 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:24.400 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.400 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.400 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:24.400 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985096+0000", 2024-04-19T01:04:24.400 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:24.400 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.400 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.400 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:24.400 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985098+0000", 2024-04-19T01:04:24.400 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0540000000000002e-06 2024-04-19T01:04:24.400 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.400 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.401 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:24.401 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985632+0000", 2024-04-19T01:04:24.401 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00053471200000000001 2024-04-19T01:04:24.401 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.401 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.401 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:24.401 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985650+0000", 2024-04-19T01:04:24.401 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7217000000000001e-05 2024-04-19T01:04:24.401 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.401 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.401 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:24.401 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985658+0000", 2024-04-19T01:04:24.401 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.8689999999999993e-06 2024-04-19T01:04:24.401 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.402 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.404 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:24.404 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.995027+0000", 2024-04-19T01:04:24.404 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0093689159999999997 2024-04-19T01:04:24.404 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.404 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.404 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:24.404 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.995141+0000", 2024-04-19T01:04:24.404 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011387000000000001 2024-04-19T01:04:24.405 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.405 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.405 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:24.405 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.996005+0000", 2024-04-19T01:04:24.405 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00086397999999999998 2024-04-19T01:04:24.405 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.405 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.405 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:24.405 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.002138+0000", 2024-04-19T01:04:24.405 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.006133182 2024-04-19T01:04:24.405 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.405 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.405 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:24.405 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.028913+0000", 2024-04-19T01:04:24.406 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.026774686999999998 2024-04-19T01:04:24.406 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.406 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.406 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:24.406 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.029008+0000", 2024-04-19T01:04:24.406 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.5381e-05 2024-04-19T01:04:24.406 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:24.406 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:24.406 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:24.406 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.406 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.406 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:255 3.ds0 3:b9317ffa:::smithi15528762-1:head [setxattr _header (55) in=7b] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:24.406 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.985699+0000", 2024-04-19T01:04:24.406 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 10.377512824, 2024-04-19T01:04:24.407 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.043535886000000003, 2024-04-19T01:04:24.407 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:24.407 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:24.407 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:24.407 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:24.407 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:24.407 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:24.407 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 255 2024-04-19T01:04:24.407 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.407 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:24.407 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.407 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:24.407 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985699+0000", 2024-04-19T01:04:24.407 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:24.407 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.408 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.408 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:24.408 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985699+0000", 2024-04-19T01:04:24.408 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:24.408 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.408 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.408 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:24.408 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985700+0000", 2024-04-19T01:04:24.408 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4199999999999995e-07 2024-04-19T01:04:24.408 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.408 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.408 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:24.408 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985705+0000", 2024-04-19T01:04:24.408 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7960000000000003e-06 2024-04-19T01:04:24.408 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.409 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.409 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:24.409 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985715+0000", 2024-04-19T01:04:24.409 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0285999999999999e-05 2024-04-19T01:04:24.409 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.409 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.409 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:24.409 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985735+0000", 2024-04-19T01:04:24.409 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9666000000000001e-05 2024-04-19T01:04:24.409 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.409 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.409 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:24.409 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.001214+0000", 2024-04-19T01:04:24.409 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015479694 2024-04-19T01:04:24.410 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.410 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.410 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:24.410 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.001417+0000", 2024-04-19T01:04:24.410 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00020274099999999999 2024-04-19T01:04:24.410 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.410 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.410 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:24.410 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.001931+0000", 2024-04-19T01:04:24.410 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000513599 2024-04-19T01:04:24.410 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.410 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.410 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:24.411 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.002446+0000", 2024-04-19T01:04:24.411 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00051521100000000001 2024-04-19T01:04:24.411 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.411 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.411 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:24.411 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.029136+0000", 2024-04-19T01:04:24.411 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.026690523000000001 2024-04-19T01:04:24.411 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.412 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.412 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:24.412 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.029235+0000", 2024-04-19T01:04:24.412 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.8727999999999994e-05 2024-04-19T01:04:24.412 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:24.413 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:24.413 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:24.413 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.413 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.413 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:256 3.ds0 3:b9317ffa:::smithi15528762-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:24.413 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.985744+0000", 2024-04-19T01:04:24.413 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 10.377468038, 2024-04-19T01:04:24.413 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.046574693, 2024-04-19T01:04:24.413 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:24.413 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:24.413 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:24.413 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:24.413 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:24.413 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:24.414 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 256 2024-04-19T01:04:24.414 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.414 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:24.414 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.414 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:24.414 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985744+0000", 2024-04-19T01:04:24.414 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:24.414 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.414 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.414 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:24.414 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985744+0000", 2024-04-19T01:04:24.414 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:24.414 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.414 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.415 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:24.415 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985744+0000", 2024-04-19T01:04:24.415 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1699999999999999e-07 2024-04-19T01:04:24.415 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.415 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.415 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:24.415 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985747+0000", 2024-04-19T01:04:24.415 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3429999999999998e-06 2024-04-19T01:04:24.415 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.415 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.415 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:24.415 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985753+0000", 2024-04-19T01:04:24.415 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8619999999999999e-06 2024-04-19T01:04:24.415 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.415 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.416 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:24.416 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985756+0000", 2024-04-19T01:04:24.416 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8820000000000003e-06 2024-04-19T01:04:24.416 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.416 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.416 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:24.416 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.002179+0000", 2024-04-19T01:04:24.416 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.01642302 2024-04-19T01:04:24.416 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.416 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.416 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:04:24.416 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.002278+0000", 2024-04-19T01:04:24.416 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.8548000000000006e-05 2024-04-19T01:04:24.416 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.416 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.417 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:24.417 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.029285+0000", 2024-04-19T01:04:24.417 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.027006935999999999 2024-04-19T01:04:24.417 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.417 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.417 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:24.417 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.029400+0000", 2024-04-19T01:04:24.417 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000115438 2024-04-19T01:04:24.417 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.417 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.417 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:24.417 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.032319+0000", 2024-04-19T01:04:24.417 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0029182470000000001 2024-04-19T01:04:24.417 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:24.417 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:24.418 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:24.418 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.418 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.418 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:271 3.ds0 3:bf91dcb5:::smithi15528762-2:head [copy-from ver 2] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:24.418 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:14.014802+0000", 2024-04-19T01:04:24.418 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 10.348409573, 2024-04-19T01:04:24.418 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015357356000000001, 2024-04-19T01:04:24.418 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:24.418 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:24.418 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:24.419 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:24.419 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:24.419 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:24.419 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 271 2024-04-19T01:04:24.419 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.419 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:24.419 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.419 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:24.419 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.014802+0000", 2024-04-19T01:04:24.419 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:24.419 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.419 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.420 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:24.420 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.014802+0000", 2024-04-19T01:04:24.420 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:24.420 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.420 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.420 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:24.420 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.014804+0000", 2024-04-19T01:04:24.420 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7570000000000001e-06 2024-04-19T01:04:24.420 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.420 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.420 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:24.420 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.014811+0000", 2024-04-19T01:04:24.420 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.3529999999999997e-06 2024-04-19T01:04:24.420 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.420 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.421 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:24.421 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.014822+0000", 2024-04-19T01:04:24.421 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0098000000000001e-05 2024-04-19T01:04:24.421 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.421 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.421 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:24.421 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.014826+0000", 2024-04-19T01:04:24.421 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8350000000000003e-06 2024-04-19T01:04:24.421 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.421 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.421 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:24.421 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.014864+0000", 2024-04-19T01:04:24.421 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7364999999999999e-05 2024-04-19T01:04:24.421 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.421 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.422 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:24.422 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.014966+0000", 2024-04-19T01:04:24.422 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000102012 2024-04-19T01:04:24.422 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.422 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.422 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:24.422 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.018689+0000", 2024-04-19T01:04:24.422 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0037234669999999998 2024-04-19T01:04:24.422 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.422 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.422 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:24.422 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.021559+0000", 2024-04-19T01:04:24.422 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0028693740000000001 2024-04-19T01:04:24.422 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.423 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.423 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:24.423 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.030075+0000", 2024-04-19T01:04:24.423 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0085168429999999996 2024-04-19T01:04:24.423 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.423 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.423 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:24.423 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.030160+0000", 2024-04-19T01:04:24.423 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.4252000000000006e-05 2024-04-19T01:04:24.423 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:24.423 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:24.423 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:24.423 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.423 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.424 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:272 3.ds0 3:bf91dcb5:::smithi15528762-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:24.424 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:14.014837+0000", 2024-04-19T01:04:24.424 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 10.348374787000001, 2024-04-19T01:04:24.424 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0062774889999999998, 2024-04-19T01:04:24.424 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:24.424 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:24.424 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:24.424 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:24.424 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:24.424 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:24.424 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 272 2024-04-19T01:04:24.424 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.424 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:24.424 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.425 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:24.425 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.014837+0000", 2024-04-19T01:04:24.425 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:24.425 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.425 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.425 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:24.425 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.014837+0000", 2024-04-19T01:04:24.425 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:24.425 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.425 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.425 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:24.425 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.014841+0000", 2024-04-19T01:04:24.425 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0319999999999997e-06 2024-04-19T01:04:24.425 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.427 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.428 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:24.428 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.014844+0000", 2024-04-19T01:04:24.428 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8310000000000001e-06 2024-04-19T01:04:24.428 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.428 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.428 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:24.428 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.014852+0000", 2024-04-19T01:04:24.428 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.4109999999999996e-06 2024-04-19T01:04:24.428 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.428 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.429 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:24.429 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.014859+0000", 2024-04-19T01:04:24.429 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.348e-06 2024-04-19T01:04:24.429 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.429 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.430 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:24.430 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.015170+0000", 2024-04-19T01:04:24.430 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00031090300000000001 2024-04-19T01:04:24.430 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.430 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.430 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for blocked object", 2024-04-19T01:04:24.430 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.015222+0000", 2024-04-19T01:04:24.430 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.1823000000000002e-05 2024-04-19T01:04:24.430 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.430 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.430 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:24.430 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.020876+0000", 2024-04-19T01:04:24.430 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0056546340000000004 2024-04-19T01:04:24.430 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.431 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.431 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:24.431 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.020979+0000", 2024-04-19T01:04:24.431 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000103021 2024-04-19T01:04:24.431 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.431 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.431 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:24.431 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.021115+0000", 2024-04-19T01:04:24.431 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013548600000000001 2024-04-19T01:04:24.431 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:24.431 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:24.431 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:24.431 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.431 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.432 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:330 3.ds0 3:bf91dcb5:::smithi15528762-2:head [append 0~507904] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:04:24.432 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.674394+0000", 2024-04-19T01:04:24.432 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 6.6888178869999999, 2024-04-19T01:04:24.432 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.036711424999999999, 2024-04-19T01:04:24.432 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:24.432 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:24.432 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:24.432 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:24.432 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:24.432 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:24.432 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 330 2024-04-19T01:04:24.432 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.432 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:24.432 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.432 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:24.433 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.674394+0000", 2024-04-19T01:04:24.433 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:24.433 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.433 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.433 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:24.433 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.674394+0000", 2024-04-19T01:04:24.433 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:24.433 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.433 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.433 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:24.433 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.674396+0000", 2024-04-19T01:04:24.433 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3870000000000002e-06 2024-04-19T01:04:24.433 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.433 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.433 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:24.434 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.675539+0000", 2024-04-19T01:04:24.434 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001142321 2024-04-19T01:04:24.434 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.434 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.434 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:24.434 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.675564+0000", 2024-04-19T01:04:24.434 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5126e-05 2024-04-19T01:04:24.434 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.434 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.434 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:24.434 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.675575+0000", 2024-04-19T01:04:24.434 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0951999999999999e-05 2024-04-19T01:04:24.434 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.434 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.434 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:24.435 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.675641+0000", 2024-04-19T01:04:24.435 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5914999999999995e-05 2024-04-19T01:04:24.435 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.435 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.435 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:24.435 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.675713+0000", 2024-04-19T01:04:24.435 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2169000000000006e-05 2024-04-19T01:04:24.435 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.435 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.435 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:24.435 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.676428+0000", 2024-04-19T01:04:24.435 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00071535000000000001 2024-04-19T01:04:24.435 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.435 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.435 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:24.436 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680290+0000", 2024-04-19T01:04:24.436 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003861532 2024-04-19T01:04:24.436 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.436 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.436 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:24.436 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.711050+0000", 2024-04-19T01:04:24.436 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.030759916000000002 2024-04-19T01:04:24.436 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.436 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.436 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:24.436 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.711106+0000", 2024-04-19T01:04:24.436 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.5757000000000002e-05 2024-04-19T01:04:24.436 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:24.436 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:24.436 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:24.437 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.437 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.437 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:331 3.ds0 3:bf91dcb5:::smithi15528762-2:head [append 0~507904] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:04:24.437 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.680101+0000", 2024-04-19T01:04:24.437 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 6.6831111080000003, 2024-04-19T01:04:24.437 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.031204012, 2024-04-19T01:04:24.437 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:24.437 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:24.437 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:24.437 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:24.437 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:24.438 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:24.438 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 331 2024-04-19T01:04:24.438 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.438 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:24.438 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.438 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:24.438 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680101+0000", 2024-04-19T01:04:24.438 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:24.438 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.438 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.438 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:24.438 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680101+0000", 2024-04-19T01:04:24.438 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:24.439 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.439 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.439 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:24.439 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680103+0000", 2024-04-19T01:04:24.439 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.956e-06 2024-04-19T01:04:24.439 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.439 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.439 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:24.439 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680444+0000", 2024-04-19T01:04:24.439 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00034149199999999999 2024-04-19T01:04:24.440 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.440 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.440 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:24.440 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680460+0000", 2024-04-19T01:04:24.440 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5554e-05 2024-04-19T01:04:24.440 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.440 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.440 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:24.440 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680471+0000", 2024-04-19T01:04:24.440 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0852e-05 2024-04-19T01:04:24.440 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.440 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.440 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:24.440 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680539+0000", 2024-04-19T01:04:24.440 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.8619000000000001e-05 2024-04-19T01:04:24.441 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.441 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.441 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:24.441 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680652+0000", 2024-04-19T01:04:24.441 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011258499999999999 2024-04-19T01:04:24.441 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.441 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.441 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:24.441 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.681392+0000", 2024-04-19T01:04:24.441 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00074039899999999998 2024-04-19T01:04:24.441 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.441 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.441 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:24.441 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.691230+0000", 2024-04-19T01:04:24.442 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0098373260000000004 2024-04-19T01:04:24.442 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.442 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.442 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:24.442 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.711264+0000", 2024-04-19T01:04:24.442 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.020034501999999999 2024-04-19T01:04:24.442 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.442 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.442 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:24.442 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.711305+0000", 2024-04-19T01:04:24.442 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0726999999999998e-05 2024-04-19T01:04:24.442 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:24.442 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:24.442 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:24.443 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.443 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.443 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 47/31 ECSubWriteReply(tid=61, last_complete=47'32, committed=1, applied=1))", 2024-04-19T01:04:24.443 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.680677+0000", 2024-04-19T01:04:24.443 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 6.6825346489999999, 2024-04-19T01:04:24.443 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0058075790000000002, 2024-04-19T01:04:24.443 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:24.443 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:24.443 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:24.443 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:24.443 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.443 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:24.443 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680677+0000", 2024-04-19T01:04:24.443 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:24.443 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.444 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.444 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:24.444 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680677+0000", 2024-04-19T01:04:24.444 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:24.444 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.444 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.444 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:24.444 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680679+0000", 2024-04-19T01:04:24.444 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3039999999999999e-06 2024-04-19T01:04:24.444 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.444 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.444 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:24.444 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680686+0000", 2024-04-19T01:04:24.444 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.9240000000000001e-06 2024-04-19T01:04:24.444 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.445 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.445 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:24.445 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680698+0000", 2024-04-19T01:04:24.445 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2343e-05 2024-04-19T01:04:24.445 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.445 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.445 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:24.445 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680706+0000", 2024-04-19T01:04:24.445 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.943e-06 2024-04-19T01:04:24.445 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.445 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.445 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:24.445 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.686448+0000", 2024-04-19T01:04:24.445 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0057425419999999998 2024-04-19T01:04:24.445 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.446 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.446 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:24.446 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.686485+0000", 2024-04-19T01:04:24.446 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.6523000000000003e-05 2024-04-19T01:04:24.446 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:24.446 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:24.446 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:24.446 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.446 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.446 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:332 3.ds0 3:bf91dcb5:::smithi15528762-2:head [append 0~262144] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:04:24.446 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.682638+0000", 2024-04-19T01:04:24.446 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 6.6805740040000003, 2024-04-19T01:04:24.446 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.029333649999999999, 2024-04-19T01:04:24.447 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:24.447 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:24.447 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:24.447 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:24.447 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:24.447 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:24.447 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 332 2024-04-19T01:04:24.447 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.447 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:24.447 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.447 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:24.447 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682638+0000", 2024-04-19T01:04:24.447 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:24.447 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.447 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.454 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:24.454 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682638+0000", 2024-04-19T01:04:24.454 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:24.454 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.454 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.455 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:24.455 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682640+0000", 2024-04-19T01:04:24.455 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5990000000000001e-06 2024-04-19T01:04:24.455 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.455 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.455 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:24.455 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682770+0000", 2024-04-19T01:04:24.455 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013076200000000001 2024-04-19T01:04:24.455 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.455 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.455 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:24.455 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682784+0000", 2024-04-19T01:04:24.455 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3786e-05 2024-04-19T01:04:24.455 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.455 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.456 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:24.456 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682792+0000", 2024-04-19T01:04:24.456 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.3000000000000002e-06 2024-04-19T01:04:24.456 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.456 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.456 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:24.456 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.686722+0000", 2024-04-19T01:04:24.456 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0039299749999999996 2024-04-19T01:04:24.456 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.456 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.456 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:24.456 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.686804+0000", 2024-04-19T01:04:24.456 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1588000000000006e-05 2024-04-19T01:04:24.456 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.456 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.457 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:24.457 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.687521+0000", 2024-04-19T01:04:24.457 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00071744300000000001 2024-04-19T01:04:24.457 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.457 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.457 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:24.457 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.691494+0000", 2024-04-19T01:04:24.457 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0039721239999999996 2024-04-19T01:04:24.457 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.457 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.457 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:24.457 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.711934+0000", 2024-04-19T01:04:24.457 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.020440649000000002 2024-04-19T01:04:24.457 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.457 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.458 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:24.458 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.711972+0000", 2024-04-19T01:04:24.458 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7423999999999999e-05 2024-04-19T01:04:24.458 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:24.458 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:24.458 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:24.458 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.458 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.458 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:333 3.ds0 3:bf91dcb5:::smithi15528762-2:head [setxattr _header (55) in=7b] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:04:24.458 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.682805+0000", 2024-04-19T01:04:24.458 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 6.6804066029999998, 2024-04-19T01:04:24.458 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.029376039, 2024-04-19T01:04:24.458 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:24.458 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:24.459 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:24.459 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:24.459 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:24.459 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:24.459 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 333 2024-04-19T01:04:24.459 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.459 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:24.459 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.459 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:24.459 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682805+0000", 2024-04-19T01:04:24.459 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:24.459 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.459 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.459 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:24.460 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682805+0000", 2024-04-19T01:04:24.460 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:24.460 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.460 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.460 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:24.460 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682806+0000", 2024-04-19T01:04:24.460 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4099999999999999e-07 2024-04-19T01:04:24.460 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.460 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.460 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:24.460 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682815+0000", 2024-04-19T01:04:24.460 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.516e-06 2024-04-19T01:04:24.460 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.460 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.461 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:24.461 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682823+0000", 2024-04-19T01:04:24.461 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6459999999999993e-06 2024-04-19T01:04:24.461 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.470 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.470 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:24.470 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682832+0000", 2024-04-19T01:04:24.471 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.6260000000000001e-06 2024-04-19T01:04:24.471 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.471 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.471 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:24.471 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.690350+0000", 2024-04-19T01:04:24.471 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0075183730000000001 2024-04-19T01:04:24.471 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.471 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.471 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:24.471 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.690456+0000", 2024-04-19T01:04:24.471 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010617599999999999 2024-04-19T01:04:24.471 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.471 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.471 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:24.472 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.690951+0000", 2024-04-19T01:04:24.472 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00049519699999999995 2024-04-19T01:04:24.472 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.472 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.472 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:24.472 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.693060+0000", 2024-04-19T01:04:24.472 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0021084239999999998 2024-04-19T01:04:24.472 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.472 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.472 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:24.472 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.712088+0000", 2024-04-19T01:04:24.473 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.019028024000000001 2024-04-19T01:04:24.473 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.473 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.473 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:24.473 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.712181+0000", 2024-04-19T01:04:24.473 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.3516000000000004e-05 2024-04-19T01:04:24.473 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:24.473 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:24.473 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:24.473 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.473 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.473 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:334 3.ds0 3:bf91dcb5:::smithi15528762-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:04:24.473 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.682840+0000", 2024-04-19T01:04:24.473 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 6.6803724540000005, 2024-04-19T01:04:24.474 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.031162179000000002, 2024-04-19T01:04:24.474 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:24.474 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:24.474 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:24.479 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:24.479 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:24.479 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:24.479 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 334 2024-04-19T01:04:24.479 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.479 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:24.479 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.479 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:24.479 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682840+0000", 2024-04-19T01:04:24.479 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:24.480 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.480 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.480 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:24.480 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682840+0000", 2024-04-19T01:04:24.480 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:24.480 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.480 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.480 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:24.480 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682840+0000", 2024-04-19T01:04:24.480 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4899999999999995e-07 2024-04-19T01:04:24.480 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.480 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.480 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:24.480 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682848+0000", 2024-04-19T01:04:24.481 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5270000000000001e-06 2024-04-19T01:04:24.481 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.481 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.481 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:24.481 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682855+0000", 2024-04-19T01:04:24.481 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.4889999999999998e-06 2024-04-19T01:04:24.481 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.481 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.481 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:24.481 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682860+0000", 2024-04-19T01:04:24.481 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.164e-06 2024-04-19T01:04:24.481 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.481 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.481 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:24.481 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.691525+0000", 2024-04-19T01:04:24.482 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0086643239999999993 2024-04-19T01:04:24.482 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.482 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.482 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:04:24.482 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.691618+0000", 2024-04-19T01:04:24.482 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2983000000000001e-05 2024-04-19T01:04:24.482 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.482 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.482 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:24.482 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.712661+0000", 2024-04-19T01:04:24.482 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021043777 2024-04-19T01:04:24.482 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.483 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.483 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:24.483 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.712733+0000", 2024-04-19T01:04:24.483 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2063000000000004e-05 2024-04-19T01:04:24.483 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.483 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.483 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:24.483 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.714002+0000", 2024-04-19T01:04:24.483 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0012683029999999999 2024-04-19T01:04:24.483 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:24.483 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:24.483 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:24.484 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.484 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.484 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.3s6 48/31 ECSubWrite(tid=43, reqid=osd.15.0:43, at_version=48'32, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:04:24.484 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.513111+0000", 2024-04-19T01:04:24.484 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.8501012330000002, 2024-04-19T01:04:24.484 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0060940860000000003, 2024-04-19T01:04:24.484 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:24.484 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:24.484 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:24.484 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:24.484 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.484 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:24.484 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.513111+0000", 2024-04-19T01:04:24.484 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:24.485 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.485 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.485 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:24.485 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.513111+0000", 2024-04-19T01:04:24.485 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:24.485 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.485 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.485 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:24.485 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.513111+0000", 2024-04-19T01:04:24.485 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.82e-07 2024-04-19T01:04:24.486 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.486 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.486 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:24.486 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.513153+0000", 2024-04-19T01:04:24.486 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1508000000000003e-05 2024-04-19T01:04:24.486 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.486 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.486 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:24.486 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.513174+0000", 2024-04-19T01:04:24.486 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1773e-05 2024-04-19T01:04:24.486 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.486 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.486 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:24.486 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.513191+0000", 2024-04-19T01:04:24.487 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6104999999999999e-05 2024-04-19T01:04:24.487 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.487 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.487 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:24.487 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.514214+0000", 2024-04-19T01:04:24.487 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010229620000000001 2024-04-19T01:04:24.487 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.487 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.487 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:24.487 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.514243+0000", 2024-04-19T01:04:24.487 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9179e-05 2024-04-19T01:04:24.488 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.488 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.488 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:24.488 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.519122+0000", 2024-04-19T01:04:24.488 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0048788549999999997 2024-04-19T01:04:24.488 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.488 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.488 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:24.488 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.519205+0000", 2024-04-19T01:04:24.488 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.3221999999999997e-05 2024-04-19T01:04:24.489 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:24.489 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:24.489 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:24.489 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.489 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.489 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:354 3.as0 3:56c32ee7:::smithi15528762-47:head [copy-from ver 35] snapc 9=[8,7] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:04:24.489 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.544030+0000", 2024-04-19T01:04:24.489 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.8191824270000003, 2024-04-19T01:04:24.489 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.049063250000000003, 2024-04-19T01:04:24.489 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:24.489 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:24.490 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:24.490 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:24.490 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:24.490 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:24.490 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 354 2024-04-19T01:04:24.490 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.490 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:24.490 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.490 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:24.490 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544030+0000", 2024-04-19T01:04:24.490 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:24.490 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.490 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.490 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:24.491 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544030+0000", 2024-04-19T01:04:24.491 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:24.491 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.491 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.491 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:24.491 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544033+0000", 2024-04-19T01:04:24.491 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.506e-06 2024-04-19T01:04:24.491 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.491 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.491 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:24.491 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544046+0000", 2024-04-19T01:04:24.491 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3208e-05 2024-04-19T01:04:24.491 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.491 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.492 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:24.492 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544082+0000", 2024-04-19T01:04:24.492 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5509999999999997e-05 2024-04-19T01:04:24.492 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.492 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.492 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:24.492 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544101+0000", 2024-04-19T01:04:24.492 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9500999999999998e-05 2024-04-19T01:04:24.492 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.492 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.492 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:24.492 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544230+0000", 2024-04-19T01:04:24.492 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000129114 2024-04-19T01:04:24.492 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.492 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.493 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:24.493 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544428+0000", 2024-04-19T01:04:24.493 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00019776799999999999 2024-04-19T01:04:24.493 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.493 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.493 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:24.493 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.556395+0000", 2024-04-19T01:04:24.493 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011967261 2024-04-19T01:04:24.493 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.493 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.493 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:24.493 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.574011+0000", 2024-04-19T01:04:24.493 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017615654000000001 2024-04-19T01:04:24.493 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.493 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.494 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:24.494 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.592920+0000", 2024-04-19T01:04:24.494 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018908812000000001 2024-04-19T01:04:24.494 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.494 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.494 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:24.494 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.593093+0000", 2024-04-19T01:04:24.494 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00017291600000000001 2024-04-19T01:04:24.494 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:24.494 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:24.495 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:24.495 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.495 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.495 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:355 3.as0 3:56c32ee7:::smithi15528762-47:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:04:24.495 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.544150+0000", 2024-04-19T01:04:24.495 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.8190616550000005, 2024-04-19T01:04:24.495 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.029722035000000001, 2024-04-19T01:04:24.495 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:24.495 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:24.495 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:24.495 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:24.495 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:24.495 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:24.495 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 355 2024-04-19T01:04:24.495 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.496 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:24.496 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.496 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:24.496 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544150+0000", 2024-04-19T01:04:24.496 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:24.496 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.496 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.496 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:24.496 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544150+0000", 2024-04-19T01:04:24.496 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:24.496 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.496 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.496 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:24.496 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544152+0000", 2024-04-19T01:04:24.497 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.28e-06 2024-04-19T01:04:24.497 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.497 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.497 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:24.497 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544158+0000", 2024-04-19T01:04:24.497 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4160000000000001e-06 2024-04-19T01:04:24.497 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.497 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.499 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:24.499 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544178+0000", 2024-04-19T01:04:24.499 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0054000000000001e-05 2024-04-19T01:04:24.499 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.499 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.499 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:24.499 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544195+0000", 2024-04-19T01:04:24.499 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6835999999999999e-05 2024-04-19T01:04:24.499 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.499 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.499 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:24.499 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544717+0000", 2024-04-19T01:04:24.499 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00052258800000000005 2024-04-19T01:04:24.500 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.500 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.500 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for blocked object", 2024-04-19T01:04:24.500 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544835+0000", 2024-04-19T01:04:24.500 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000117448 2024-04-19T01:04:24.500 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.500 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.500 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:24.500 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.573609+0000", 2024-04-19T01:04:24.500 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.028774094 2024-04-19T01:04:24.500 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.500 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.500 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:24.500 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.573721+0000", 2024-04-19T01:04:24.500 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011205900000000001 2024-04-19T01:04:24.501 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.501 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.501 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:24.501 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.573872+0000", 2024-04-19T01:04:24.501 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015126000000000001 2024-04-19T01:04:24.501 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:24.501 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:24.501 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:24.501 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.501 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.501 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 49/31 ECSubWriteReply(tid=74, last_complete=49'41, committed=1, applied=1))", 2024-04-19T01:04:24.501 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:19.532353+0000", 2024-04-19T01:04:24.501 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 4.8308590469999997, 2024-04-19T01:04:24.501 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0070731969999999998, 2024-04-19T01:04:24.501 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:24.502 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:24.502 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:24.502 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:24.502 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.502 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:24.502 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.532353+0000", 2024-04-19T01:04:24.502 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:24.502 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.502 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.502 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:24.502 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.532353+0000", 2024-04-19T01:04:24.502 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:24.502 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.502 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.503 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:24.503 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.532355+0000", 2024-04-19T01:04:24.503 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.703e-06 2024-04-19T01:04:24.503 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.503 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.503 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:24.503 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.532364+0000", 2024-04-19T01:04:24.503 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9740000000000008e-06 2024-04-19T01:04:24.503 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.503 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.503 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:24.503 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.532388+0000", 2024-04-19T01:04:24.503 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4584e-05 2024-04-19T01:04:24.503 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.503 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.504 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:24.504 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.532401+0000", 2024-04-19T01:04:24.504 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2853999999999999e-05 2024-04-19T01:04:24.504 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.504 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.504 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:24.504 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.539398+0000", 2024-04-19T01:04:24.504 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0069973220000000003 2024-04-19T01:04:24.504 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.504 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.504 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:24.504 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.539426+0000", 2024-04-19T01:04:24.504 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7759999999999998e-05 2024-04-19T01:04:24.504 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:24.504 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:24.505 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:24.505 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.505 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.505 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 48/31 ECSubWriteReply(tid=74, last_complete=49'41, committed=1, applied=1))", 2024-04-19T01:04:24.505 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:19.533345+0000", 2024-04-19T01:04:24.505 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 4.8298670789999996, 2024-04-19T01:04:24.505 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0062278180000000004, 2024-04-19T01:04:24.505 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:24.505 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:24.505 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:24.505 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:24.505 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.505 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:24.505 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.533345+0000", 2024-04-19T01:04:24.505 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:24.506 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.506 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.506 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:24.506 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.533345+0000", 2024-04-19T01:04:24.506 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:24.506 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.506 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.506 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:24.506 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.533346+0000", 2024-04-19T01:04:24.506 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.217e-06 2024-04-19T01:04:24.506 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.506 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.507 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:24.507 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.533353+0000", 2024-04-19T01:04:24.507 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.3479999999999998e-06 2024-04-19T01:04:24.507 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.507 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.507 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:24.507 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.533380+0000", 2024-04-19T01:04:24.507 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6361999999999999e-05 2024-04-19T01:04:24.507 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.507 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.507 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:24.507 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.533392+0000", 2024-04-19T01:04:24.507 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1878e-05 2024-04-19T01:04:24.507 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.507 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.508 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:24.508 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.539552+0000", 2024-04-19T01:04:24.508 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0061600830000000002 2024-04-19T01:04:24.508 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:24.508 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:24.508 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:24.508 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.539573+0000", 2024-04-19T01:04:24.508 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0930000000000001e-05 2024-04-19T01:04:24.508 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:24.508 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:24.508 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:24.508 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:24.508 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:24.508 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:24.510 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-19T01:04:24.511 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:04:24.509+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:04:24.523 INFO:tasks.rados.rados.0.smithi155.stdout:144: done (1 left) 2024-04-19T01:04:24.523 INFO:tasks.rados.rados.0.smithi155.stdout:148: done (0 left) 2024-04-19T01:04:24.524 INFO:tasks.rados.rados.0.smithi155.stdout:149: snap_remove snap 8 2024-04-19T01:04:24.611 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:24.611+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:24.646 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:24.646 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [], 2024-04-19T01:04:24.646 INFO:teuthology.orchestra.run.smithi110.stdout: "num_ops": 0 2024-04-19T01:04:24.646 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:24.661 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-19T01:04:24.712 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:04:24.713+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:04:24.813 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:04:24.810+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:04:24.826 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:24.827 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [], 2024-04-19T01:04:24.827 INFO:teuthology.orchestra.run.smithi110.stdout: "complaint_time": 30, 2024-04-19T01:04:24.827 INFO:teuthology.orchestra.run.smithi110.stdout: "num_blocked_ops": 0 2024-04-19T01:04:24.827 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:24.842 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-19T01:04:24.931 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:04:24.930+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:04:24.997 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-19T01:04:25.015 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:25.015 INFO:teuthology.orchestra.run.smithi110.stdout: "size": 20, 2024-04-19T01:04:25.015 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 600, 2024-04-19T01:04:25.015 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [ 2024-04-19T01:04:25.015 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.015 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:169 3.0s0 3:0abdaea3:::smithi15528762-5:head [append 0~0] snapc 5=[5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-19T01:04:25.016 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:08.801953+0000", 2024-04-19T01:04:25.016 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 16.211785653, 2024-04-19T01:04:25.016 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.015928824000000001, 2024-04-19T01:04:25.016 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:25.016 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:25.016 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:25.016 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:25.016 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:25.016 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:25.016 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 169 2024-04-19T01:04:25.016 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.016 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:25.016 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.016 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:25.017 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.801953+0000", 2024-04-19T01:04:25.017 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:25.017 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.019 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.019 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:25.019 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.801953+0000", 2024-04-19T01:04:25.019 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:25.019 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.019 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.019 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:25.019 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.801958+0000", 2024-04-19T01:04:25.020 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.4399999999999998e-06 2024-04-19T01:04:25.020 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.020 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.020 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:25.020 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.801977+0000", 2024-04-19T01:04:25.020 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.8831999999999999e-05 2024-04-19T01:04:25.020 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.020 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.020 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:25.020 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.802015+0000", 2024-04-19T01:04:25.020 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.8705000000000003e-05 2024-04-19T01:04:25.020 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.020 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.020 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:25.020 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.802034+0000", 2024-04-19T01:04:25.021 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9025000000000002e-05 2024-04-19T01:04:25.021 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.021 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.021 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:25.021 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.802244+0000", 2024-04-19T01:04:25.021 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000209249 2024-04-19T01:04:25.021 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.021 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.021 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "op must wait for map", 2024-04-19T01:04:25.022 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.802265+0000", 2024-04-19T01:04:25.022 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1282999999999999e-05 2024-04-19T01:04:25.022 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.022 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.022 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:25.023 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.806425+0000", 2024-04-19T01:04:25.023 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0041602840000000002 2024-04-19T01:04:25.023 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.023 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.023 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:25.023 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.806665+0000", 2024-04-19T01:04:25.023 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00023976999999999999 2024-04-19T01:04:25.023 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.023 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.023 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:25.023 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.808837+0000", 2024-04-19T01:04:25.023 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.002172326 2024-04-19T01:04:25.023 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.023 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.024 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:25.024 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.815971+0000", 2024-04-19T01:04:25.024 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0071333300000000002 2024-04-19T01:04:25.024 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.024 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.024 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:25.024 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.817696+0000", 2024-04-19T01:04:25.024 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001725039 2024-04-19T01:04:25.024 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.024 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.024 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:25.024 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.817882+0000", 2024-04-19T01:04:25.024 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000186541 2024-04-19T01:04:25.024 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:25.024 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:25.025 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:25.025 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.025 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.025 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:170 3.0s0 3:0abdaea3:::smithi15528762-5:head [rollback 3] snapc 5=[5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-19T01:04:25.025 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:08.802081+0000", 2024-04-19T01:04:25.025 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 16.211657589000001, 2024-04-19T01:04:25.025 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.017167723999999999, 2024-04-19T01:04:25.025 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:25.025 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:25.025 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:25.025 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:25.025 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:25.025 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:25.025 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 170 2024-04-19T01:04:25.026 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.026 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:25.026 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.026 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:25.026 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.802081+0000", 2024-04-19T01:04:25.026 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:25.026 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.026 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.026 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:25.026 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.802081+0000", 2024-04-19T01:04:25.026 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:25.026 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.026 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.026 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:25.027 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.802083+0000", 2024-04-19T01:04:25.027 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.1230000000000001e-06 2024-04-19T01:04:25.027 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.027 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.027 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:25.027 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.802088+0000", 2024-04-19T01:04:25.027 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.0529999999999997e-06 2024-04-19T01:04:25.027 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.027 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.027 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:25.027 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.802102+0000", 2024-04-19T01:04:25.027 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.4569000000000001e-05 2024-04-19T01:04:25.027 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.027 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.027 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:25.028 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.802113+0000", 2024-04-19T01:04:25.028 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.0556e-05 2024-04-19T01:04:25.028 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.028 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.028 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:25.028 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.802425+0000", 2024-04-19T01:04:25.028 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00031243699999999999 2024-04-19T01:04:25.028 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.028 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.028 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:04:25.028 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.802438+0000", 2024-04-19T01:04:25.029 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.2493e-05 2024-04-19T01:04:25.029 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.029 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.030 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:25.030 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.810002+0000", 2024-04-19T01:04:25.030 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0075644969999999999 2024-04-19T01:04:25.030 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.030 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.030 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:25.030 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.810220+0000", 2024-04-19T01:04:25.030 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000217436 2024-04-19T01:04:25.030 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.030 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.030 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:25.030 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.811138+0000", 2024-04-19T01:04:25.030 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00091886099999999998 2024-04-19T01:04:25.030 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.031 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.031 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:25.031 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.815974+0000", 2024-04-19T01:04:25.031 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0048358519999999999 2024-04-19T01:04:25.031 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.031 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.031 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:25.031 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.819154+0000", 2024-04-19T01:04:25.031 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0031794890000000002 2024-04-19T01:04:25.031 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.031 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.031 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:25.032 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.819249+0000", 2024-04-19T01:04:25.032 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.5358000000000004e-05 2024-04-19T01:04:25.032 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:25.032 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:25.032 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:25.033 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.033 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.033 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:171 3.0s0 3:0abdaea3:::smithi15528762-5:head [append 0~0] snapc 5=[5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-19T01:04:25.033 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:08.802222+0000", 2024-04-19T01:04:25.033 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 16.211516947, 2024-04-19T01:04:25.033 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.018484289000000001, 2024-04-19T01:04:25.033 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:25.033 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:25.033 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:25.033 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:25.034 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:25.034 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:25.034 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 171 2024-04-19T01:04:25.034 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.034 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:25.034 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.034 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:25.034 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.802222+0000", 2024-04-19T01:04:25.034 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:25.034 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.034 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.034 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:25.034 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.802222+0000", 2024-04-19T01:04:25.035 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:25.035 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.035 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.035 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:25.035 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.802223+0000", 2024-04-19T01:04:25.035 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.3170000000000001e-06 2024-04-19T01:04:25.035 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.035 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.035 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:25.035 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.802232+0000", 2024-04-19T01:04:25.035 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.3960000000000006e-06 2024-04-19T01:04:25.035 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.035 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.035 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:25.036 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.802251+0000", 2024-04-19T01:04:25.036 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9609000000000001e-05 2024-04-19T01:04:25.036 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.036 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.036 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:25.036 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.802266+0000", 2024-04-19T01:04:25.036 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5051999999999999e-05 2024-04-19T01:04:25.036 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.036 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.036 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:25.036 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.802513+0000", 2024-04-19T01:04:25.036 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000246371 2024-04-19T01:04:25.036 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.036 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.037 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:04:25.037 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.802524+0000", 2024-04-19T01:04:25.037 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.1535999999999999e-05 2024-04-19T01:04:25.037 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.037 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.037 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:25.037 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.812512+0000", 2024-04-19T01:04:25.037 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0099875599999999995 2024-04-19T01:04:25.037 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.037 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.037 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:25.037 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.812675+0000", 2024-04-19T01:04:25.037 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000163525 2024-04-19T01:04:25.037 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.038 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.038 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:25.038 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.813813+0000", 2024-04-19T01:04:25.038 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001137676 2024-04-19T01:04:25.038 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.038 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.038 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:25.038 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.816015+0000", 2024-04-19T01:04:25.038 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0022017410000000001 2024-04-19T01:04:25.038 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.038 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.038 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:25.038 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.820527+0000", 2024-04-19T01:04:25.038 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0045121720000000001 2024-04-19T01:04:25.039 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.039 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.039 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:25.039 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.820706+0000", 2024-04-19T01:04:25.039 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00017933399999999999 2024-04-19T01:04:25.039 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:25.040 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:25.040 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:25.040 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.040 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.040 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs5 43/31 ECSubWrite(tid=47, reqid=client.4604.0:263, at_version=43'8, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:04:25.040 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.993731+0000", 2024-04-19T01:04:25.040 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 11.020008309, 2024-04-19T01:04:25.040 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.025167086000000002, 2024-04-19T01:04:25.040 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:25.040 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:25.041 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:25.041 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:25.041 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.041 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:25.041 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.993731+0000", 2024-04-19T01:04:25.041 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:25.041 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.041 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.041 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:25.041 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.993731+0000", 2024-04-19T01:04:25.041 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:25.041 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.041 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.041 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:25.041 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.993733+0000", 2024-04-19T01:04:25.042 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.114e-06 2024-04-19T01:04:25.042 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.042 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.042 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:25.042 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.994282+0000", 2024-04-19T01:04:25.042 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00054962800000000003 2024-04-19T01:04:25.042 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.043 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.043 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:25.044 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.994320+0000", 2024-04-19T01:04:25.044 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.7824999999999998e-05 2024-04-19T01:04:25.044 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.044 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.044 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:25.044 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.994345+0000", 2024-04-19T01:04:25.044 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4295e-05 2024-04-19T01:04:25.044 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.044 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.044 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:25.045 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.994505+0000", 2024-04-19T01:04:25.045 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00016080700000000001 2024-04-19T01:04:25.045 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.045 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.045 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:25.045 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.994525+0000", 2024-04-19T01:04:25.045 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0049000000000001e-05 2024-04-19T01:04:25.045 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.045 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.045 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:25.045 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.018713+0000", 2024-04-19T01:04:25.045 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.024187810000000001 2024-04-19T01:04:25.046 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.046 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.046 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:25.046 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.018898+0000", 2024-04-19T01:04:25.046 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000184558 2024-04-19T01:04:25.046 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:25.046 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:25.046 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:25.046 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.046 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.046 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs5 43/31 ECSubWrite(tid=48, reqid=client.4604.0:264, at_version=43'9, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:04:25.046 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.006038+0000", 2024-04-19T01:04:25.046 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 11.007701043999999, 2024-04-19T01:04:25.046 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.018149728, 2024-04-19T01:04:25.047 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:25.047 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:25.047 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:25.047 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:25.047 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.047 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:25.047 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006038+0000", 2024-04-19T01:04:25.047 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:25.047 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.047 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.047 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:25.047 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006038+0000", 2024-04-19T01:04:25.047 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:25.047 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.047 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.048 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:25.048 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006040+0000", 2024-04-19T01:04:25.048 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0829999999999998e-06 2024-04-19T01:04:25.048 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.048 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.048 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:25.048 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006230+0000", 2024-04-19T01:04:25.048 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00018986799999999999 2024-04-19T01:04:25.048 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.048 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.048 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:25.048 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006274+0000", 2024-04-19T01:04:25.048 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.3671999999999998e-05 2024-04-19T01:04:25.048 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.049 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.049 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:25.049 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006299+0000", 2024-04-19T01:04:25.049 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5062000000000002e-05 2024-04-19T01:04:25.049 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.049 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.049 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:25.049 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.008623+0000", 2024-04-19T01:04:25.049 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0023245459999999998 2024-04-19T01:04:25.049 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.049 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.049 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:25.049 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.008653+0000", 2024-04-19T01:04:25.050 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.0212000000000001e-05 2024-04-19T01:04:25.050 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.050 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.050 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:25.050 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.024106+0000", 2024-04-19T01:04:25.050 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.015452815 2024-04-19T01:04:25.050 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.050 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.051 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:25.051 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.024188+0000", 2024-04-19T01:04:25.051 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.1470000000000004e-05 2024-04-19T01:04:25.051 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:25.051 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:25.051 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:25.051 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.051 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.051 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs5 43/31 ECSubWrite(tid=49, reqid=client.4604.0:265, at_version=43'10, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:04:25.051 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.006353+0000", 2024-04-19T01:04:25.051 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 11.007385651, 2024-04-19T01:04:25.051 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.024130236999999999, 2024-04-19T01:04:25.051 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:25.052 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:25.052 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:25.052 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:25.052 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.052 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:25.052 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006353+0000", 2024-04-19T01:04:25.052 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:25.052 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.052 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.052 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:25.052 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006353+0000", 2024-04-19T01:04:25.052 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:25.052 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.052 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.053 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:25.053 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006354+0000", 2024-04-19T01:04:25.053 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.0899999999999999e-06 2024-04-19T01:04:25.053 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.053 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.054 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:25.054 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006393+0000", 2024-04-19T01:04:25.054 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.8897999999999999e-05 2024-04-19T01:04:25.054 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.054 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.054 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:25.054 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006417+0000", 2024-04-19T01:04:25.054 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3881999999999999e-05 2024-04-19T01:04:25.054 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.054 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.054 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:25.055 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006436+0000", 2024-04-19T01:04:25.055 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.8488999999999999e-05 2024-04-19T01:04:25.055 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.055 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.055 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:25.055 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.029248+0000", 2024-04-19T01:04:25.055 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.022812136 2024-04-19T01:04:25.055 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.055 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.055 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:25.055 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.029278+0000", 2024-04-19T01:04:25.055 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.9702000000000001e-05 2024-04-19T01:04:25.055 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.055 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.055 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:25.056 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.030387+0000", 2024-04-19T01:04:25.056 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001109814 2024-04-19T01:04:25.056 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.056 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.056 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:25.056 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.030484+0000", 2024-04-19T01:04:25.056 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.6225999999999997e-05 2024-04-19T01:04:25.056 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:25.056 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:25.056 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:25.056 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.056 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.056 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:307 3.0s0 3:0abdaea3:::smithi15528762-5:head [append 0~442368] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:04:25.056 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:17.548168+0000", 2024-04-19T01:04:25.057 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 7.4655711059999996, 2024-04-19T01:04:25.057 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.016367771999999999, 2024-04-19T01:04:25.057 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:25.057 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:25.057 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:25.057 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:25.057 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:25.057 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:25.057 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 307 2024-04-19T01:04:25.057 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.057 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:25.057 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.057 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:25.057 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.548168+0000", 2024-04-19T01:04:25.058 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:25.058 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.058 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.058 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:25.058 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.548168+0000", 2024-04-19T01:04:25.058 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:25.058 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.058 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.058 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:25.058 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.548171+0000", 2024-04-19T01:04:25.058 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.9699999999999999e-06 2024-04-19T01:04:25.058 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.058 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.058 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:25.059 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.549150+0000", 2024-04-19T01:04:25.059 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0009788869999999999 2024-04-19T01:04:25.059 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.059 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.059 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:25.059 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.549184+0000", 2024-04-19T01:04:25.059 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.4068e-05 2024-04-19T01:04:25.059 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.059 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.059 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:25.059 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.549199+0000", 2024-04-19T01:04:25.059 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5373e-05 2024-04-19T01:04:25.059 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.059 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.060 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:25.060 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.549291+0000", 2024-04-19T01:04:25.060 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.1339000000000001e-05 2024-04-19T01:04:25.060 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.060 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.060 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:25.060 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.549424+0000", 2024-04-19T01:04:25.060 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00013346899999999999 2024-04-19T01:04:25.060 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.061 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.061 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:25.061 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.550698+0000", 2024-04-19T01:04:25.061 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001273707 2024-04-19T01:04:25.061 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.061 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.061 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:25.061 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.556021+0000", 2024-04-19T01:04:25.061 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0053232410000000003 2024-04-19T01:04:25.061 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.061 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.061 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:25.061 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564445+0000", 2024-04-19T01:04:25.062 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.008423692 2024-04-19T01:04:25.062 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.062 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.062 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:25.062 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564536+0000", 2024-04-19T01:04:25.062 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.1026000000000006e-05 2024-04-19T01:04:25.062 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:25.062 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:25.062 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:25.062 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.062 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.062 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:308 3.0s0 3:0abdaea3:::smithi15528762-5:head [append 0~720896] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:04:25.062 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:17.563573+0000", 2024-04-19T01:04:25.062 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 7.4501655060000003, 2024-04-19T01:04:25.063 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.022395149, 2024-04-19T01:04:25.063 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:25.063 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:25.063 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:25.063 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:25.063 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:25.063 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:25.063 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 308 2024-04-19T01:04:25.063 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.063 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:25.063 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.063 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:25.063 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.563573+0000", 2024-04-19T01:04:25.063 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:25.063 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.064 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.064 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:25.064 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.563573+0000", 2024-04-19T01:04:25.064 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:25.064 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.064 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.064 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:25.064 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.563575+0000", 2024-04-19T01:04:25.064 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6339999999999999e-06 2024-04-19T01:04:25.064 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.064 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.064 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:25.064 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564347+0000", 2024-04-19T01:04:25.064 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00077217599999999996 2024-04-19T01:04:25.065 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.065 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.065 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:25.065 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564361+0000", 2024-04-19T01:04:25.065 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.3665e-05 2024-04-19T01:04:25.065 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.065 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.065 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:25.065 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564372+0000", 2024-04-19T01:04:25.065 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.0834e-05 2024-04-19T01:04:25.065 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.065 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.065 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:25.065 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564954+0000", 2024-04-19T01:04:25.065 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00058264700000000005 2024-04-19T01:04:25.065 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.066 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.066 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:25.066 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.565068+0000", 2024-04-19T01:04:25.066 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000113174 2024-04-19T01:04:25.066 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.066 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.066 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:25.066 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.566092+0000", 2024-04-19T01:04:25.066 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0010239649999999999 2024-04-19T01:04:25.066 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.066 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.066 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:25.066 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.574730+0000", 2024-04-19T01:04:25.067 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0086380120000000005 2024-04-19T01:04:25.067 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.067 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.067 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:25.067 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.585859+0000", 2024-04-19T01:04:25.067 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.011129515 2024-04-19T01:04:25.067 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.067 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.067 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:25.067 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.585969+0000", 2024-04-19T01:04:25.067 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000109527 2024-04-19T01:04:25.067 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:25.067 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:25.067 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:25.068 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.068 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.068 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:309 3.0s0 3:0abdaea3:::smithi15528762-5:head [append 0~49152] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:04:25.068 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:17.564572+0000", 2024-04-19T01:04:25.068 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 7.4491666810000003, 2024-04-19T01:04:25.068 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.022344373000000001, 2024-04-19T01:04:25.068 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:25.068 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:25.068 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:25.070 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:25.070 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:25.070 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:25.070 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 309 2024-04-19T01:04:25.070 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.070 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:25.070 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.070 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:25.070 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564572+0000", 2024-04-19T01:04:25.070 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:25.071 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.071 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.071 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:25.071 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564572+0000", 2024-04-19T01:04:25.071 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:25.071 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.072 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.073 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:25.074 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564573+0000", 2024-04-19T01:04:25.074 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.5600000000000004e-07 2024-04-19T01:04:25.074 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.074 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.074 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:25.074 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564647+0000", 2024-04-19T01:04:25.074 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.4212e-05 2024-04-19T01:04:25.075 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.075 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.075 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:25.075 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564659+0000", 2024-04-19T01:04:25.075 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.1516e-05 2024-04-19T01:04:25.075 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.075 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.075 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:25.075 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564669+0000", 2024-04-19T01:04:25.075 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.9250000000000007e-06 2024-04-19T01:04:25.075 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.075 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.075 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:25.075 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.573061+0000", 2024-04-19T01:04:25.075 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0083917249999999992 2024-04-19T01:04:25.076 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.076 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.076 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:25.076 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.573228+0000", 2024-04-19T01:04:25.076 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00016785899999999999 2024-04-19T01:04:25.076 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.076 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.076 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:25.076 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.573957+0000", 2024-04-19T01:04:25.076 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00072819900000000001 2024-04-19T01:04:25.076 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.076 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.076 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:25.076 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.575986+0000", 2024-04-19T01:04:25.076 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0020295640000000002 2024-04-19T01:04:25.077 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.077 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.077 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:25.077 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.586818+0000", 2024-04-19T01:04:25.077 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.010831685000000001 2024-04-19T01:04:25.077 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.078 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.078 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:25.078 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.586917+0000", 2024-04-19T01:04:25.078 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.8732000000000003e-05 2024-04-19T01:04:25.078 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:25.080 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:25.080 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:25.080 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.081 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.081 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:310 3.0s0 3:0abdaea3:::smithi15528762-5:head [setxattr _header (55) in=7b] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:04:25.081 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:17.564685+0000", 2024-04-19T01:04:25.081 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 7.4490538830000004, 2024-04-19T01:04:25.081 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.022864789999999999, 2024-04-19T01:04:25.081 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:25.081 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:25.081 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:25.081 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:25.081 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:25.081 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:25.082 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 310 2024-04-19T01:04:25.082 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.082 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:25.082 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.082 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:25.082 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564685+0000", 2024-04-19T01:04:25.082 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:25.082 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.082 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.082 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:25.082 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564685+0000", 2024-04-19T01:04:25.082 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:25.082 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.082 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.083 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:25.083 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564686+0000", 2024-04-19T01:04:25.083 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.0999999999999997e-07 2024-04-19T01:04:25.083 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.083 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.083 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:25.083 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564690+0000", 2024-04-19T01:04:25.083 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.3100000000000002e-06 2024-04-19T01:04:25.083 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.083 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.083 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:25.083 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564699+0000", 2024-04-19T01:04:25.083 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.6640000000000004e-06 2024-04-19T01:04:25.083 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.083 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.084 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:25.084 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564705+0000", 2024-04-19T01:04:25.084 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.4910000000000002e-06 2024-04-19T01:04:25.084 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.084 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.084 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:25.084 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.575044+0000", 2024-04-19T01:04:25.084 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.010338293 2024-04-19T01:04:25.084 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.084 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.084 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:25.084 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.575112+0000", 2024-04-19T01:04:25.085 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.7898999999999995e-05 2024-04-19T01:04:25.085 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.085 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.085 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:25.085 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.575501+0000", 2024-04-19T01:04:25.085 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000389491 2024-04-19T01:04:25.085 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.085 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.085 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:25.085 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.576333+0000", 2024-04-19T01:04:25.085 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00083234599999999998 2024-04-19T01:04:25.085 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.085 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.085 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:25.085 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.587286+0000", 2024-04-19T01:04:25.086 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.010952437000000001 2024-04-19T01:04:25.086 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.086 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.086 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:25.086 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.587550+0000", 2024-04-19T01:04:25.086 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00026404899999999998 2024-04-19T01:04:25.086 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:25.086 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:25.086 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:25.086 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.086 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.086 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:311 3.0s0 3:0abdaea3:::smithi15528762-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:04:25.086 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:17.564725+0000", 2024-04-19T01:04:25.086 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 7.4490139830000004, 2024-04-19T01:04:25.086 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.026756121000000001, 2024-04-19T01:04:25.087 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:25.087 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:25.087 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:04:25.088 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:25.088 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:25.088 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:25.088 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 311 2024-04-19T01:04:25.088 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.088 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:25.088 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.088 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:25.088 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564725+0000", 2024-04-19T01:04:25.088 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:25.088 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.089 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.089 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:25.089 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564725+0000", 2024-04-19T01:04:25.089 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:25.089 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.089 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.089 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:25.089 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564726+0000", 2024-04-19T01:04:25.090 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.5700000000000002e-07 2024-04-19T01:04:25.090 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.090 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.090 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:25.090 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564730+0000", 2024-04-19T01:04:25.090 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.476e-06 2024-04-19T01:04:25.090 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.090 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.091 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:25.091 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564740+0000", 2024-04-19T01:04:25.092 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.2750000000000001e-06 2024-04-19T01:04:25.092 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.092 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.092 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:25.092 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564749+0000", 2024-04-19T01:04:25.092 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.1640000000000007e-06 2024-04-19T01:04:25.092 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.092 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.092 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:25.092 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.576057+0000", 2024-04-19T01:04:25.092 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.011308573000000001 2024-04-19T01:04:25.092 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.092 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.093 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting for rw locks", 2024-04-19T01:04:25.093 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.576308+0000", 2024-04-19T01:04:25.093 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00025028300000000002 2024-04-19T01:04:25.093 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.093 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.093 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:25.093 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.588445+0000", 2024-04-19T01:04:25.093 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.012137129 2024-04-19T01:04:25.093 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.093 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.093 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:25.093 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.588680+0000", 2024-04-19T01:04:25.093 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00023515400000000001 2024-04-19T01:04:25.093 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.093 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.094 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:25.094 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.591481+0000", 2024-04-19T01:04:25.094 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00280131 2024-04-19T01:04:25.094 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:25.094 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:25.094 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:25.094 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.094 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.094 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs5 48/31 ECSubWrite(tid=76, reqid=client.4604.0:345, at_version=48'22, trim_to=0'0, roll_forward_to=47'19))", 2024-04-19T01:04:25.094 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.538139+0000", 2024-04-19T01:04:25.094 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 6.4755999549999999, 2024-04-19T01:04:25.094 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.030818128, 2024-04-19T01:04:25.094 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:25.095 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:25.095 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:25.095 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:25.095 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.095 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:25.095 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538139+0000", 2024-04-19T01:04:25.095 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:25.095 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.095 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.095 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:25.095 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538139+0000", 2024-04-19T01:04:25.095 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:25.095 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.095 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.095 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:25.096 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538141+0000", 2024-04-19T01:04:25.096 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.796e-06 2024-04-19T01:04:25.096 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.096 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.096 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:25.096 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538876+0000", 2024-04-19T01:04:25.096 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00073526699999999995 2024-04-19T01:04:25.096 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.096 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.096 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:25.096 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538919+0000", 2024-04-19T01:04:25.096 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.3065000000000002e-05 2024-04-19T01:04:25.096 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.096 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.097 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:25.097 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538943+0000", 2024-04-19T01:04:25.097 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3578999999999999e-05 2024-04-19T01:04:25.097 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.097 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.097 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:25.097 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.539114+0000", 2024-04-19T01:04:25.097 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000171383 2024-04-19T01:04:25.097 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.097 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.097 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:25.097 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.539140+0000", 2024-04-19T01:04:25.097 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5709000000000001e-05 2024-04-19T01:04:25.100 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.100 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.100 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:25.100 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.568880+0000", 2024-04-19T01:04:25.100 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.029740128000000001 2024-04-19T01:04:25.100 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.100 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.100 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:25.101 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.568957+0000", 2024-04-19T01:04:25.101 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.7200999999999995e-05 2024-04-19T01:04:25.101 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:25.101 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:25.101 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:25.101 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.101 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.102 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs5 48/31 ECSubWrite(tid=78, reqid=client.4604.0:346, at_version=48'23, trim_to=0'0, roll_forward_to=48'21))", 2024-04-19T01:04:25.102 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.548746+0000", 2024-04-19T01:04:25.102 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 6.4649932799999998, 2024-04-19T01:04:25.102 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.023711396999999999, 2024-04-19T01:04:25.102 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:25.102 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:25.102 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:25.102 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:25.102 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.102 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:25.102 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.548746+0000", 2024-04-19T01:04:25.102 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:25.102 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.103 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.103 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:25.103 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.548746+0000", 2024-04-19T01:04:25.103 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:25.103 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.103 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.103 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:25.103 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.548747+0000", 2024-04-19T01:04:25.103 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5799999999999999e-06 2024-04-19T01:04:25.103 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.103 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.103 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:25.103 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.549144+0000", 2024-04-19T01:04:25.103 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00039714900000000002 2024-04-19T01:04:25.104 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.104 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.104 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:25.104 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.549174+0000", 2024-04-19T01:04:25.104 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.9652999999999999e-05 2024-04-19T01:04:25.104 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.104 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.104 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:25.104 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.549195+0000", 2024-04-19T01:04:25.104 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0585e-05 2024-04-19T01:04:25.104 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.104 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.104 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:25.104 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.554787+0000", 2024-04-19T01:04:25.104 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0055927160000000002 2024-04-19T01:04:25.105 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.105 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.105 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:25.105 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.554813+0000", 2024-04-19T01:04:25.105 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5669000000000001e-05 2024-04-19T01:04:25.105 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.105 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.105 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:25.105 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.572370+0000", 2024-04-19T01:04:25.105 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.017556852000000001 2024-04-19T01:04:25.105 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.105 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.105 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:25.107 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.572457+0000", 2024-04-19T01:04:25.107 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.7193000000000004e-05 2024-04-19T01:04:25.107 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:25.107 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:25.107 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:25.107 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.107 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.107 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs5 48/31 ECSubWrite(tid=79, reqid=client.4604.0:347, at_version=48'24, trim_to=0'0, roll_forward_to=48'21))", 2024-04-19T01:04:25.107 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.556212+0000", 2024-04-19T01:04:25.107 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 6.4575265130000004, 2024-04-19T01:04:25.107 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.018061275000000002, 2024-04-19T01:04:25.107 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:25.107 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:25.107 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:25.108 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:25.108 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.108 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:25.108 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.556212+0000", 2024-04-19T01:04:25.108 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:25.108 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.108 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.108 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:25.108 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.556212+0000", 2024-04-19T01:04:25.108 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:25.108 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.108 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.108 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:25.108 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.556214+0000", 2024-04-19T01:04:25.109 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.883e-06 2024-04-19T01:04:25.109 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.109 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.109 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:25.109 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.556284+0000", 2024-04-19T01:04:25.109 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.9491000000000003e-05 2024-04-19T01:04:25.109 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.109 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.109 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:25.109 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.556314+0000", 2024-04-19T01:04:25.109 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.0233e-05 2024-04-19T01:04:25.109 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.109 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.109 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:25.109 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.556335+0000", 2024-04-19T01:04:25.110 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1094000000000002e-05 2024-04-19T01:04:25.110 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.110 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.110 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:25.110 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.569301+0000", 2024-04-19T01:04:25.110 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.012965617 2024-04-19T01:04:25.110 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.111 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.111 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:25.111 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.569340+0000", 2024-04-19T01:04:25.111 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.9232e-05 2024-04-19T01:04:25.111 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.111 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.111 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:25.111 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.574195+0000", 2024-04-19T01:04:25.111 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0048553779999999996 2024-04-19T01:04:25.111 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.112 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.112 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:25.112 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.574274+0000", 2024-04-19T01:04:25.112 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.8347000000000001e-05 2024-04-19T01:04:25.112 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:25.112 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:25.112 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:25.112 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.112 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.113 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.as5 48/31 ECSubWrite(tid=71, reqid=client.4604.0:354, at_version=48'10, trim_to=0'0, roll_forward_to=34'8))", 2024-04-19T01:04:25.113 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.557068+0000", 2024-04-19T01:04:25.113 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 6.456671257, 2024-04-19T01:04:25.113 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.029586061, 2024-04-19T01:04:25.113 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:25.113 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:25.113 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:25.113 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:25.113 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.113 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:25.113 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557068+0000", 2024-04-19T01:04:25.113 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:25.113 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.113 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.114 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:25.114 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557068+0000", 2024-04-19T01:04:25.114 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:25.114 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.114 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.114 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:25.114 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557069+0000", 2024-04-19T01:04:25.114 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.9500000000000001e-07 2024-04-19T01:04:25.114 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.114 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.114 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:25.114 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.558172+0000", 2024-04-19T01:04:25.114 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001103637 2024-04-19T01:04:25.115 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.115 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.115 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:25.115 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.558206+0000", 2024-04-19T01:04:25.115 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.4069000000000002e-05 2024-04-19T01:04:25.115 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.115 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.115 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:25.115 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.558229+0000", 2024-04-19T01:04:25.115 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2495999999999999e-05 2024-04-19T01:04:25.115 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.115 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.116 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:25.116 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.558450+0000", 2024-04-19T01:04:25.116 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000221747 2024-04-19T01:04:25.116 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.116 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.116 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:25.116 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.558476+0000", 2024-04-19T01:04:25.116 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5755999999999999e-05 2024-04-19T01:04:25.116 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.116 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.116 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:25.116 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.586611+0000", 2024-04-19T01:04:25.116 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.028134731999999999 2024-04-19T01:04:25.116 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.116 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.117 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:25.117 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.586654+0000", 2024-04-19T01:04:25.117 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.2828999999999999e-05 2024-04-19T01:04:25.117 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:25.117 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:25.117 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:25.117 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.117 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.117 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.8s2 49/31 ECSubWrite(tid=202, reqid=osd.11.0:201, at_version=0'0, trim_to=0'0, roll_forward_to=49'24))", 2024-04-19T01:04:25.117 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:19.535477+0000", 2024-04-19T01:04:25.117 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 5.4782620529999999, 2024-04-19T01:04:25.117 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.020657227, 2024-04-19T01:04:25.117 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:25.117 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:25.117 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:25.118 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:25.118 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.118 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:25.118 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.535477+0000", 2024-04-19T01:04:25.118 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:25.118 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.118 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.118 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:25.118 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.535477+0000", 2024-04-19T01:04:25.118 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:25.118 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.118 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.118 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:25.118 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.535479+0000", 2024-04-19T01:04:25.119 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6700000000000001e-06 2024-04-19T01:04:25.119 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.119 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.119 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:25.119 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.535504+0000", 2024-04-19T01:04:25.119 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5626000000000002e-05 2024-04-19T01:04:25.119 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.119 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.119 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:25.119 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.535538+0000", 2024-04-19T01:04:25.119 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.3457999999999997e-05 2024-04-19T01:04:25.119 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.119 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.120 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:25.120 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.535560+0000", 2024-04-19T01:04:25.120 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2503000000000001e-05 2024-04-19T01:04:25.120 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.120 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.120 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:25.120 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.536980+0000", 2024-04-19T01:04:25.120 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0014200700000000001 2024-04-19T01:04:25.120 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.120 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.120 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:25.121 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.537005+0000", 2024-04-19T01:04:25.121 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4579e-05 2024-04-19T01:04:25.121 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.121 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.121 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:25.121 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.556071+0000", 2024-04-19T01:04:25.121 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.019066422999999999 2024-04-19T01:04:25.121 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.121 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.121 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:25.121 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.556134+0000", 2024-04-19T01:04:25.122 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.2898000000000007e-05 2024-04-19T01:04:25.122 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:25.122 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:25.122 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:25.122 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.122 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.122 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.8s2 49/31 ECSubWrite(tid=203, reqid=client.4604.0:360, at_version=49'25, trim_to=0'0, roll_forward_to=49'24))", 2024-04-19T01:04:25.123 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:19.538361+0000", 2024-04-19T01:04:25.123 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 5.4753778210000004, 2024-04-19T01:04:25.123 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.028842518000000001, 2024-04-19T01:04:25.123 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:25.123 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:25.123 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:25.123 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:25.123 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.123 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:25.123 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.538361+0000", 2024-04-19T01:04:25.123 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:25.123 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.123 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.123 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:25.124 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.538361+0000", 2024-04-19T01:04:25.124 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:25.124 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.124 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.124 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:25.124 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.538363+0000", 2024-04-19T01:04:25.124 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7600000000000001e-06 2024-04-19T01:04:25.124 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.124 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.124 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:25.124 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.540481+0000", 2024-04-19T01:04:25.124 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.002117833 2024-04-19T01:04:25.124 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.124 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.125 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:25.125 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.540516+0000", 2024-04-19T01:04:25.125 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.5435000000000002e-05 2024-04-19T01:04:25.125 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.125 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.125 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:25.125 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.540539+0000", 2024-04-19T01:04:25.125 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3136e-05 2024-04-19T01:04:25.125 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.125 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.125 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:25.125 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.540906+0000", 2024-04-19T01:04:25.125 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00036633199999999997 2024-04-19T01:04:25.125 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.126 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.126 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:25.126 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.540932+0000", 2024-04-19T01:04:25.126 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5972000000000001e-05 2024-04-19T01:04:25.126 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.126 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.126 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:25.126 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.567132+0000", 2024-04-19T01:04:25.126 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.026200655999999999 2024-04-19T01:04:25.126 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.126 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.126 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:25.126 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.567204+0000", 2024-04-19T01:04:25.126 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.1394000000000001e-05 2024-04-19T01:04:25.127 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:25.127 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:25.127 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:25.127 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.127 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.127 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:365 3.0s0 3:0e44b6f6:::smithi15528762-26:head [read 0~1605632,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-04-19T01:04:25.127 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:19.549248+0000", 2024-04-19T01:04:25.127 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 5.4644912330000004, 2024-04-19T01:04:25.127 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.026478386999999999, 2024-04-19T01:04:25.127 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:25.127 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:25.127 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:04:25.127 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:25.127 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:25.128 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:25.128 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 365 2024-04-19T01:04:25.128 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.128 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:25.128 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.128 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:25.128 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549248+0000", 2024-04-19T01:04:25.128 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:25.128 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.128 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.128 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:25.129 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549248+0000", 2024-04-19T01:04:25.129 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:25.129 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.129 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.129 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:25.129 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549251+0000", 2024-04-19T01:04:25.129 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.8169999999999999e-06 2024-04-19T01:04:25.129 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.129 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.129 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:25.129 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549262+0000", 2024-04-19T01:04:25.129 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.1449e-05 2024-04-19T01:04:25.129 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.129 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.130 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:25.130 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549288+0000", 2024-04-19T01:04:25.130 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6169e-05 2024-04-19T01:04:25.130 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.130 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.130 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:25.130 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549312+0000", 2024-04-19T01:04:25.130 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3833e-05 2024-04-19T01:04:25.130 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.130 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.130 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:25.130 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.566277+0000", 2024-04-19T01:04:25.130 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.016964888000000001 2024-04-19T01:04:25.130 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.131 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.131 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:25.131 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.566503+0000", 2024-04-19T01:04:25.131 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00022616800000000001 2024-04-19T01:04:25.131 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.131 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.131 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:25.131 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.575726+0000", 2024-04-19T01:04:25.131 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0092230630000000001 2024-04-19T01:04:25.131 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:25.131 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:25.131 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:25.131 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.131 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.131 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:366 3.0s0 3:0e44b6f6:::smithi15528762-26:head [read 0~1605632,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-04-19T01:04:25.132 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:19.549572+0000", 2024-04-19T01:04:25.132 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 5.464166809, 2024-04-19T01:04:25.132 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.028385865999999999, 2024-04-19T01:04:25.132 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:25.132 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:25.132 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:04:25.132 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:25.132 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:25.132 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:25.132 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 366 2024-04-19T01:04:25.132 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.133 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:25.133 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.133 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:25.133 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549572+0000", 2024-04-19T01:04:25.133 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:25.133 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.133 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.133 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:25.134 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549572+0000", 2024-04-19T01:04:25.134 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:25.134 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.134 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.134 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:25.134 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549574+0000", 2024-04-19T01:04:25.134 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0260000000000002e-06 2024-04-19T01:04:25.134 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.134 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.134 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:25.134 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549588+0000", 2024-04-19T01:04:25.134 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.3737000000000001e-05 2024-04-19T01:04:25.134 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.134 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.135 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:25.135 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549616+0000", 2024-04-19T01:04:25.135 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.7619000000000001e-05 2024-04-19T01:04:25.135 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.135 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.135 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:25.135 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549633+0000", 2024-04-19T01:04:25.135 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7025999999999999e-05 2024-04-19T01:04:25.135 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.135 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.135 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:25.135 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.567197+0000", 2024-04-19T01:04:25.135 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.017564377999999999 2024-04-19T01:04:25.135 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.136 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.136 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:25.136 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.567461+0000", 2024-04-19T01:04:25.136 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00026393899999999998 2024-04-19T01:04:25.136 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.136 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.136 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:25.136 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.577958+0000", 2024-04-19T01:04:25.136 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.010497141 2024-04-19T01:04:25.136 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:25.136 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:25.136 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:25.136 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.136 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.136 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:367 3.0s0 3:0e44b6f6:::smithi15528762-26:head [read 0~1605632,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-04-19T01:04:25.137 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:19.549659+0000", 2024-04-19T01:04:25.137 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 5.4640800020000002, 2024-04-19T01:04:25.137 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.030292887000000001, 2024-04-19T01:04:25.137 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:25.137 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:25.137 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:04:25.137 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:25.137 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:25.137 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:25.137 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 367 2024-04-19T01:04:25.137 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.137 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:25.137 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.137 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:25.138 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549659+0000", 2024-04-19T01:04:25.138 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:25.138 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.138 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.138 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:25.138 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549659+0000", 2024-04-19T01:04:25.138 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:25.138 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.138 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.138 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:25.138 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549661+0000", 2024-04-19T01:04:25.138 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2500000000000001e-06 2024-04-19T01:04:25.138 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.138 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.138 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:25.139 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549669+0000", 2024-04-19T01:04:25.139 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.7060000000000004e-06 2024-04-19T01:04:25.139 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.139 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.139 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:25.139 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549688+0000", 2024-04-19T01:04:25.139 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9309000000000001e-05 2024-04-19T01:04:25.139 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.139 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.139 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:25.139 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549704+0000", 2024-04-19T01:04:25.139 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5398999999999999e-05 2024-04-19T01:04:25.139 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.139 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.140 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:25.140 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.570745+0000", 2024-04-19T01:04:25.140 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.021041432999999998 2024-04-19T01:04:25.140 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.140 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.140 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:25.140 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.570983+0000", 2024-04-19T01:04:25.140 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00023819299999999999 2024-04-19T01:04:25.140 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:25.140 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:25.140 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:25.140 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.579952+0000", 2024-04-19T01:04:25.140 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.008968597 2024-04-19T01:04:25.140 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:25.140 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:25.141 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:25.141 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:25.141 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:25.141 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:25.142 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:04:25.043+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:04:25.143 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:04:25.141+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:04:25.144 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-19T01:04:25.159 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-04-19T01:04:25.159 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T01:04:25.180 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-04-19T01:04:25.181 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T01:04:25.201 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-04-19T01:04:25.201 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T01:04:25.222 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-04-19T01:04:25.223 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T01:04:25.241 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-04-19T01:04:25.241 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T01:04:25.244 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:04:25.244+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:04:25.259 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-04-19T01:04:25.260 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T01:04:25.281 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-04-19T01:04:25.281 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T01:04:25.305 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-04-19T01:04:25.305 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T01:04:25.315 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:25.315 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:25.325 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2024-04-19T01:04:25.325 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T01:04:25.330 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-19T01:04:25.346 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:04:25.339+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:04:25.346 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2024-04-19T01:04:25.347 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T01:04:25.368 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2024-04-19T01:04:25.368 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T01:04:25.391 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2024-04-19T01:04:25.391 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T01:04:25.411 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2024-04-19T01:04:25.411 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-19T01:04:25.434 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2024-04-19T01:04:25.435 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-19T01:04:25.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:04:25.439+0000 7f1bb8dc7640 0 -- 172.21.15.33:0/2845453368 >> [v2:172.21.15.118:6800/506416463,v1:172.21.15.118:6801/506416463] conn(0x7f1bb42221e0 msgr2=0x7f1bb42245d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-19T01:04:25.444 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:04:25.444+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:04:25.453 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:25.454 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:25.461 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2024-04-19T01:04:25.461 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-19T01:04:25.467 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-19T01:04:25.486 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2024-04-19T01:04:25.486 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-19T01:04:25.522 INFO:tasks.rados.rados.0.smithi155.stdout:149: done (0 left) 2024-04-19T01:04:25.522 INFO:tasks.rados.rados.0.smithi155.stdout:150: setattr oid 25 current snap is 11 2024-04-19T01:04:25.528 INFO:tasks.rados.rados.0.smithi155.stdout:151: snap_create 2024-04-19T01:04:25.530 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-19T01:04:25.542 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 25 v 20 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-04-19T01:04:25.545 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:04:25.542+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:04:25.594 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:25.594 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:25.609 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-19T01:04:25.647 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:04:25.645+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:04:25.747 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:04:25.749+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:04:25.774 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:25.774 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:25.788 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-19T01:04:25.849 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:04:25.848+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:04:25.911 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:25.911 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:25.925 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-19T01:04:25.949 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:04:25.947+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:04:26.049 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:26.049 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:26.050 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:04:26.049+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:04:26.062 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-04-19T01:04:26.151 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:26.150+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:26.186 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:26.186 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:26.200 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-04-19T01:04:26.286 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:26.286+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:26.353 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:04:26.355+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:04:26.366 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:26.367 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:26.380 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-04-19T01:04:26.455 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:04:26.453+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:04:26.499 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2024-04-19T01:04:26.504 INFO:tasks.rados.rados.0.smithi155.stdout:150: done (1 left) 2024-04-19T01:04:26.504 INFO:tasks.rados.rados.0.smithi155.stdout:151: done (0 left) 2024-04-19T01:04:26.504 INFO:tasks.rados.rados.0.smithi155.stdout:152: read oid 27 snap -1 2024-04-19T01:04:26.504 INFO:tasks.rados.rados.0.smithi155.stdout:152: expect (ObjNum 26 snap 0 seq_num 26) 2024-04-19T01:04:26.504 INFO:tasks.rados.rados.0.smithi155.stdout:153: snap_create 2024-04-19T01:04:26.516 INFO:tasks.rados.rados.0.smithi155.stdout:152: expect (ObjNum 26 snap 0 seq_num 26) 2024-04-19T01:04:26.549 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:26.549 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:26.563 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-19T01:04:26.569 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:04:26.571+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:04:26.658 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:26.658+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:26.694 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:26.694 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:26.709 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-19T01:04:26.758 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:04:26.759+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:04:26.829 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:26.830 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:26.844 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-19T01:04:26.864 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:04:26.862+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:04:26.965 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:04:26.964+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:04:26.967 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:26.967 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:26.981 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-19T01:04:27.068 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:04:27.068+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:04:27.143 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:27.143 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:27.157 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-19T01:04:27.169 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:04:27.169+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:04:27.270 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:04:27.272+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:04:27.327 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:27.327 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:27.336 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-19T01:04:27.371 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:04:27.369+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:04:27.472 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:04:27.473+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:04:27.508 INFO:tasks.rados.rados.0.smithi155.stdout:152: done (1 left) 2024-04-19T01:04:27.509 INFO:tasks.rados.rados.0.smithi155.stdout:153: done (0 left) 2024-04-19T01:04:27.509 INFO:tasks.rados.rados.0.smithi155.stdout:154: copy_from oid 43 from oid 31 current snap is 13 2024-04-19T01:04:27.509 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 7 current snap is 13 2024-04-19T01:04:27.509 INFO:tasks.rados.rados.0.smithi155.stdout:155: seq_num 80 ranges {114688=589824,704512=573440,1277952=196608} 2024-04-19T01:04:27.509 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-19T01:04:27.510 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:27.510 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:27.520 INFO:tasks.rados.rados.0.smithi155.stdout:155: writing smithi15528762-7 from 114688 to 704512 tid 1 2024-04-19T01:04:27.526 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-19T01:04:27.530 INFO:tasks.rados.rados.0.smithi155.stdout:155: writing smithi15528762-7 from 704512 to 1277952 tid 2 2024-04-19T01:04:27.536 INFO:tasks.rados.rados.0.smithi155.stdout:155: writing smithi15528762-7 from 1277952 to 1474560 tid 3 2024-04-19T01:04:27.536 INFO:tasks.rados.rados.0.smithi155.stdout:156: rollback oid 32 current snap is 13 2024-04-19T01:04:27.536 INFO:tasks.rados.rados.0.smithi155.stdout:rollback oid 32 to 10 2024-04-19T01:04:27.536 INFO:tasks.rados.rados.0.smithi155.stdout:154: finishing copy_from to smithi15528762-43 2024-04-19T01:04:27.536 INFO:tasks.rados.rados.0.smithi155.stdout:154: got expected ENOENT (src dne) 2024-04-19T01:04:27.536 INFO:tasks.rados.rados.0.smithi155.stdout:154: finishing copy_from racing read to smithi15528762-43 2024-04-19T01:04:27.536 INFO:tasks.rados.rados.0.smithi155.stdout:154: done (2 left) 2024-04-19T01:04:27.536 INFO:tasks.rados.rados.0.smithi155.stdout:157: read oid 37 snap -1 2024-04-19T01:04:27.536 INFO:tasks.rados.rados.0.smithi155.stdout:157: expect deleted 2024-04-19T01:04:27.536 INFO:tasks.rados.rados.0.smithi155.stdout:158: read oid 33 snap 11 2024-04-19T01:04:27.536 INFO:tasks.rados.rados.0.smithi155.stdout:158: expect (ObjNum 32 snap 0 seq_num 32) 2024-04-19T01:04:27.536 INFO:tasks.rados.rados.0.smithi155.stdout:159: read oid 15 snap 12 2024-04-19T01:04:27.537 INFO:tasks.rados.rados.0.smithi155.stdout:159: expect (ObjNum 14 snap 0 seq_num 14) 2024-04-19T01:04:27.537 INFO:tasks.rados.rados.0.smithi155.stdout:160: delete oid 43 current snap is 13 2024-04-19T01:04:27.545 INFO:tasks.rados.rados.0.smithi155.stdout:159: expect (ObjNum 14 snap 0 seq_num 14) 2024-04-19T01:04:27.569 INFO:tasks.rados.rados.0.smithi155.stdout:158: expect (ObjNum 32 snap 0 seq_num 32) 2024-04-19T01:04:27.572 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:04:27.570+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:04:27.608 INFO:tasks.rados.rados.0.smithi155.stdout:156: finishing rollback tid 0 to smithi15528762-32 2024-04-19T01:04:27.608 INFO:tasks.rados.rados.0.smithi155.stdout:156: finishing rollback tid 1 to smithi15528762-32 2024-04-19T01:04:27.608 INFO:tasks.rados.rados.0.smithi155.stdout:156: finishing rollback tid 2 to smithi15528762-32 2024-04-19T01:04:27.608 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 32 v 87 (ObjNum 209 snap 0 seq_num 1588497319) dirty exists 2024-04-19T01:04:27.608 INFO:tasks.rados.rados.0.smithi155.stdout:155: finishing write tid 1 to smithi15528762-7 2024-04-19T01:04:27.608 INFO:tasks.rados.rados.0.smithi155.stdout:155: finishing write tid 2 to smithi15528762-7 2024-04-19T01:04:27.608 INFO:tasks.rados.rados.0.smithi155.stdout:155: finishing write tid 3 to smithi15528762-7 2024-04-19T01:04:27.608 INFO:tasks.rados.rados.0.smithi155.stdout:155: finishing write tid 4 to smithi15528762-7 2024-04-19T01:04:27.608 INFO:tasks.rados.rados.0.smithi155.stdout:155: finishing write tid 5 to smithi15528762-7 2024-04-19T01:04:27.608 INFO:tasks.rados.rados.0.smithi155.stdout:155: oid 7 updating version 0 to 23 2024-04-19T01:04:27.609 INFO:tasks.rados.rados.0.smithi155.stdout:155: oid 7 version 23 is already newer than 22 2024-04-19T01:04:27.609 INFO:tasks.rados.rados.0.smithi155.stdout:155: oid 7 updating version 23 to 24 2024-04-19T01:04:27.609 INFO:tasks.rados.rados.0.smithi155.stdout:155: oid 7 updating version 24 to 25 2024-04-19T01:04:27.609 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 7 v 25 (ObjNum 80 snap 13 seq_num 80) dirty exists 2024-04-19T01:04:27.609 INFO:tasks.rados.rados.0.smithi155.stdout:155: left oid 7 (ObjNum 80 snap 13 seq_num 80) 2024-04-19T01:04:27.609 INFO:tasks.rados.rados.0.smithi155.stdout:155: done (5 left) 2024-04-19T01:04:27.609 INFO:tasks.rados.rados.0.smithi155.stdout:156: done (4 left) 2024-04-19T01:04:27.609 INFO:tasks.rados.rados.0.smithi155.stdout:157: done (3 left) 2024-04-19T01:04:27.609 INFO:tasks.rados.rados.0.smithi155.stdout:158: done (2 left) 2024-04-19T01:04:27.609 INFO:tasks.rados.rados.0.smithi155.stdout:159: done (1 left) 2024-04-19T01:04:27.609 INFO:tasks.rados.rados.0.smithi155.stdout:160: done (0 left) 2024-04-19T01:04:27.609 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 33 current snap is 13 2024-04-19T01:04:27.609 INFO:tasks.rados.rados.0.smithi155.stdout:161: seq_num 82 ranges {540672=622592,1163264=475136,1638400=294912} 2024-04-19T01:04:27.626 INFO:tasks.rados.rados.0.smithi155.stdout:161: writing smithi15528762-33 from 540672 to 1163264 tid 1 2024-04-19T01:04:27.635 INFO:tasks.rados.rados.0.smithi155.stdout:161: writing smithi15528762-33 from 1163264 to 1638400 tid 2 2024-04-19T01:04:27.640 INFO:tasks.rados.rados.0.smithi155.stdout:161: writing smithi15528762-33 from 1638400 to 1933312 tid 3 2024-04-19T01:04:27.640 INFO:tasks.rados.rados.0.smithi155.stdout:162: delete oid 47 current snap is 13 2024-04-19T01:04:27.652 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:27.653 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:27.653 INFO:tasks.rados.rados.0.smithi155.stdout:162: done (1 left) 2024-04-19T01:04:27.653 INFO:tasks.rados.rados.0.smithi155.stdout:163: read oid 4 snap -1 2024-04-19T01:04:27.653 INFO:tasks.rados.rados.0.smithi155.stdout:163: expect deleted 2024-04-19T01:04:27.654 INFO:tasks.rados.rados.0.smithi155.stdout:164: setattr oid 31 current snap is 13 2024-04-19T01:04:27.659 INFO:tasks.rados.rados.0.smithi155.stdout:163: done (2 left) 2024-04-19T01:04:27.659 INFO:tasks.rados.rados.0.smithi155.stdout:165: read oid 2 snap -1 2024-04-19T01:04:27.659 INFO:tasks.rados.rados.0.smithi155.stdout:165: expect (ObjNum 76 snap 8 seq_num 76) 2024-04-19T01:04:27.659 INFO:tasks.rados.rados.0.smithi155.stdout:166: snap_remove snap 12 2024-04-19T01:04:27.669 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-19T01:04:27.672 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:04:27.673+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:04:27.684 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-04-19T01:04:27.684 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T01:04:27.705 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-04-19T01:04:27.705 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T01:04:27.725 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-04-19T01:04:27.725 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T01:04:27.750 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-04-19T01:04:27.751 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T01:04:27.769 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-04-19T01:04:27.769 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T01:04:27.777 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:04:27.774+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:04:27.790 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:27.790 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:27.792 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-04-19T01:04:27.792 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T01:04:27.807 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-19T01:04:27.814 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-04-19T01:04:27.814 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T01:04:27.876 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:04:27.874+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:04:27.930 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:27.930 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:27.946 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-04-19T01:04:27.977 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:04:27.974+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:04:28.044 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-04-19T01:04:28.044 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T01:04:28.065 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:28.066 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:28.077 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2024-04-19T01:04:28.077 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T01:04:28.077 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:04:28.079+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:04:28.078 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-04-19T01:04:28.115 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2024-04-19T01:04:28.115 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T01:04:28.147 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2024-04-19T01:04:28.147 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T01:04:28.178 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:04:28.178+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:04:28.181 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2024-04-19T01:04:28.182 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T01:04:28.194 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:28.194 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:28.209 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-04-19T01:04:28.215 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2024-04-19T01:04:28.215 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-19T01:04:28.244 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2024-04-19T01:04:28.244 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-19T01:04:28.273 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2024-04-19T01:04:28.273 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-19T01:04:28.278 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:28.280+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:28.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:04:28.303+0000 7fb99759e640 0 -- 172.21.15.33:0/2914389506 >> [v2:172.21.15.155:6807/741132602,v1:172.21.15.155:6810/741132602] conn(0x7fb9982263b0 msgr2=0x7fb9982287a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-19T01:04:28.310 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2024-04-19T01:04:28.334 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:04:28.334 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-04-19T01:04:28.334 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-04-19T01:04:28.334 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-04-19T01:04:28.334 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.334 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s2 39/31 ECSubWrite(tid=32, reqid=client.4604.0:181, at_version=39'16, trim_to=0'0, roll_forward_to=38'14))", 2024-04-19T01:04:28.334 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:09.821438+0000", 2024-04-19T01:04:28.334 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 18.513710917000001, 2024-04-19T01:04:28.335 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020304547999999999, 2024-04-19T01:04:28.335 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:28.335 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:28.335 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:28.335 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:28.335 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.335 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:28.335 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.821438+0000", 2024-04-19T01:04:28.335 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:28.335 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.335 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.335 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:28.335 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.821438+0000", 2024-04-19T01:04:28.336 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:28.336 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.336 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.336 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:28.336 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.821439+0000", 2024-04-19T01:04:28.336 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2699999999999999e-07 2024-04-19T01:04:28.336 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.336 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.336 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:28.336 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.822000+0000", 2024-04-19T01:04:28.336 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00056130600000000002 2024-04-19T01:04:28.336 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.336 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.337 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:28.337 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.822034+0000", 2024-04-19T01:04:28.337 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3785000000000003e-05 2024-04-19T01:04:28.337 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.337 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.337 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:28.337 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.822057+0000", 2024-04-19T01:04:28.337 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3289e-05 2024-04-19T01:04:28.337 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.337 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.337 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:28.337 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.822234+0000", 2024-04-19T01:04:28.337 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017660000000000001 2024-04-19T01:04:28.337 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.338 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.338 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:28.338 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.822249+0000", 2024-04-19T01:04:28.338 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5415999999999999e-05 2024-04-19T01:04:28.338 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.340 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.341 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:28.341 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.841628+0000", 2024-04-19T01:04:28.341 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019379159999999999 2024-04-19T01:04:28.341 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.341 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.341 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:28.341 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.841743+0000", 2024-04-19T01:04:28.341 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000114265 2024-04-19T01:04:28.341 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:28.341 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:28.341 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:28.341 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.341 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.342 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s2 39/31 ECSubWrite(tid=33, reqid=client.4604.0:182, at_version=39'17, trim_to=0'0, roll_forward_to=38'14))", 2024-04-19T01:04:28.342 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:09.829850+0000", 2024-04-19T01:04:28.342 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 18.505299394000001, 2024-04-19T01:04:28.342 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012982403999999999, 2024-04-19T01:04:28.342 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:28.342 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:28.342 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:28.342 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:28.342 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.342 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:28.342 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.829850+0000", 2024-04-19T01:04:28.342 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:28.342 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.342 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.343 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:28.343 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.829850+0000", 2024-04-19T01:04:28.343 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:28.343 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.343 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.343 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:28.343 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.829851+0000", 2024-04-19T01:04:28.343 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7269999999999999e-06 2024-04-19T01:04:28.344 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.344 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.344 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:28.344 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.830044+0000", 2024-04-19T01:04:28.344 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000192449 2024-04-19T01:04:28.344 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.345 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.345 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:28.345 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.830072+0000", 2024-04-19T01:04:28.345 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8492000000000001e-05 2024-04-19T01:04:28.345 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.345 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.350 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:28.350 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.830088+0000", 2024-04-19T01:04:28.350 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5423000000000001e-05 2024-04-19T01:04:28.350 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.350 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.350 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:28.351 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.834720+0000", 2024-04-19T01:04:28.351 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0046326320000000002 2024-04-19T01:04:28.351 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.351 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.351 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:28.352 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.834735+0000", 2024-04-19T01:04:28.352 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.468e-05 2024-04-19T01:04:28.352 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.352 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.352 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:28.352 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.842759+0000", 2024-04-19T01:04:28.352 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0080238400000000008 2024-04-19T01:04:28.353 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.353 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.353 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:28.353 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.842832+0000", 2024-04-19T01:04:28.353 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3161e-05 2024-04-19T01:04:28.353 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:28.353 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:28.354 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:28.355 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.355 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.355 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s2 39/31 ECSubWrite(tid=38, reqid=client.4604.0:187, at_version=39'20, trim_to=0'0, roll_forward_to=39'18))", 2024-04-19T01:04:28.355 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:09.890157+0000", 2024-04-19T01:04:28.355 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 18.444992428999999, 2024-04-19T01:04:28.355 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025377373000000002, 2024-04-19T01:04:28.355 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:28.355 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:28.355 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:28.355 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:28.356 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.356 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:28.356 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.890157+0000", 2024-04-19T01:04:28.356 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:28.356 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.357 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.357 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:28.357 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.890157+0000", 2024-04-19T01:04:28.357 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:28.357 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.357 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.357 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:28.358 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.890159+0000", 2024-04-19T01:04:28.358 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3140000000000002e-06 2024-04-19T01:04:28.359 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.359 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.359 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:28.360 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.890514+0000", 2024-04-19T01:04:28.360 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035502899999999998 2024-04-19T01:04:28.360 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.360 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.360 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:28.360 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.890556+0000", 2024-04-19T01:04:28.360 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1819000000000001e-05 2024-04-19T01:04:28.360 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.360 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.360 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:28.360 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.890581+0000", 2024-04-19T01:04:28.360 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5548999999999999e-05 2024-04-19T01:04:28.361 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.361 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.361 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:28.361 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.890752+0000", 2024-04-19T01:04:28.361 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000170147 2024-04-19T01:04:28.361 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.361 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.361 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:28.361 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.890781+0000", 2024-04-19T01:04:28.361 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9275e-05 2024-04-19T01:04:28.361 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.365 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.365 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:28.365 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.915444+0000", 2024-04-19T01:04:28.366 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024662726999999999 2024-04-19T01:04:28.366 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.366 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.366 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:28.366 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.915534+0000", 2024-04-19T01:04:28.366 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0513000000000006e-05 2024-04-19T01:04:28.366 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:28.366 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:28.366 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:28.366 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.366 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.366 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s2 39/31 ECSubWrite(tid=39, reqid=client.4604.0:188, at_version=39'21, trim_to=0'0, roll_forward_to=39'18))", 2024-04-19T01:04:28.366 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:09.897182+0000", 2024-04-19T01:04:28.366 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 18.437966834000001, 2024-04-19T01:04:28.367 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018721683999999999, 2024-04-19T01:04:28.367 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:28.367 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:28.367 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:28.367 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:28.367 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.367 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:28.367 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.897182+0000", 2024-04-19T01:04:28.367 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:28.367 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.367 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.367 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:28.367 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.897182+0000", 2024-04-19T01:04:28.368 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:28.368 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.368 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.368 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:28.368 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.897185+0000", 2024-04-19T01:04:28.368 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4099999999999998e-06 2024-04-19T01:04:28.368 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.368 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.368 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:28.368 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.897464+0000", 2024-04-19T01:04:28.368 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00027915900000000002 2024-04-19T01:04:28.368 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.368 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.368 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:28.369 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.897515+0000", 2024-04-19T01:04:28.369 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0778000000000001e-05 2024-04-19T01:04:28.369 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.369 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.369 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:28.369 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.897536+0000", 2024-04-19T01:04:28.369 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1685999999999999e-05 2024-04-19T01:04:28.369 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.369 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.369 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:28.369 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.905546+0000", 2024-04-19T01:04:28.369 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0080096409999999996 2024-04-19T01:04:28.369 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.369 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.369 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:28.370 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.905571+0000", 2024-04-19T01:04:28.370 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.482e-05 2024-04-19T01:04:28.370 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.370 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.370 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:28.370 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.915777+0000", 2024-04-19T01:04:28.370 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010205881999999999 2024-04-19T01:04:28.370 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.370 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.370 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:28.370 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.915904+0000", 2024-04-19T01:04:28.370 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012730799999999999 2024-04-19T01:04:28.370 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:28.371 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:28.371 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:28.371 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.371 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.371 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s2 39/31 ECSubWrite(tid=40, reqid=client.4604.0:189, at_version=39'22, trim_to=0'0, roll_forward_to=39'18))", 2024-04-19T01:04:28.371 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:09.900556+0000", 2024-04-19T01:04:28.371 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 18.434593286999998, 2024-04-19T01:04:28.371 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016123281, 2024-04-19T01:04:28.371 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:28.371 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:28.371 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:28.371 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:28.371 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.371 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:28.371 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.900556+0000", 2024-04-19T01:04:28.372 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:28.372 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.372 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.372 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:28.372 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.900556+0000", 2024-04-19T01:04:28.372 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:28.372 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.372 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.372 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:28.372 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.900558+0000", 2024-04-19T01:04:28.372 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0159999999999998e-06 2024-04-19T01:04:28.372 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.372 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.372 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:28.373 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.900599+0000", 2024-04-19T01:04:28.373 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1424999999999997e-05 2024-04-19T01:04:28.373 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.373 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.373 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:28.373 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.900639+0000", 2024-04-19T01:04:28.373 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9916000000000002e-05 2024-04-19T01:04:28.373 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.373 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.373 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:28.373 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.900662+0000", 2024-04-19T01:04:28.373 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3295e-05 2024-04-19T01:04:28.373 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.373 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.373 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:28.374 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.915152+0000", 2024-04-19T01:04:28.374 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014489837 2024-04-19T01:04:28.374 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.374 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.374 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:28.374 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.915168+0000", 2024-04-19T01:04:28.374 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5233e-05 2024-04-19T01:04:28.374 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.374 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.374 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:28.374 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.916606+0000", 2024-04-19T01:04:28.374 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001437993 2024-04-19T01:04:28.374 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.374 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.375 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:28.375 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.916679+0000", 2024-04-19T01:04:28.375 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3565999999999993e-05 2024-04-19T01:04:28.375 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:28.375 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:28.375 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:28.375 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.375 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.375 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 43/31 ECSubWrite(tid=38, reqid=osd.8.0:38, at_version=43'27, trim_to=0'0, roll_forward_to=42'25))", 2024-04-19T01:04:28.375 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.847035+0000", 2024-04-19T01:04:28.375 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 14.48811392, 2024-04-19T01:04:28.375 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010311450999999999, 2024-04-19T01:04:28.375 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:28.375 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:28.376 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:28.376 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:28.376 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.376 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:28.376 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.847035+0000", 2024-04-19T01:04:28.376 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:28.376 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.376 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.376 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:28.376 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.847035+0000", 2024-04-19T01:04:28.376 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:28.376 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.376 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.376 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:28.377 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.847036+0000", 2024-04-19T01:04:28.377 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8000000000000005e-07 2024-04-19T01:04:28.377 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.377 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.377 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:28.377 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.847113+0000", 2024-04-19T01:04:28.377 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6667000000000003e-05 2024-04-19T01:04:28.377 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.377 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.377 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:28.377 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.847140+0000", 2024-04-19T01:04:28.377 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7146000000000001e-05 2024-04-19T01:04:28.377 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.377 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.377 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:28.378 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.847162+0000", 2024-04-19T01:04:28.378 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2079e-05 2024-04-19T01:04:28.378 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.378 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.378 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:28.378 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.847314+0000", 2024-04-19T01:04:28.378 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000152175 2024-04-19T01:04:28.378 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.378 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.378 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:28.378 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.847337+0000", 2024-04-19T01:04:28.378 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3275999999999999e-05 2024-04-19T01:04:28.378 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.378 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.379 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:28.379 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.857249+0000", 2024-04-19T01:04:28.379 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0099113880000000001 2024-04-19T01:04:28.379 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.379 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.379 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:28.379 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.857347+0000", 2024-04-19T01:04:28.379 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7940000000000001e-05 2024-04-19T01:04:28.379 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:28.379 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:28.379 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:28.379 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.379 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.379 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s2 43/31 ECSubWrite(tid=70, reqid=client.4604.0:235, at_version=43'43, trim_to=0'0, roll_forward_to=41'37))", 2024-04-19T01:04:28.379 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.874738+0000", 2024-04-19T01:04:28.380 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 14.460410862, 2024-04-19T01:04:28.380 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024645897999999999, 2024-04-19T01:04:28.380 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:28.380 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:28.380 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:28.380 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:28.380 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.380 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:28.380 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.874738+0000", 2024-04-19T01:04:28.380 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:28.380 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.380 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.380 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:28.381 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.874738+0000", 2024-04-19T01:04:28.381 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:28.381 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.381 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.381 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:28.381 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.874739+0000", 2024-04-19T01:04:28.381 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.203e-06 2024-04-19T01:04:28.381 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.381 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.381 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:28.381 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.875466+0000", 2024-04-19T01:04:28.381 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00072702499999999996 2024-04-19T01:04:28.381 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.381 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.382 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:28.382 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.875489+0000", 2024-04-19T01:04:28.382 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2932e-05 2024-04-19T01:04:28.382 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.382 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.382 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:28.382 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.875509+0000", 2024-04-19T01:04:28.382 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9996999999999998e-05 2024-04-19T01:04:28.382 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.382 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.382 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:28.382 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.875656+0000", 2024-04-19T01:04:28.383 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014699099999999999 2024-04-19T01:04:28.383 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.383 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.383 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:28.383 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.875683+0000", 2024-04-19T01:04:28.383 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6435999999999999e-05 2024-04-19T01:04:28.383 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.384 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.384 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:28.384 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.899339+0000", 2024-04-19T01:04:28.384 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023656074999999999 2024-04-19T01:04:28.384 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.384 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.384 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:28.385 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.899384+0000", 2024-04-19T01:04:28.385 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5238999999999998e-05 2024-04-19T01:04:28.385 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:28.385 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:28.385 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:28.385 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.385 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.385 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s2 43/31 ECSubWrite(tid=72, reqid=client.4604.0:236, at_version=43'44, trim_to=0'0, roll_forward_to=41'37))", 2024-04-19T01:04:28.385 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.890996+0000", 2024-04-19T01:04:28.385 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 14.44415272, 2024-04-19T01:04:28.385 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0093733779999999999, 2024-04-19T01:04:28.385 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:28.385 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:28.385 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:28.386 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:28.386 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.386 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:28.386 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.890996+0000", 2024-04-19T01:04:28.386 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:28.386 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.386 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.386 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:28.386 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.890996+0000", 2024-04-19T01:04:28.386 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:28.387 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.387 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.387 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:28.387 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.890998+0000", 2024-04-19T01:04:28.387 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8530000000000001e-06 2024-04-19T01:04:28.387 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.387 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.387 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:28.387 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.891336+0000", 2024-04-19T01:04:28.387 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00033820200000000001 2024-04-19T01:04:28.387 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.387 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.388 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:28.388 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.891374+0000", 2024-04-19T01:04:28.388 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7795e-05 2024-04-19T01:04:28.388 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.388 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.388 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:28.388 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.891397+0000", 2024-04-19T01:04:28.388 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2742999999999999e-05 2024-04-19T01:04:28.388 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.389 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.389 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:28.389 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.893872+0000", 2024-04-19T01:04:28.389 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002474979 2024-04-19T01:04:28.389 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.389 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.389 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:28.389 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.893891+0000", 2024-04-19T01:04:28.389 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9465000000000001e-05 2024-04-19T01:04:28.389 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.389 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.389 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:28.389 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.900302+0000", 2024-04-19T01:04:28.389 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.006410692 2024-04-19T01:04:28.390 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.390 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.390 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:28.390 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.900370+0000", 2024-04-19T01:04:28.390 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7649000000000002e-05 2024-04-19T01:04:28.390 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:28.390 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:28.390 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:28.390 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.390 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.390 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 43/31 ECSubWrite(tid=43, reqid=client.4604.0:251, at_version=43'29, trim_to=0'0, roll_forward_to=43'27))", 2024-04-19T01:04:28.390 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.985922+0000", 2024-04-19T01:04:28.390 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 14.349227296, 2024-04-19T01:04:28.390 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.021650376999999998, 2024-04-19T01:04:28.391 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:28.391 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:28.391 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:28.391 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:28.391 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.391 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:28.391 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.985922+0000", 2024-04-19T01:04:28.391 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:28.391 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.391 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.391 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:28.391 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.985922+0000", 2024-04-19T01:04:28.391 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:28.391 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.392 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.392 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:28.392 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.985924+0000", 2024-04-19T01:04:28.392 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1169999999999998e-06 2024-04-19T01:04:28.392 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.392 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.392 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:28.392 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986891+0000", 2024-04-19T01:04:28.392 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00096720599999999997 2024-04-19T01:04:28.392 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.392 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.392 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:28.392 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986935+0000", 2024-04-19T01:04:28.392 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3776000000000002e-05 2024-04-19T01:04:28.392 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.393 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.393 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:28.393 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986960+0000", 2024-04-19T01:04:28.393 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5418999999999998e-05 2024-04-19T01:04:28.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.393 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.393 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:28.393 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987142+0000", 2024-04-19T01:04:28.393 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018212 2024-04-19T01:04:28.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.393 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.393 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:28.393 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987174+0000", 2024-04-19T01:04:28.393 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1782000000000002e-05 2024-04-19T01:04:28.394 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.394 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.394 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:28.394 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.007479+0000", 2024-04-19T01:04:28.394 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020304543000000001 2024-04-19T01:04:28.394 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.394 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.394 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:28.394 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.007572+0000", 2024-04-19T01:04:28.394 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3413999999999997e-05 2024-04-19T01:04:28.394 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:28.394 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:28.394 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:28.394 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.394 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.395 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s2 43/31 ECSubWrite(tid=76, reqid=client.4604.0:267, at_version=43'46, trim_to=0'0, roll_forward_to=43'45))", 2024-04-19T01:04:28.395 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.026909+0000", 2024-04-19T01:04:28.395 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 14.308240165000001, 2024-04-19T01:04:28.395 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010819211, 2024-04-19T01:04:28.395 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:28.395 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:28.395 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:28.395 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:28.395 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.395 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:28.395 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.026909+0000", 2024-04-19T01:04:28.395 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:28.395 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.395 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.396 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:28.396 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.026909+0000", 2024-04-19T01:04:28.396 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:28.396 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.396 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.396 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:28.396 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.026911+0000", 2024-04-19T01:04:28.396 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3080000000000002e-06 2024-04-19T01:04:28.396 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.396 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.396 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:28.396 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.027220+0000", 2024-04-19T01:04:28.396 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00030838399999999998 2024-04-19T01:04:28.396 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.397 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.397 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:28.397 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.027247+0000", 2024-04-19T01:04:28.397 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7634e-05 2024-04-19T01:04:28.397 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.397 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.397 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:28.397 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.027268+0000", 2024-04-19T01:04:28.397 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0398999999999999e-05 2024-04-19T01:04:28.397 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.397 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.397 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:28.397 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.027373+0000", 2024-04-19T01:04:28.397 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000105709 2024-04-19T01:04:28.398 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.398 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.398 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:28.398 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.027384+0000", 2024-04-19T01:04:28.398 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1151e-05 2024-04-19T01:04:28.398 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.398 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.399 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:28.399 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.037668+0000", 2024-04-19T01:04:28.399 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010283494000000001 2024-04-19T01:04:28.399 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.399 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.399 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:28.399 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.037728+0000", 2024-04-19T01:04:28.399 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0132e-05 2024-04-19T01:04:28.399 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:28.399 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:28.399 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:28.399 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.399 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.400 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 44/31 ECSubWrite(tid=48, reqid=osd.8.0:48, at_version=44'31, trim_to=0'0, roll_forward_to=43'29))", 2024-04-19T01:04:28.400 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.875721+0000", 2024-04-19T01:04:28.400 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 13.459428358, 2024-04-19T01:04:28.400 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011047829, 2024-04-19T01:04:28.400 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:28.400 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:28.400 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:28.400 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:28.400 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.400 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:28.400 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.875721+0000", 2024-04-19T01:04:28.400 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:28.401 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.401 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.401 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:28.401 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.875721+0000", 2024-04-19T01:04:28.401 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:28.401 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.401 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.401 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:28.401 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.875721+0000", 2024-04-19T01:04:28.401 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4000000000000001e-07 2024-04-19T01:04:28.401 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.401 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.401 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:28.401 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.875777+0000", 2024-04-19T01:04:28.402 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5352000000000001e-05 2024-04-19T01:04:28.402 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.402 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.402 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:28.402 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.875812+0000", 2024-04-19T01:04:28.402 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5274999999999997e-05 2024-04-19T01:04:28.402 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.402 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.402 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:28.402 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.875837+0000", 2024-04-19T01:04:28.402 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5035999999999999e-05 2024-04-19T01:04:28.402 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.402 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.402 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:28.403 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876027+0000", 2024-04-19T01:04:28.403 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000190354 2024-04-19T01:04:28.403 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.403 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.403 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:28.403 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876055+0000", 2024-04-19T01:04:28.403 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7866000000000001e-05 2024-04-19T01:04:28.403 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.403 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.403 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:28.403 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.886413+0000", 2024-04-19T01:04:28.403 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010357857 2024-04-19T01:04:28.403 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.403 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.403 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:28.404 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.886769+0000", 2024-04-19T01:04:28.404 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035544899999999998 2024-04-19T01:04:28.404 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:28.404 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:28.404 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:28.404 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.404 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.404 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 44/31 ECSubWrite(tid=49, reqid=osd.8.0:49, at_version=44'33, trim_to=0'0, roll_forward_to=43'29))", 2024-04-19T01:04:28.404 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.876887+0000", 2024-04-19T01:04:28.404 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 13.458261773, 2024-04-19T01:04:28.404 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017055688999999999, 2024-04-19T01:04:28.404 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:28.404 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:28.404 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:28.405 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:28.405 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.405 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:28.405 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876887+0000", 2024-04-19T01:04:28.405 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:28.405 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.405 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.405 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:28.405 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876887+0000", 2024-04-19T01:04:28.405 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:28.405 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.405 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.405 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:28.405 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876889+0000", 2024-04-19T01:04:28.405 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.827e-06 2024-04-19T01:04:28.406 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.406 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.406 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:28.406 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876942+0000", 2024-04-19T01:04:28.406 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3248000000000002e-05 2024-04-19T01:04:28.406 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.406 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.406 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:28.406 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876984+0000", 2024-04-19T01:04:28.406 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1598999999999999e-05 2024-04-19T01:04:28.406 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.406 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.406 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:28.406 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.877007+0000", 2024-04-19T01:04:28.407 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3209000000000001e-05 2024-04-19T01:04:28.407 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.407 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.407 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:28.407 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.878441+0000", 2024-04-19T01:04:28.407 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0014341790000000001 2024-04-19T01:04:28.407 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.407 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.407 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:28.407 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.878457+0000", 2024-04-19T01:04:28.407 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5787e-05 2024-04-19T01:04:28.407 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.407 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.407 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:28.407 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.893844+0000", 2024-04-19T01:04:28.408 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015387283 2024-04-19T01:04:28.408 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.408 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.408 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:28.408 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.893943+0000", 2024-04-19T01:04:28.408 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8556999999999998e-05 2024-04-19T01:04:28.408 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:28.408 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:28.408 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:28.408 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.408 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.408 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s2 44/31 ECSubWrite(tid=82, reqid=client.4604.0:280, at_version=44'50, trim_to=0'0, roll_forward_to=43'48))", 2024-04-19T01:04:28.409 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.882222+0000", 2024-04-19T01:04:28.409 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 13.452927488, 2024-04-19T01:04:28.409 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.035827673999999997, 2024-04-19T01:04:28.409 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:28.409 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:28.409 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:28.409 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:28.409 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.410 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:28.410 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882222+0000", 2024-04-19T01:04:28.410 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:28.410 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.410 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.410 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:28.410 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882222+0000", 2024-04-19T01:04:28.410 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:28.410 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.410 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.410 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:28.410 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882222+0000", 2024-04-19T01:04:28.410 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7999999999999995e-07 2024-04-19T01:04:28.410 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.411 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.411 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:28.411 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882709+0000", 2024-04-19T01:04:28.411 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00048730799999999999 2024-04-19T01:04:28.411 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.420 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.420 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:28.421 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882742+0000", 2024-04-19T01:04:28.421 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2428000000000002e-05 2024-04-19T01:04:28.421 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.421 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.421 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:28.421 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882765+0000", 2024-04-19T01:04:28.421 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2693000000000001e-05 2024-04-19T01:04:28.421 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.421 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.421 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:28.421 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.883062+0000", 2024-04-19T01:04:28.421 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029777100000000001 2024-04-19T01:04:28.422 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.422 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.422 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:28.422 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.883088+0000", 2024-04-19T01:04:28.422 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6018999999999999e-05 2024-04-19T01:04:28.422 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.422 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.422 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:28.422 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.917995+0000", 2024-04-19T01:04:28.422 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.034906988999999999 2024-04-19T01:04:28.422 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.423 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.423 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:28.423 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.918049+0000", 2024-04-19T01:04:28.423 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3885999999999999e-05 2024-04-19T01:04:28.423 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:28.423 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:28.423 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:28.423 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.423 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.423 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s4 44/31 ECSubWrite(tid=186, reqid=osd.11.0:186, at_version=44'12, trim_to=0'0, roll_forward_to=43'10))", 2024-04-19T01:04:28.423 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.883784+0000", 2024-04-19T01:04:28.423 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 13.451365332, 2024-04-19T01:04:28.423 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.01028681, 2024-04-19T01:04:28.423 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:28.424 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:28.424 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:28.424 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:28.424 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.424 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:28.424 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.883784+0000", 2024-04-19T01:04:28.424 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:28.424 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.424 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.424 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:28.424 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.883784+0000", 2024-04-19T01:04:28.424 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:28.424 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.425 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.425 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:28.425 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.883784+0000", 2024-04-19T01:04:28.425 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1600000000000001e-07 2024-04-19T01:04:28.425 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.425 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.425 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:28.425 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.883860+0000", 2024-04-19T01:04:28.425 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5955e-05 2024-04-19T01:04:28.425 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.425 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.425 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:28.425 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.883889+0000", 2024-04-19T01:04:28.426 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9116e-05 2024-04-19T01:04:28.426 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.426 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.426 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:28.426 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.883911+0000", 2024-04-19T01:04:28.426 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1282000000000001e-05 2024-04-19T01:04:28.426 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.426 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.426 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:28.426 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.885494+0000", 2024-04-19T01:04:28.426 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001583575 2024-04-19T01:04:28.427 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.427 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.427 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:28.427 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.885519+0000", 2024-04-19T01:04:28.427 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5092999999999998e-05 2024-04-19T01:04:28.427 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.427 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.427 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:28.427 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.893986+0000", 2024-04-19T01:04:28.427 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0084664300000000005 2024-04-19T01:04:28.427 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.428 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.428 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:28.428 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.894071+0000", 2024-04-19T01:04:28.428 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4742999999999998e-05 2024-04-19T01:04:28.428 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:28.428 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:28.428 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:28.428 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.428 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.428 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.bs6 44/31 ECSubWrite(tid=42, reqid=osd.10.0:42, at_version=44'16, trim_to=0'0, roll_forward_to=43'14))", 2024-04-19T01:04:28.428 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.887817+0000", 2024-04-19T01:04:28.428 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 13.447331604, 2024-04-19T01:04:28.428 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011064671, 2024-04-19T01:04:28.429 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:28.429 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:28.429 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:28.429 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:28.429 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.429 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:28.429 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.887817+0000", 2024-04-19T01:04:28.429 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:28.429 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.429 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.429 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:28.429 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.887817+0000", 2024-04-19T01:04:28.429 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:28.429 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.430 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.430 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:28.430 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.887819+0000", 2024-04-19T01:04:28.430 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6029999999999999e-06 2024-04-19T01:04:28.430 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.430 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.430 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:28.431 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.887875+0000", 2024-04-19T01:04:28.431 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5828999999999997e-05 2024-04-19T01:04:28.431 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.431 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.431 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:28.431 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.887911+0000", 2024-04-19T01:04:28.431 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6191e-05 2024-04-19T01:04:28.431 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.431 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.431 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:28.431 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.887938+0000", 2024-04-19T01:04:28.431 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6950000000000001e-05 2024-04-19T01:04:28.431 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.431 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.432 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:28.432 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.888169+0000", 2024-04-19T01:04:28.432 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000230609 2024-04-19T01:04:28.432 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.432 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.432 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:28.432 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.888195+0000", 2024-04-19T01:04:28.432 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6018999999999999e-05 2024-04-19T01:04:28.432 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.432 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.432 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:28.432 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.898790+0000", 2024-04-19T01:04:28.432 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010594927000000001 2024-04-19T01:04:28.433 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.433 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.433 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:28.433 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.898882+0000", 2024-04-19T01:04:28.433 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2542999999999998e-05 2024-04-19T01:04:28.433 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:28.433 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:28.433 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:28.433 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.433 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.433 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 44/31 ECSubWrite(tid=50, reqid=osd.8.0:49, at_version=0'0, trim_to=0'0, roll_forward_to=44'33))", 2024-04-19T01:04:28.433 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.901832+0000", 2024-04-19T01:04:28.433 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 13.433316985999999, 2024-04-19T01:04:28.433 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011627197000000001, 2024-04-19T01:04:28.434 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:28.434 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:28.434 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:28.434 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:28.434 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.434 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:28.434 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.901832+0000", 2024-04-19T01:04:28.434 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:28.434 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.434 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.434 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:28.434 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.901832+0000", 2024-04-19T01:04:28.435 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:28.435 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.435 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.435 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:28.435 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.901834+0000", 2024-04-19T01:04:28.435 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.08e-06 2024-04-19T01:04:28.435 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.435 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.435 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:28.435 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.901863+0000", 2024-04-19T01:04:28.435 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8445999999999998e-05 2024-04-19T01:04:28.435 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.435 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.435 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:28.436 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.901895+0000", 2024-04-19T01:04:28.436 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2308999999999998e-05 2024-04-19T01:04:28.436 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.436 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.436 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:28.436 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.901917+0000", 2024-04-19T01:04:28.436 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2532999999999999e-05 2024-04-19T01:04:28.436 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.436 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.436 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:28.436 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.902071+0000", 2024-04-19T01:04:28.436 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015368099999999999 2024-04-19T01:04:28.436 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.436 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.436 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:28.437 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.902161+0000", 2024-04-19T01:04:28.437 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9703999999999994e-05 2024-04-19T01:04:28.437 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.437 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.437 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:28.437 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.913373+0000", 2024-04-19T01:04:28.437 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011212353 2024-04-19T01:04:28.437 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.437 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.437 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:28.437 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.913459+0000", 2024-04-19T01:04:28.437 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6091e-05 2024-04-19T01:04:28.437 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:28.438 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:28.438 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:28.438 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.438 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.438 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s2 44/31 ECSubWrite(tid=84, reqid=client.4604.0:281, at_version=44'51, trim_to=0'0, roll_forward_to=43'48))", 2024-04-19T01:04:28.438 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.907234+0000", 2024-04-19T01:04:28.438 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 13.427915553, 2024-04-19T01:04:28.438 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011122504, 2024-04-19T01:04:28.438 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:28.438 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:28.438 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:28.439 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:28.439 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.439 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:28.439 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.907234+0000", 2024-04-19T01:04:28.439 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:28.439 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.439 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.439 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:28.439 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.907234+0000", 2024-04-19T01:04:28.439 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:28.440 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.440 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.440 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:28.440 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.907235+0000", 2024-04-19T01:04:28.440 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.824e-06 2024-04-19T01:04:28.440 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.440 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.440 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:28.441 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.907488+0000", 2024-04-19T01:04:28.441 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00025270499999999999 2024-04-19T01:04:28.441 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.441 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.441 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:28.441 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.907520+0000", 2024-04-19T01:04:28.441 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2308999999999998e-05 2024-04-19T01:04:28.441 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.441 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.441 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:28.441 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.907544+0000", 2024-04-19T01:04:28.441 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3519e-05 2024-04-19T01:04:28.442 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.442 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.442 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:28.442 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.908322+0000", 2024-04-19T01:04:28.442 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00077773499999999999 2024-04-19T01:04:28.442 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.442 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.442 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:28.442 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.908349+0000", 2024-04-19T01:04:28.442 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6916000000000001e-05 2024-04-19T01:04:28.442 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.442 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.442 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:28.442 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.918305+0000", 2024-04-19T01:04:28.443 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0099564100000000006 2024-04-19T01:04:28.443 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.443 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.443 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:28.443 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.918356+0000", 2024-04-19T01:04:28.443 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1085999999999998e-05 2024-04-19T01:04:28.443 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:28.443 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:28.443 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:28.443 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.443 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.443 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s4 49/31 ECSubWrite(tid=202, reqid=osd.11.0:201, at_version=0'0, trim_to=0'0, roll_forward_to=49'24))", 2024-04-19T01:04:28.443 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:19.537002+0000", 2024-04-19T01:04:28.443 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 8.7981467920000007, 2024-04-19T01:04:28.444 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023787906000000001, 2024-04-19T01:04:28.444 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:28.444 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:28.444 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:28.444 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:28.444 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.444 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:28.444 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.537002+0000", 2024-04-19T01:04:28.444 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:28.444 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.444 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.444 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:28.444 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.537002+0000", 2024-04-19T01:04:28.444 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:28.445 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.445 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.445 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:28.445 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.537004+0000", 2024-04-19T01:04:28.445 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.553e-06 2024-04-19T01:04:28.445 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.445 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.445 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:28.445 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.537031+0000", 2024-04-19T01:04:28.445 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7039999999999999e-05 2024-04-19T01:04:28.445 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.445 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.445 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:28.445 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.537063+0000", 2024-04-19T01:04:28.445 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1705000000000002e-05 2024-04-19T01:04:28.446 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.446 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.446 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:28.446 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.537084+0000", 2024-04-19T01:04:28.446 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1852999999999999e-05 2024-04-19T01:04:28.446 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.446 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.446 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:28.446 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.537559+0000", 2024-04-19T01:04:28.446 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00047440999999999998 2024-04-19T01:04:28.446 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.447 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.447 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:28.447 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.537585+0000", 2024-04-19T01:04:28.447 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6319e-05 2024-04-19T01:04:28.447 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.447 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.447 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:28.447 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.560733+0000", 2024-04-19T01:04:28.447 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023147581 2024-04-19T01:04:28.447 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.447 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.447 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:28.447 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.560790+0000", 2024-04-19T01:04:28.447 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7445000000000003e-05 2024-04-19T01:04:28.448 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:28.448 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:28.448 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:28.448 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.448 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.448 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s4 49/31 ECSubWrite(tid=203, reqid=client.4604.0:360, at_version=49'25, trim_to=0'0, roll_forward_to=49'24))", 2024-04-19T01:04:28.448 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:19.539163+0000", 2024-04-19T01:04:28.448 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 8.7959857279999998, 2024-04-19T01:04:28.448 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022464522000000001, 2024-04-19T01:04:28.448 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:28.448 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:28.448 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:28.448 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:28.448 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.448 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:28.449 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.539163+0000", 2024-04-19T01:04:28.449 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:28.449 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.449 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.449 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:28.449 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.539163+0000", 2024-04-19T01:04:28.449 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:28.449 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.449 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.449 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:28.449 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.539165+0000", 2024-04-19T01:04:28.449 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.95e-06 2024-04-19T01:04:28.449 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.449 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.449 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:28.450 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.539991+0000", 2024-04-19T01:04:28.450 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00082553499999999996 2024-04-19T01:04:28.450 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.450 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.450 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:28.450 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.540003+0000", 2024-04-19T01:04:28.450 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1838000000000001e-05 2024-04-19T01:04:28.450 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.450 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.450 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:28.450 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.540011+0000", 2024-04-19T01:04:28.450 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3829999999999992e-06 2024-04-19T01:04:28.450 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.451 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.451 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:28.451 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.540704+0000", 2024-04-19T01:04:28.451 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00069282000000000002 2024-04-19T01:04:28.451 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.451 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.451 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:28.451 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.540730+0000", 2024-04-19T01:04:28.452 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5649000000000001e-05 2024-04-19T01:04:28.452 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.452 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.452 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:28.452 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.561559+0000", 2024-04-19T01:04:28.452 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020829555 2024-04-19T01:04:28.452 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.452 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.452 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:28.452 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.561628+0000", 2024-04-19T01:04:28.452 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8792000000000001e-05 2024-04-19T01:04:28.452 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:28.452 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:28.452 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:28.452 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.453 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.453 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s4 49/31 ECSubWrite(tid=204, reqid=client.4604.0:361, at_version=49'26, trim_to=0'0, roll_forward_to=49'24))", 2024-04-19T01:04:28.453 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:19.561460+0000", 2024-04-19T01:04:28.453 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 8.7736889520000005, 2024-04-19T01:04:28.453 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010050376, 2024-04-19T01:04:28.453 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:28.453 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:28.453 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:28.453 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:28.453 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.453 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:28.453 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.561460+0000", 2024-04-19T01:04:28.453 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:28.453 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.454 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.454 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:28.454 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.561460+0000", 2024-04-19T01:04:28.454 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:28.454 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.454 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.454 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:28.454 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.561462+0000", 2024-04-19T01:04:28.454 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.567e-06 2024-04-19T01:04:28.454 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.454 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.454 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:28.454 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.561717+0000", 2024-04-19T01:04:28.454 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00025506599999999997 2024-04-19T01:04:28.455 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.455 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.455 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:28.455 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.561741+0000", 2024-04-19T01:04:28.455 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3788999999999999e-05 2024-04-19T01:04:28.455 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.455 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.455 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:28.455 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.561755+0000", 2024-04-19T01:04:28.455 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4059999999999999e-05 2024-04-19T01:04:28.455 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.455 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.455 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:28.455 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.561912+0000", 2024-04-19T01:04:28.456 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000157429 2024-04-19T01:04:28.456 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.456 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.456 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:28.456 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.561941+0000", 2024-04-19T01:04:28.456 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8756e-05 2024-04-19T01:04:28.456 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.456 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.456 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:28.456 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.571478+0000", 2024-04-19T01:04:28.456 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0095372459999999992 2024-04-19T01:04:28.456 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:28.456 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:28.456 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:28.456 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.571511+0000", 2024-04-19T01:04:28.457 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2462999999999997e-05 2024-04-19T01:04:28.457 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:28.457 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:28.457 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:28.457 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:28.457 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:28.457 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:04:28.459 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-19T01:04:28.459 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:04:28.458+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:04:28.514 INFO:tasks.rados.rados.0.smithi155.stdout:166: done (3 left) 2024-04-19T01:04:28.514 INFO:tasks.rados.rados.0.smithi155.stdout:167: rmattr oid 14 current snap is 13 2024-04-19T01:04:28.514 INFO:tasks.rados.rados.0.smithi155.stdout:167: done (3 left) 2024-04-19T01:04:28.514 INFO:tasks.rados.rados.0.smithi155.stdout:168: setattr oid 39 current snap is 13 2024-04-19T01:04:28.515 INFO:tasks.rados.rados.0.smithi155.stdout:161: finishing write tid 1 to smithi15528762-33 2024-04-19T01:04:28.515 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 31 v 49 (ObjNum 177 snap 0 seq_num 1) dirty exists 2024-04-19T01:04:28.515 INFO:tasks.rados.rados.0.smithi155.stdout:161: finishing write tid 2 to smithi15528762-33 2024-04-19T01:04:28.515 INFO:tasks.rados.rados.0.smithi155.stdout:161: finishing write tid 3 to smithi15528762-33 2024-04-19T01:04:28.515 INFO:tasks.rados.rados.0.smithi155.stdout:165: expect (ObjNum 76 snap 8 seq_num 76) 2024-04-19T01:04:28.528 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-19T01:04:28.529 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-04-19T01:04:28.529 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-19T01:04:28.529 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-04-19T01:04:28.529 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2024-04-19T01:04:28.560 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:04:28.560+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:04:28.577 INFO:tasks.rados.rados.0.smithi155.stdout:161: finishing write tid 4 to smithi15528762-33 2024-04-19T01:04:28.577 INFO:tasks.rados.rados.0.smithi155.stdout:161: finishing write tid 5 to smithi15528762-33 2024-04-19T01:04:28.577 INFO:tasks.rados.rados.0.smithi155.stdout:161: oid 33 updating version 0 to 34 2024-04-19T01:04:28.577 INFO:tasks.rados.rados.0.smithi155.stdout:161: oid 33 updating version 34 to 35 2024-04-19T01:04:28.577 INFO:tasks.rados.rados.0.smithi155.stdout:161: oid 33 updating version 35 to 36 2024-04-19T01:04:28.577 INFO:tasks.rados.rados.0.smithi155.stdout:161: oid 33 updating version 36 to 37 2024-04-19T01:04:28.577 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 33 v 37 (ObjNum 82 snap 13 seq_num 82) dirty exists 2024-04-19T01:04:28.578 INFO:tasks.rados.rados.0.smithi155.stdout:161: left oid 33 (ObjNum 82 snap 13 seq_num 82) 2024-04-19T01:04:28.578 INFO:tasks.rados.rados.0.smithi155.stdout:161: done (3 left) 2024-04-19T01:04:28.578 INFO:tasks.rados.rados.0.smithi155.stdout:164: done (2 left) 2024-04-19T01:04:28.578 INFO:tasks.rados.rados.0.smithi155.stdout:165: done (1 left) 2024-04-19T01:04:28.578 INFO:tasks.rados.rados.0.smithi155.stdout:169: snap_remove snap 10 2024-04-19T01:04:28.595 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:04:28.595 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:04:28.595 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-04-19T01:04:28.595 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:04:28.609 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-19T01:04:28.661 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:04:28.658+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:04:28.709 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2024-04-19T01:04:28.733 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:04:28.733 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:04:28.734 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-04-19T01:04:28.734 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-04-19T01:04:28.734 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:04:28.748 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-19T01:04:28.762 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:04:28.760+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:04:28.862 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:04:28.863+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:04:28.874 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2024-04-19T01:04:28.875 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:04:28.876 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-04-19T01:04:28.876 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-04-19T01:04:28.876 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-04-19T01:04:28.876 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.876 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.0s3 38/31 ECSubWrite(tid=11, reqid=client.4604.0:169, at_version=38'8, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:04:28.876 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:08.807627+0000", 2024-04-19T01:04:28.876 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 20.066922319, 2024-04-19T01:04:28.876 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0036417569999999998, 2024-04-19T01:04:28.876 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:28.876 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:28.876 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:28.876 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:28.876 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.877 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:28.877 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.807627+0000", 2024-04-19T01:04:28.877 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:28.877 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.877 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.877 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:28.877 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.807627+0000", 2024-04-19T01:04:28.877 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:28.877 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.877 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.877 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:28.877 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.807628+0000", 2024-04-19T01:04:28.877 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5699999999999998e-07 2024-04-19T01:04:28.877 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.878 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.878 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:28.878 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.807669+0000", 2024-04-19T01:04:28.878 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0889000000000001e-05 2024-04-19T01:04:28.878 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.879 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.879 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:28.879 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.807702+0000", 2024-04-19T01:04:28.879 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3096000000000003e-05 2024-04-19T01:04:28.879 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.879 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.879 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:28.879 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.807725+0000", 2024-04-19T01:04:28.880 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3329e-05 2024-04-19T01:04:28.880 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.880 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.880 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:28.880 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.807890+0000", 2024-04-19T01:04:28.880 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016469600000000001 2024-04-19T01:04:28.880 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.880 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.880 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:28.880 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.807918+0000", 2024-04-19T01:04:28.880 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8693000000000001e-05 2024-04-19T01:04:28.880 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.880 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.881 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:28.881 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.811184+0000", 2024-04-19T01:04:28.881 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0032654060000000002 2024-04-19T01:04:28.881 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.881 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.881 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:28.881 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.811269+0000", 2024-04-19T01:04:28.881 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5290999999999994e-05 2024-04-19T01:04:28.881 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:28.881 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:28.881 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:28.881 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.881 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.882 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.2s5 43/31 ECSubWrite(tid=180, reqid=osd.11.0:180, at_version=43'10, trim_to=0'0, roll_forward_to=41'8))", 2024-04-19T01:04:28.882 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:13.844393+0000", 2024-04-19T01:04:28.882 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 15.030156318, 2024-04-19T01:04:28.882 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0044436120000000004, 2024-04-19T01:04:28.882 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:28.882 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:28.882 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:28.882 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:28.882 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.882 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:28.882 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.844393+0000", 2024-04-19T01:04:28.882 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:28.882 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.882 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.883 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:28.883 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.844393+0000", 2024-04-19T01:04:28.883 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:28.883 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.883 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.883 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:28.883 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.844394+0000", 2024-04-19T01:04:28.883 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9400000000000001e-07 2024-04-19T01:04:28.884 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.884 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.884 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:28.884 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.844445+0000", 2024-04-19T01:04:28.884 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.1523999999999997e-05 2024-04-19T01:04:28.884 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.884 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.885 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:28.885 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.844472+0000", 2024-04-19T01:04:28.885 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7161e-05 2024-04-19T01:04:28.885 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.886 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.886 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:28.886 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.844488+0000", 2024-04-19T01:04:28.886 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6127e-05 2024-04-19T01:04:28.886 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.886 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.886 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:28.886 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.844592+0000", 2024-04-19T01:04:28.886 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000104034 2024-04-19T01:04:28.887 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.887 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.887 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:28.887 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.844608+0000", 2024-04-19T01:04:28.887 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5931999999999999e-05 2024-04-19T01:04:28.887 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.887 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.887 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:28.887 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.848782+0000", 2024-04-19T01:04:28.887 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.004173398 2024-04-19T01:04:28.887 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.887 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.887 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:28.887 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.848837+0000", 2024-04-19T01:04:28.887 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.5142000000000002e-05 2024-04-19T01:04:28.888 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:28.888 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:28.888 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:28.888 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.888 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.888 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.0s3 44/31 ECSubWrite(tid=23, reqid=osd.1.0:23, at_version=44'20, trim_to=0'0, roll_forward_to=43'18))", 2024-04-19T01:04:28.888 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:14.891502+0000", 2024-04-19T01:04:28.888 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 13.983047634, 2024-04-19T01:04:28.888 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0073445860000000002, 2024-04-19T01:04:28.888 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:28.888 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:28.888 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:28.888 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:28.888 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.889 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:28.889 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.891502+0000", 2024-04-19T01:04:28.889 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:28.889 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.889 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.889 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:28.889 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.891502+0000", 2024-04-19T01:04:28.889 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:28.889 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.889 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.889 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:28.889 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.891502+0000", 2024-04-19T01:04:28.889 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6100000000000002e-07 2024-04-19T01:04:28.889 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.889 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.890 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:28.890 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.891547+0000", 2024-04-19T01:04:28.890 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5077999999999998e-05 2024-04-19T01:04:28.890 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.890 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.890 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:28.890 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.891565+0000", 2024-04-19T01:04:28.890 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7574000000000001e-05 2024-04-19T01:04:28.890 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.890 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.890 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:28.890 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.891578+0000", 2024-04-19T01:04:28.890 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2700999999999999e-05 2024-04-19T01:04:28.890 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.890 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.891 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:28.891 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.891729+0000", 2024-04-19T01:04:28.891 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015078300000000001 2024-04-19T01:04:28.891 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.891 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.891 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:28.891 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.891759+0000", 2024-04-19T01:04:28.891 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0722999999999997e-05 2024-04-19T01:04:28.892 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.892 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.892 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:28.892 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.898759+0000", 2024-04-19T01:04:28.892 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0069993640000000001 2024-04-19T01:04:28.893 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.893 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.893 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:28.893 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.898847+0000", 2024-04-19T01:04:28.894 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.8002000000000002e-05 2024-04-19T01:04:28.894 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:28.894 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:28.894 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:28.894 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.894 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.894 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.4s3 47/31 ECSubWrite(tid=47, reqid=client.4604.0:299, at_version=47'12, trim_to=0'0, roll_forward_to=35'10))", 2024-04-19T01:04:28.894 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:17.523701+0000", 2024-04-19T01:04:28.894 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 11.350848374, 2024-04-19T01:04:28.894 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0044232689999999996, 2024-04-19T01:04:28.894 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:28.894 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:28.894 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:28.895 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:28.895 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.895 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:28.895 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.523701+0000", 2024-04-19T01:04:28.895 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:28.899 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.899 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.899 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:28.899 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.523701+0000", 2024-04-19T01:04:28.899 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:28.899 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.900 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.900 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:28.900 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.523702+0000", 2024-04-19T01:04:28.900 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.08e-06 2024-04-19T01:04:28.900 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.900 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.900 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:28.900 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.523758+0000", 2024-04-19T01:04:28.900 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.5488999999999997e-05 2024-04-19T01:04:28.900 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.900 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.900 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:28.900 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.523794+0000", 2024-04-19T01:04:28.900 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6502999999999999e-05 2024-04-19T01:04:28.901 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.901 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.901 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:28.901 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.523840+0000", 2024-04-19T01:04:28.901 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5899000000000002e-05 2024-04-19T01:04:28.901 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.901 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.901 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:28.902 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.524232+0000", 2024-04-19T01:04:28.902 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00039216499999999998 2024-04-19T01:04:28.902 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.902 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.902 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:28.902 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.524270+0000", 2024-04-19T01:04:28.902 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7725999999999998e-05 2024-04-19T01:04:28.902 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.902 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.902 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:28.902 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.528011+0000", 2024-04-19T01:04:28.902 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0037405590000000001 2024-04-19T01:04:28.902 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.902 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.903 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:28.903 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.528125+0000", 2024-04-19T01:04:28.903 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000113848 2024-04-19T01:04:28.903 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:28.903 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:28.903 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:28.903 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.903 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.903 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.4s3 47/31 ECSubWrite(tid=48, reqid=client.4604.0:300, at_version=47'13, trim_to=0'0, roll_forward_to=35'10))", 2024-04-19T01:04:28.903 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:17.526628+0000", 2024-04-19T01:04:28.903 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 11.347921877999999, 2024-04-19T01:04:28.903 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0032629030000000002, 2024-04-19T01:04:28.903 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:28.903 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:28.904 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:28.904 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:28.904 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.904 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:28.904 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.526628+0000", 2024-04-19T01:04:28.904 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:28.904 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.904 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.904 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:28.904 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.526628+0000", 2024-04-19T01:04:28.904 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:28.905 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.905 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.905 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:28.905 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.526629+0000", 2024-04-19T01:04:28.905 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5829999999999999e-06 2024-04-19T01:04:28.905 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.905 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.905 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:28.905 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.526694+0000", 2024-04-19T01:04:28.905 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4323000000000001e-05 2024-04-19T01:04:28.905 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.905 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.905 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:28.905 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.526773+0000", 2024-04-19T01:04:28.906 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.9067999999999996e-05 2024-04-19T01:04:28.906 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.906 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.906 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:28.906 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.526845+0000", 2024-04-19T01:04:28.906 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2032000000000004e-05 2024-04-19T01:04:28.906 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.907 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.907 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:28.907 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.527018+0000", 2024-04-19T01:04:28.907 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017353299999999999 2024-04-19T01:04:28.907 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.907 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.907 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:28.907 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.527045+0000", 2024-04-19T01:04:28.907 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6650000000000001e-05 2024-04-19T01:04:28.907 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.907 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.908 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:28.908 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.529816+0000", 2024-04-19T01:04:28.908 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0027709380000000001 2024-04-19T01:04:28.908 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.908 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.908 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:28.908 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.529891+0000", 2024-04-19T01:04:28.908 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4776000000000004e-05 2024-04-19T01:04:28.908 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:28.908 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:28.908 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:28.908 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.908 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.908 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.0s3 47/31 ECSubWrite(tid=30, reqid=client.4604.0:307, at_version=47'28, trim_to=0'0, roll_forward_to=45'26))", 2024-04-19T01:04:28.908 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:17.550051+0000", 2024-04-19T01:04:28.909 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 11.32449853, 2024-04-19T01:04:28.909 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0077366320000000002, 2024-04-19T01:04:28.909 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:28.909 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:28.909 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:28.909 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:28.909 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.909 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:28.909 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.550051+0000", 2024-04-19T01:04:28.909 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:28.909 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.909 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.909 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:28.909 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.550051+0000", 2024-04-19T01:04:28.909 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:28.910 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.910 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.910 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:28.910 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.550052+0000", 2024-04-19T01:04:28.910 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7700000000000004e-07 2024-04-19T01:04:28.910 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.910 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.910 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:28.910 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.550470+0000", 2024-04-19T01:04:28.910 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00041806800000000001 2024-04-19T01:04:28.910 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.910 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.910 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:28.910 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.550509+0000", 2024-04-19T01:04:28.911 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8773000000000003e-05 2024-04-19T01:04:28.911 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.911 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.911 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:28.911 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.550531+0000", 2024-04-19T01:04:28.912 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2841999999999999e-05 2024-04-19T01:04:28.912 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.912 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.912 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:28.912 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.550645+0000", 2024-04-19T01:04:28.912 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011352499999999999 2024-04-19T01:04:28.912 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.912 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.912 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:28.912 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.550657+0000", 2024-04-19T01:04:28.912 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2252000000000001e-05 2024-04-19T01:04:28.912 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.913 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.913 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:28.913 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.557738+0000", 2024-04-19T01:04:28.913 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0070807329999999996 2024-04-19T01:04:28.913 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.913 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.913 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:28.913 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.557788+0000", 2024-04-19T01:04:28.913 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9762000000000002e-05 2024-04-19T01:04:28.913 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:28.913 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:28.913 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:28.913 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.914 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.914 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.0s3 47/31 ECSubWrite(tid=32, reqid=client.4604.0:308, at_version=47'29, trim_to=0'0, roll_forward_to=47'28))", 2024-04-19T01:04:28.914 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:17.565539+0000", 2024-04-19T01:04:28.914 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 11.309010204, 2024-04-19T01:04:28.914 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.018109618000000001, 2024-04-19T01:04:28.914 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:28.914 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:28.914 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:28.914 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:28.914 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.914 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:28.914 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.565539+0000", 2024-04-19T01:04:28.914 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:28.915 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.915 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.915 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:28.915 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.565539+0000", 2024-04-19T01:04:28.915 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:28.915 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.915 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.915 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:28.915 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.565541+0000", 2024-04-19T01:04:28.915 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.04e-06 2024-04-19T01:04:28.915 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.915 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.915 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:28.915 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.566554+0000", 2024-04-19T01:04:28.916 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0010129029999999999 2024-04-19T01:04:28.916 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.916 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.916 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:28.916 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.566591+0000", 2024-04-19T01:04:28.916 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7061999999999999e-05 2024-04-19T01:04:28.916 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.916 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.916 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:28.916 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.566613+0000", 2024-04-19T01:04:28.916 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.145e-05 2024-04-19T01:04:28.916 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.916 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.916 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:28.916 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.566752+0000", 2024-04-19T01:04:28.917 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000138868 2024-04-19T01:04:28.917 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.917 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.917 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:28.917 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.566783+0000", 2024-04-19T01:04:28.917 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0772000000000003e-05 2024-04-19T01:04:28.917 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.917 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.918 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:28.918 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.583581+0000", 2024-04-19T01:04:28.918 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.016798124000000001 2024-04-19T01:04:28.918 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.918 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.918 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:28.918 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.583649+0000", 2024-04-19T01:04:28.918 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.8398999999999993e-05 2024-04-19T01:04:28.918 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:28.918 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:28.918 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:28.918 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.918 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.919 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.0s3 47/31 ECSubWrite(tid=33, reqid=client.4604.0:309, at_version=47'30, trim_to=0'0, roll_forward_to=47'28))", 2024-04-19T01:04:28.919 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:17.573407+0000", 2024-04-19T01:04:28.919 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 11.301142597, 2024-04-19T01:04:28.919 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.010864425, 2024-04-19T01:04:28.919 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:28.919 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:28.919 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:28.919 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:28.919 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.919 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:28.919 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.573407+0000", 2024-04-19T01:04:28.919 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:28.919 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.919 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.919 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:28.920 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.573407+0000", 2024-04-19T01:04:28.920 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:28.920 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.920 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.920 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:28.920 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.573408+0000", 2024-04-19T01:04:28.920 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4300000000000001e-06 2024-04-19T01:04:28.920 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.920 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.920 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:28.920 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.573474+0000", 2024-04-19T01:04:28.920 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5711999999999997e-05 2024-04-19T01:04:28.920 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.921 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.921 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:28.921 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.573506+0000", 2024-04-19T01:04:28.921 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1445999999999997e-05 2024-04-19T01:04:28.921 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.921 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.921 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:28.921 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.573526+0000", 2024-04-19T01:04:28.921 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9919999999999999e-05 2024-04-19T01:04:28.921 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.921 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.921 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:28.921 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.582034+0000", 2024-04-19T01:04:28.922 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.008508837 2024-04-19T01:04:28.922 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.922 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.922 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:28.922 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.582053+0000", 2024-04-19T01:04:28.922 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8816000000000001e-05 2024-04-19T01:04:28.922 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.922 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.922 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:28.922 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.584163+0000", 2024-04-19T01:04:28.923 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0021099439999999999 2024-04-19T01:04:28.923 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.923 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.923 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:28.923 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.584271+0000", 2024-04-19T01:04:28.923 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010832000000000001 2024-04-19T01:04:28.923 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:28.923 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:28.923 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:28.923 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.923 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.923 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.0s3 47/31 ECSubWrite(tid=34, reqid=client.4604.0:310, at_version=47'31, trim_to=0'0, roll_forward_to=47'28))", 2024-04-19T01:04:28.923 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:17.574889+0000", 2024-04-19T01:04:28.923 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 11.299661084, 2024-04-19T01:04:28.924 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0098864950000000004, 2024-04-19T01:04:28.924 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:28.924 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:28.924 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:28.924 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:28.924 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.924 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:28.924 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.574889+0000", 2024-04-19T01:04:28.924 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:28.924 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.924 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.925 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:28.925 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.574889+0000", 2024-04-19T01:04:28.925 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:28.925 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.925 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.925 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:28.925 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.574890+0000", 2024-04-19T01:04:28.925 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.457e-06 2024-04-19T01:04:28.925 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.925 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.925 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:28.925 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.574924+0000", 2024-04-19T01:04:28.925 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4181999999999999e-05 2024-04-19T01:04:28.926 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.926 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.926 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:28.926 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.574953+0000", 2024-04-19T01:04:28.926 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9226000000000001e-05 2024-04-19T01:04:28.926 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.926 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.926 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:28.926 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.574974+0000", 2024-04-19T01:04:28.926 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0273000000000001e-05 2024-04-19T01:04:28.926 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.926 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.926 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:28.926 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.583276+0000", 2024-04-19T01:04:28.926 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.008302495 2024-04-19T01:04:28.927 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.927 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.927 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:28.927 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.583290+0000", 2024-04-19T01:04:28.927 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3696999999999999e-05 2024-04-19T01:04:28.927 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.927 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.927 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:28.927 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.584690+0000", 2024-04-19T01:04:28.928 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001400047 2024-04-19T01:04:28.928 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.928 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.928 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:28.928 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.584775+0000", 2024-04-19T01:04:28.928 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5117999999999994e-05 2024-04-19T01:04:28.928 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:28.928 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:28.928 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:28.928 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.928 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.928 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.0s3 48/31 ECSubWrite(tid=37, reqid=osd.1.0:37, at_version=48'33, trim_to=0'0, roll_forward_to=47'31))", 2024-04-19T01:04:28.928 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.511777+0000", 2024-04-19T01:04:28.929 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.362773021000001, 2024-04-19T01:04:28.929 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.003385781, 2024-04-19T01:04:28.929 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:28.929 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:28.929 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:28.929 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:28.929 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.929 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:28.929 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511777+0000", 2024-04-19T01:04:28.929 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:28.929 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.929 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.929 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:28.929 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511777+0000", 2024-04-19T01:04:28.929 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:28.930 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.930 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.930 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:28.930 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511777+0000", 2024-04-19T01:04:28.930 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1699999999999998e-07 2024-04-19T01:04:28.930 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.930 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.930 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:28.930 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511834+0000", 2024-04-19T01:04:28.930 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6409000000000002e-05 2024-04-19T01:04:28.930 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.930 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.930 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:28.930 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511861+0000", 2024-04-19T01:04:28.931 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6959999999999999e-05 2024-04-19T01:04:28.931 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.931 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.931 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:28.931 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511890+0000", 2024-04-19T01:04:28.931 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9399000000000001e-05 2024-04-19T01:04:28.931 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.931 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.931 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:28.931 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.512098+0000", 2024-04-19T01:04:28.931 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000208109 2024-04-19T01:04:28.931 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.931 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.931 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:28.932 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.512116+0000", 2024-04-19T01:04:28.932 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.755e-05 2024-04-19T01:04:28.932 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.932 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.932 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:28.932 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.515062+0000", 2024-04-19T01:04:28.932 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0029467320000000001 2024-04-19T01:04:28.932 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.932 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.933 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:28.933 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.515162+0000", 2024-04-19T01:04:28.933 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000100005 2024-04-19T01:04:28.933 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:28.933 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:28.933 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:28.933 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.933 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.933 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.0s3 48/31 ECSubWrite(tid=38, reqid=osd.1.0:37, at_version=0'0, trim_to=0'0, roll_forward_to=48'33))", 2024-04-19T01:04:28.933 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.519877+0000", 2024-04-19T01:04:28.933 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.354673033999999, 2024-04-19T01:04:28.933 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0037958449999999999, 2024-04-19T01:04:28.933 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:28.933 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:28.934 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:28.934 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:28.934 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.934 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:28.934 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.519877+0000", 2024-04-19T01:04:28.934 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:28.935 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.935 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.935 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:28.935 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.519877+0000", 2024-04-19T01:04:28.935 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:28.936 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.936 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.936 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:28.936 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.519879+0000", 2024-04-19T01:04:28.936 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0439999999999998e-06 2024-04-19T01:04:28.936 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.936 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.936 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:28.936 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.519908+0000", 2024-04-19T01:04:28.936 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9258999999999999e-05 2024-04-19T01:04:28.936 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.936 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.936 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:28.936 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.519946+0000", 2024-04-19T01:04:28.937 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7725999999999998e-05 2024-04-19T01:04:28.937 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.937 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.937 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:28.937 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.519973+0000", 2024-04-19T01:04:28.937 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.722e-05 2024-04-19T01:04:28.937 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.937 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.937 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:28.937 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.520195+0000", 2024-04-19T01:04:28.937 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00022196800000000001 2024-04-19T01:04:28.937 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.937 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.937 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:28.938 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.520223+0000", 2024-04-19T01:04:28.938 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.775e-05 2024-04-19T01:04:28.938 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.938 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.938 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:28.938 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.523576+0000", 2024-04-19T01:04:28.939 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0033530769999999999 2024-04-19T01:04:28.939 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.939 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.939 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:28.939 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.523672+0000", 2024-04-19T01:04:28.939 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6800999999999997e-05 2024-04-19T01:04:28.939 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:28.939 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:28.939 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:28.939 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.939 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.939 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.4s3 48/31 ECSubWrite(tid=51, reqid=osd.10.0:51, at_version=48'16, trim_to=0'0, roll_forward_to=47'14))", 2024-04-19T01:04:28.939 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.520259+0000", 2024-04-19T01:04:28.939 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.354290370999999, 2024-04-19T01:04:28.939 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0040451769999999996, 2024-04-19T01:04:28.940 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:28.940 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:28.940 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:28.940 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:28.940 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.940 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:28.940 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.520259+0000", 2024-04-19T01:04:28.940 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:28.940 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.940 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.941 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:28.941 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.520259+0000", 2024-04-19T01:04:28.941 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:28.941 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.941 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.941 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:28.941 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.520260+0000", 2024-04-19T01:04:28.941 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7999999999999996e-07 2024-04-19T01:04:28.941 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.941 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.941 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:28.941 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.520338+0000", 2024-04-19T01:04:28.941 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7828000000000001e-05 2024-04-19T01:04:28.941 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.942 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.942 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:28.942 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.520360+0000", 2024-04-19T01:04:28.942 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1926999999999999e-05 2024-04-19T01:04:28.942 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.942 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.942 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:28.942 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.520374+0000", 2024-04-19T01:04:28.942 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4552999999999999e-05 2024-04-19T01:04:28.942 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.943 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.943 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:28.943 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.520519+0000", 2024-04-19T01:04:28.943 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014465699999999999 2024-04-19T01:04:28.943 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.943 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.943 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:28.943 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.520543+0000", 2024-04-19T01:04:28.943 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3975999999999999e-05 2024-04-19T01:04:28.944 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.944 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.944 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:28.944 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.524217+0000", 2024-04-19T01:04:28.944 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0036739239999999999 2024-04-19T01:04:28.944 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.944 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.944 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:28.944 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.524304+0000", 2024-04-19T01:04:28.944 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.7831999999999995e-05 2024-04-19T01:04:28.944 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:28.944 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:28.944 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:28.944 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.945 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.945 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as3 48/31 ECSubWrite(tid=71, reqid=client.4604.0:354, at_version=48'10, trim_to=0'0, roll_forward_to=34'8))", 2024-04-19T01:04:28.945 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.556344+0000", 2024-04-19T01:04:28.945 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.318205439, 2024-04-19T01:04:28.945 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.030027702, 2024-04-19T01:04:28.945 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:28.945 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:28.945 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:28.945 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:28.945 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.945 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:28.945 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.556344+0000", 2024-04-19T01:04:28.945 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:28.945 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.945 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.946 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:28.946 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.556344+0000", 2024-04-19T01:04:28.946 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:28.946 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.946 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.946 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:28.946 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.556345+0000", 2024-04-19T01:04:28.946 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1199999999999999e-07 2024-04-19T01:04:28.946 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.946 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.946 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:28.946 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.557285+0000", 2024-04-19T01:04:28.946 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00094096299999999996 2024-04-19T01:04:28.946 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.946 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.947 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:28.947 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.557335+0000", 2024-04-19T01:04:28.947 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9669000000000002e-05 2024-04-19T01:04:28.947 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.947 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.947 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:28.947 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.557358+0000", 2024-04-19T01:04:28.947 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3099999999999999e-05 2024-04-19T01:04:28.947 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.947 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.947 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:28.947 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.557490+0000", 2024-04-19T01:04:28.947 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013213400000000001 2024-04-19T01:04:28.947 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.948 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.948 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:28.948 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.557512+0000", 2024-04-19T01:04:28.948 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1603999999999998e-05 2024-04-19T01:04:28.948 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.948 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.948 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:28.948 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.586316+0000", 2024-04-19T01:04:28.949 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.028803714000000001 2024-04-19T01:04:28.949 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.949 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.949 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:28.949 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.586372+0000", 2024-04-19T01:04:28.949 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6205999999999997e-05 2024-04-19T01:04:28.949 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:28.949 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:28.949 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:28.949 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.949 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.950 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.4s3 49/31 ECSubWrite(tid=54, reqid=osd.10.0:54, at_version=49'18, trim_to=0'0, roll_forward_to=48'16))", 2024-04-19T01:04:28.950 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:19.527630+0000", 2024-04-19T01:04:28.950 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 9.3469195490000008, 2024-04-19T01:04:28.950 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0041525590000000001, 2024-04-19T01:04:28.950 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:28.950 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:28.950 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:28.950 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:28.950 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.950 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:28.950 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.527630+0000", 2024-04-19T01:04:28.950 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:28.950 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.950 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.951 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:28.951 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.527630+0000", 2024-04-19T01:04:28.951 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:28.951 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.951 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.951 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:28.951 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.527631+0000", 2024-04-19T01:04:28.951 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4900000000000005e-07 2024-04-19T01:04:28.951 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.951 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.951 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:28.951 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.527668+0000", 2024-04-19T01:04:28.951 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7277000000000002e-05 2024-04-19T01:04:28.951 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.952 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.952 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:28.952 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.527691+0000", 2024-04-19T01:04:28.952 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2527e-05 2024-04-19T01:04:28.952 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.952 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.952 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:28.952 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.527710+0000", 2024-04-19T01:04:28.952 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9068000000000001e-05 2024-04-19T01:04:28.952 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.952 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.952 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:28.952 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.527877+0000", 2024-04-19T01:04:28.952 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016754899999999999 2024-04-19T01:04:28.953 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.953 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.953 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:28.954 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.527898+0000", 2024-04-19T01:04:28.954 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0687e-05 2024-04-19T01:04:28.954 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.954 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.954 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:28.954 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.531706+0000", 2024-04-19T01:04:28.954 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0038080950000000001 2024-04-19T01:04:28.954 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.954 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.954 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:28.954 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.531783+0000", 2024-04-19T01:04:28.954 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6507000000000005e-05 2024-04-19T01:04:28.954 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:28.955 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:28.955 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:28.955 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.955 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.955 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as3 49/31 ECSubWrite(tid=73, reqid=osd.12.0:73, at_version=49'12, trim_to=0'0, roll_forward_to=48'10))", 2024-04-19T01:04:28.955 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:19.530679+0000", 2024-04-19T01:04:28.955 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 9.3438706150000002, 2024-04-19T01:04:28.955 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0088712419999999997, 2024-04-19T01:04:28.955 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:28.955 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:28.955 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:28.955 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:28.955 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.955 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:28.955 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.530679+0000", 2024-04-19T01:04:28.956 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:28.956 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.956 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.956 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:28.956 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.530679+0000", 2024-04-19T01:04:28.956 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:28.956 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.956 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.956 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:28.956 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.530680+0000", 2024-04-19T01:04:28.956 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7999999999999996e-07 2024-04-19T01:04:28.956 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.956 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.956 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:28.956 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.530735+0000", 2024-04-19T01:04:28.957 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.5235000000000002e-05 2024-04-19T01:04:28.957 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.957 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.957 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:28.957 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.530759+0000", 2024-04-19T01:04:28.957 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4479999999999999e-05 2024-04-19T01:04:28.957 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.957 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.957 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:28.957 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.530781+0000", 2024-04-19T01:04:28.957 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2259000000000002e-05 2024-04-19T01:04:28.957 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.957 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.957 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:28.957 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.530943+0000", 2024-04-19T01:04:28.958 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000161268 2024-04-19T01:04:28.958 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.958 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.958 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:28.958 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.530967+0000", 2024-04-19T01:04:28.958 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4499e-05 2024-04-19T01:04:28.958 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.958 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.958 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:28.958 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.539434+0000", 2024-04-19T01:04:28.958 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0084667590000000008 2024-04-19T01:04:28.958 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.958 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.958 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:28.959 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.539550+0000", 2024-04-19T01:04:28.959 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011626200000000001 2024-04-19T01:04:28.959 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:28.959 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:28.959 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:28.959 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.959 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.960 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.0s3 49/31 ECSubWrite(tid=39, reqid=osd.1.0:39, at_version=49'35, trim_to=0'0, roll_forward_to=48'33))", 2024-04-19T01:04:28.960 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:19.540342+0000", 2024-04-19T01:04:28.960 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 9.3342071989999997, 2024-04-19T01:04:28.960 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0060086879999999999, 2024-04-19T01:04:28.960 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:28.960 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:28.960 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:28.960 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:28.960 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.960 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:28.960 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540342+0000", 2024-04-19T01:04:28.960 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:28.960 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.960 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.960 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:28.961 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540342+0000", 2024-04-19T01:04:28.961 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:28.961 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.961 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.961 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:28.961 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540343+0000", 2024-04-19T01:04:28.961 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4300000000000002e-07 2024-04-19T01:04:28.961 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.961 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.961 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:28.961 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540394+0000", 2024-04-19T01:04:28.961 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.1258999999999998e-05 2024-04-19T01:04:28.961 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.962 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.962 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:28.962 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540427+0000", 2024-04-19T01:04:28.962 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2898999999999998e-05 2024-04-19T01:04:28.962 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.962 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.962 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:28.962 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540450+0000", 2024-04-19T01:04:28.962 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2690000000000001e-05 2024-04-19T01:04:28.962 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.962 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.962 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:28.962 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540584+0000", 2024-04-19T01:04:28.962 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000133714 2024-04-19T01:04:28.963 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.963 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.963 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:28.963 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540604+0000", 2024-04-19T01:04:28.963 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0061e-05 2024-04-19T01:04:28.963 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.963 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.963 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:28.963 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.546254+0000", 2024-04-19T01:04:28.963 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0056502369999999998 2024-04-19T01:04:28.963 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.963 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.963 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:28.963 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.546351+0000", 2024-04-19T01:04:28.964 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7084999999999996e-05 2024-04-19T01:04:28.965 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:28.966 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:28.966 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:28.966 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.966 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.966 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.4s3 49/31 ECSubWrite(tid=55, reqid=osd.10.0:54, at_version=0'0, trim_to=0'0, roll_forward_to=49'18))", 2024-04-19T01:04:28.966 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:19.551133+0000", 2024-04-19T01:04:28.966 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 9.3234168979999996, 2024-04-19T01:04:28.966 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0031049319999999999, 2024-04-19T01:04:28.966 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:28.966 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:28.966 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:28.966 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:28.966 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.966 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:28.967 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.551133+0000", 2024-04-19T01:04:28.967 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:28.967 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.967 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.967 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:28.967 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.551133+0000", 2024-04-19T01:04:28.967 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:28.968 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.968 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.968 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:28.968 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.551135+0000", 2024-04-19T01:04:28.968 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7e-06 2024-04-19T01:04:28.968 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.968 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.968 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:28.968 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.551161+0000", 2024-04-19T01:04:28.968 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5462000000000001e-05 2024-04-19T01:04:28.968 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.968 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.968 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:28.968 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.551197+0000", 2024-04-19T01:04:28.968 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5884999999999999e-05 2024-04-19T01:04:28.969 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.969 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.969 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:28.969 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.551216+0000", 2024-04-19T01:04:28.969 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8890999999999999e-05 2024-04-19T01:04:28.969 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.969 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.970 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:28.970 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.551383+0000", 2024-04-19T01:04:28.970 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016767299999999999 2024-04-19T01:04:28.970 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.970 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.970 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:28.970 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.551411+0000", 2024-04-19T01:04:28.970 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7820000000000001e-05 2024-04-19T01:04:28.970 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.970 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.970 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:28.970 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.554145+0000", 2024-04-19T01:04:28.970 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.002734069 2024-04-19T01:04:28.971 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.971 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.971 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:28.971 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.554238+0000", 2024-04-19T01:04:28.971 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2431999999999999e-05 2024-04-19T01:04:28.971 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:28.971 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:28.971 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:28.971 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.971 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.971 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as3 49/31 ECSubWrite(tid=77, reqid=osd.12.0:73, at_version=0'0, trim_to=0'0, roll_forward_to=49'12))", 2024-04-19T01:04:28.971 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:19.558269+0000", 2024-04-19T01:04:28.971 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 9.3162810700000005, 2024-04-19T01:04:28.971 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.003018841, 2024-04-19T01:04:28.971 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:28.972 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:28.972 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:28.972 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:28.972 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.972 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:28.972 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.558269+0000", 2024-04-19T01:04:28.972 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:28.972 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.972 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.972 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:28.972 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.558269+0000", 2024-04-19T01:04:28.972 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:28.972 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.972 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.972 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:28.973 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.558271+0000", 2024-04-19T01:04:28.973 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.95e-06 2024-04-19T01:04:28.973 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.973 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.973 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:28.973 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.558297+0000", 2024-04-19T01:04:28.973 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.686e-05 2024-04-19T01:04:28.973 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.973 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.973 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:28.973 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.558340+0000", 2024-04-19T01:04:28.973 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3016000000000003e-05 2024-04-19T01:04:28.974 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.974 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.974 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:28.974 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.558364+0000", 2024-04-19T01:04:28.974 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3869000000000001e-05 2024-04-19T01:04:28.974 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.974 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.974 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:28.974 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.558522+0000", 2024-04-19T01:04:28.974 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015731299999999999 2024-04-19T01:04:28.974 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.974 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.974 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:28.974 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.558549+0000", 2024-04-19T01:04:28.975 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7339999999999999e-05 2024-04-19T01:04:28.975 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.975 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.975 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:28.975 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.561177+0000", 2024-04-19T01:04:28.975 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0026280420000000001 2024-04-19T01:04:28.975 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.975 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.975 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:28.975 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.561287+0000", 2024-04-19T01:04:28.975 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000110451 2024-04-19T01:04:28.975 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:28.975 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:28.975 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:28.975 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.976 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.976 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.0s3 49/31 ECSubWrite(tid=40, reqid=osd.1.0:39, at_version=0'0, trim_to=0'0, roll_forward_to=49'35))", 2024-04-19T01:04:28.976 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:19.565188+0000", 2024-04-19T01:04:28.976 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 9.3093615130000007, 2024-04-19T01:04:28.976 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.003494764, 2024-04-19T01:04:28.976 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:28.976 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:28.976 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:28.976 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:28.976 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.976 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:28.976 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.565188+0000", 2024-04-19T01:04:28.976 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:28.977 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.977 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.977 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:28.977 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.565188+0000", 2024-04-19T01:04:28.977 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:28.977 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.977 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.977 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:28.977 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.565191+0000", 2024-04-19T01:04:28.977 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7439999999999999e-06 2024-04-19T01:04:28.977 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.977 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.978 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:28.978 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.565217+0000", 2024-04-19T01:04:28.978 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5712999999999999e-05 2024-04-19T01:04:28.978 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.978 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.978 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:28.978 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.565246+0000", 2024-04-19T01:04:28.978 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8996000000000002e-05 2024-04-19T01:04:28.978 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.978 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.978 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:28.978 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.565262+0000", 2024-04-19T01:04:28.978 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6296000000000001e-05 2024-04-19T01:04:28.978 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.978 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.979 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:28.979 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.565432+0000", 2024-04-19T01:04:28.979 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016969700000000001 2024-04-19T01:04:28.979 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.979 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.979 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:28.979 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.565463+0000", 2024-04-19T01:04:28.979 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1915999999999997e-05 2024-04-19T01:04:28.979 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.979 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.979 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:28.979 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.568629+0000", 2024-04-19T01:04:28.979 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0031654719999999999 2024-04-19T01:04:28.979 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.980 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.980 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:28.980 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.568683+0000", 2024-04-19T01:04:28.980 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.393e-05 2024-04-19T01:04:28.980 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:28.980 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:28.980 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:28.981 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.981 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.981 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as3 58/31 ECSubWrite(tid=90, reqid=osd.12.0:89, at_version=0'0, trim_to=0'0, roll_forward_to=58'18))", 2024-04-19T01:04:28.981 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:28.666810+0000", 2024-04-19T01:04:28.981 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.207739436, 2024-04-19T01:04:28.981 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0062188399999999998, 2024-04-19T01:04:28.981 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:28.981 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:28.981 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:28.981 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:28.981 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.981 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:28.981 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:28.666810+0000", 2024-04-19T01:04:28.981 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:28.981 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.982 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.982 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:28.982 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:28.666810+0000", 2024-04-19T01:04:28.982 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:28.982 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.982 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.982 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:28.982 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:28.666811+0000", 2024-04-19T01:04:28.982 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.9999999999999996e-07 2024-04-19T01:04:28.982 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.982 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.982 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:28.982 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:28.666824+0000", 2024-04-19T01:04:28.982 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.276e-05 2024-04-19T01:04:28.983 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.983 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.983 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:28.983 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:28.666841+0000", 2024-04-19T01:04:28.983 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7234000000000001e-05 2024-04-19T01:04:28.983 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.983 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.983 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:28.983 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:28.666852+0000", 2024-04-19T01:04:28.983 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0625000000000001e-05 2024-04-19T01:04:28.983 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.983 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.983 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:28.983 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:28.666919+0000", 2024-04-19T01:04:28.983 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7657000000000006e-05 2024-04-19T01:04:28.984 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.984 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.984 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:28.984 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:28.666931+0000", 2024-04-19T01:04:28.984 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1608e-05 2024-04-19T01:04:28.984 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.984 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.984 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:28.984 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:28.672986+0000", 2024-04-19T01:04:28.984 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0060553100000000004 2024-04-19T01:04:28.984 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:28.984 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:28.984 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:28.984 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:28.673029+0000", 2024-04-19T01:04:28.985 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2746e-05 2024-04-19T01:04:28.985 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:28.985 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:28.985 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:28.985 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:28.985 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:28.985 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:04:28.987 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-19T01:04:28.988 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:04:28.988+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:04:29.087 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:29.086+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:29.114 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:04:29.114 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:04:29.114 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-04-19T01:04:29.114 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:04:29.128 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-19T01:04:29.166 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2024-04-19T01:04:29.189 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:04:29.190+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:04:29.245 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:04:29.245 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:04:29.246 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-04-19T01:04:29.246 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-04-19T01:04:29.246 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:04:29.260 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-19T01:04:29.290 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:29.292+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:29.336 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 reset_purged_snaps_last 2024-04-19T01:04:29.383 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:04:29.383 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-04-19T01:04:29.383 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-04-19T01:04:29.384 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-04-19T01:04:29.384 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.384 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.2s6 43/31 ECSubWrite(tid=180, reqid=osd.11.0:180, at_version=43'10, trim_to=0'0, roll_forward_to=41'8))", 2024-04-19T01:04:29.384 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:13.845872+0000", 2024-04-19T01:04:29.384 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 15.537938970999999, 2024-04-19T01:04:29.384 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0035934399999999998, 2024-04-19T01:04:29.384 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:29.384 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:29.384 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:29.384 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:29.385 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.385 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:29.385 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.845872+0000", 2024-04-19T01:04:29.385 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:29.385 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.385 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.385 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:29.385 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.845872+0000", 2024-04-19T01:04:29.385 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:29.385 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.385 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.385 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:29.385 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.845872+0000", 2024-04-19T01:04:29.385 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9299999999999998e-07 2024-04-19T01:04:29.386 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.386 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.386 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:29.386 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.845906+0000", 2024-04-19T01:04:29.386 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3352000000000002e-05 2024-04-19T01:04:29.386 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.386 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.386 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:29.386 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.845932+0000", 2024-04-19T01:04:29.386 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6653999999999999e-05 2024-04-19T01:04:29.386 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.386 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.387 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:29.387 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.845948+0000", 2024-04-19T01:04:29.387 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5379e-05 2024-04-19T01:04:29.387 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.387 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.387 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:29.387 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.847979+0000", 2024-04-19T01:04:29.387 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00203127 2024-04-19T01:04:29.387 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.387 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.387 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:29.387 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.848008+0000", 2024-04-19T01:04:29.387 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9073000000000001e-05 2024-04-19T01:04:29.387 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.387 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.388 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:29.388 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.849418+0000", 2024-04-19T01:04:29.388 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0014101059999999999 2024-04-19T01:04:29.388 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.388 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.388 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:29.388 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.849465+0000", 2024-04-19T01:04:29.388 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7113000000000001e-05 2024-04-19T01:04:29.388 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:29.388 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:29.388 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:29.388 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.388 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.388 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 43/31 ECSubWrite(tid=47, reqid=client.4604.0:263, at_version=43'8, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:04:29.389 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:13.999456+0000", 2024-04-19T01:04:29.389 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 15.384354832, 2024-04-19T01:04:29.389 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0090503839999999999, 2024-04-19T01:04:29.389 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:29.389 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:29.389 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:29.389 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:29.389 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.390 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:29.390 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.999456+0000", 2024-04-19T01:04:29.397 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:29.398 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.398 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.398 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:29.398 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.999456+0000", 2024-04-19T01:04:29.398 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:29.398 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.398 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.398 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:29.398 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.999458+0000", 2024-04-19T01:04:29.398 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.874e-06 2024-04-19T01:04:29.398 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.398 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.399 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:29.399 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.999813+0000", 2024-04-19T01:04:29.399 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00035491100000000002 2024-04-19T01:04:29.399 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.399 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.399 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:29.399 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.999832+0000", 2024-04-19T01:04:29.399 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9425000000000001e-05 2024-04-19T01:04:29.399 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.400 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.400 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:29.400 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.999844+0000", 2024-04-19T01:04:29.400 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1838000000000001e-05 2024-04-19T01:04:29.400 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.400 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.400 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:29.400 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.999950+0000", 2024-04-19T01:04:29.400 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000106125 2024-04-19T01:04:29.400 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.401 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.401 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:29.401 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.999965+0000", 2024-04-19T01:04:29.401 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4966e-05 2024-04-19T01:04:29.401 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.401 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.401 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:29.401 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.008459+0000", 2024-04-19T01:04:29.401 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0084938979999999997 2024-04-19T01:04:29.401 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.401 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.401 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:29.402 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.008506+0000", 2024-04-19T01:04:29.402 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7346999999999999e-05 2024-04-19T01:04:29.405 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:29.405 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:29.405 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:29.405 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.405 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.405 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 43/31 ECSubWrite(tid=48, reqid=client.4604.0:264, at_version=43'9, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:04:29.405 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:14.001305+0000", 2024-04-19T01:04:29.405 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 15.382505878, 2024-04-19T01:04:29.405 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0076904640000000002, 2024-04-19T01:04:29.406 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:29.406 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:29.406 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:29.406 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:29.406 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.406 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:29.406 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.001305+0000", 2024-04-19T01:04:29.406 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:29.406 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.406 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.406 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:29.406 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.001305+0000", 2024-04-19T01:04:29.406 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:29.407 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.407 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.407 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:29.407 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.001306+0000", 2024-04-19T01:04:29.407 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5700000000000001e-07 2024-04-19T01:04:29.407 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.407 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.407 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:29.407 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.001417+0000", 2024-04-19T01:04:29.407 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011103900000000001 2024-04-19T01:04:29.407 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.407 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.407 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:29.407 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.001434+0000", 2024-04-19T01:04:29.407 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7075000000000002e-05 2024-04-19T01:04:29.408 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.408 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.408 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:29.408 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.001442+0000", 2024-04-19T01:04:29.408 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6930000000000007e-06 2024-04-19T01:04:29.408 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.408 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.408 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:29.408 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.005465+0000", 2024-04-19T01:04:29.408 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0040232990000000001 2024-04-19T01:04:29.408 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.408 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.408 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:29.408 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.005476+0000", 2024-04-19T01:04:29.408 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0781999999999999e-05 2024-04-19T01:04:29.409 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.409 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.409 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:29.409 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.008961+0000", 2024-04-19T01:04:29.409 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0034857299999999998 2024-04-19T01:04:29.409 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.409 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.409 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:29.409 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.008995+0000", 2024-04-19T01:04:29.409 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3989000000000003e-05 2024-04-19T01:04:29.409 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:29.409 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:29.409 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:29.409 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.410 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.410 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 43/31 ECSubWrite(tid=49, reqid=client.4604.0:265, at_version=43'10, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:04:29.410 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:14.004792+0000", 2024-04-19T01:04:29.410 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 15.379019231000001, 2024-04-19T01:04:29.410 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0064878169999999999, 2024-04-19T01:04:29.410 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:29.410 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:29.410 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:29.410 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:29.411 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.411 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:29.411 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.004792+0000", 2024-04-19T01:04:29.411 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:29.411 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.411 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.411 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:29.411 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.004792+0000", 2024-04-19T01:04:29.411 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:29.411 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.411 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.411 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:29.411 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.004792+0000", 2024-04-19T01:04:29.411 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2299999999999999e-07 2024-04-19T01:04:29.412 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.412 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.412 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:29.412 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.004821+0000", 2024-04-19T01:04:29.412 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8365e-05 2024-04-19T01:04:29.412 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.412 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.412 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:29.412 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.004836+0000", 2024-04-19T01:04:29.412 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4894e-05 2024-04-19T01:04:29.412 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.412 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.412 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:29.412 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.004846+0000", 2024-04-19T01:04:29.413 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0579e-05 2024-04-19T01:04:29.413 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.413 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.413 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:29.413 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.008569+0000", 2024-04-19T01:04:29.413 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0037230079999999999 2024-04-19T01:04:29.413 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.413 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.413 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:29.413 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.008582+0000", 2024-04-19T01:04:29.413 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2330000000000001e-05 2024-04-19T01:04:29.413 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.413 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.413 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:29.414 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.011248+0000", 2024-04-19T01:04:29.414 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0026668220000000001 2024-04-19T01:04:29.414 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.414 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.421 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:29.422 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.011279+0000", 2024-04-19T01:04:29.422 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0895999999999997e-05 2024-04-19T01:04:29.431 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:29.431 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:29.431 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:29.431 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.431 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.431 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.7s2 44/31 ECSubWrite(tid=11, reqid=osd.14.0:11, at_version=44'10, trim_to=0'0, roll_forward_to=43'8))", 2024-04-19T01:04:29.431 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:14.885453+0000", 2024-04-19T01:04:29.432 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.498357799000001, 2024-04-19T01:04:29.432 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0054383390000000004, 2024-04-19T01:04:29.432 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:29.432 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:29.432 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:29.432 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:29.432 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.432 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:29.432 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.885453+0000", 2024-04-19T01:04:29.432 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:29.432 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.432 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.432 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:29.432 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.885453+0000", 2024-04-19T01:04:29.433 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:29.433 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.433 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.433 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:29.433 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.885454+0000", 2024-04-19T01:04:29.433 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.164e-06 2024-04-19T01:04:29.433 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.433 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.433 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:29.433 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.885526+0000", 2024-04-19T01:04:29.433 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.1773999999999994e-05 2024-04-19T01:04:29.433 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.433 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.433 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:29.434 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.885556+0000", 2024-04-19T01:04:29.434 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0147000000000001e-05 2024-04-19T01:04:29.434 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.434 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.434 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:29.434 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.885576+0000", 2024-04-19T01:04:29.434 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9809e-05 2024-04-19T01:04:29.434 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.434 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.434 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:29.434 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.885706+0000", 2024-04-19T01:04:29.434 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000130481 2024-04-19T01:04:29.434 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.434 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.435 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:29.435 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.885719+0000", 2024-04-19T01:04:29.435 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2454e-05 2024-04-19T01:04:29.435 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.435 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.435 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:29.435 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.890793+0000", 2024-04-19T01:04:29.435 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0050742390000000004 2024-04-19T01:04:29.435 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.435 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.435 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:29.435 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.890891+0000", 2024-04-19T01:04:29.435 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.8270999999999995e-05 2024-04-19T01:04:29.435 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:29.435 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:29.436 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:29.442 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.442 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.442 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 44/31 ECSubWrite(tid=53, reqid=osd.0.0:53, at_version=44'12, trim_to=0'0, roll_forward_to=43'10))", 2024-04-19T01:04:29.442 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:14.887273+0000", 2024-04-19T01:04:29.442 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.496538125000001, 2024-04-19T01:04:29.443 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0081171100000000003, 2024-04-19T01:04:29.443 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:29.443 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:29.443 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:29.443 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:29.443 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.443 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:29.443 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.887273+0000", 2024-04-19T01:04:29.443 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:29.443 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.443 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.443 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:29.443 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.887273+0000", 2024-04-19T01:04:29.443 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:29.444 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.444 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.444 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:29.444 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.887273+0000", 2024-04-19T01:04:29.444 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8999999999999995e-07 2024-04-19T01:04:29.444 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.444 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.444 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:29.444 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.887331+0000", 2024-04-19T01:04:29.444 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7905000000000002e-05 2024-04-19T01:04:29.444 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.444 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.444 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:29.444 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.887362+0000", 2024-04-19T01:04:29.444 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1086e-05 2024-04-19T01:04:29.445 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.445 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.445 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:29.445 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.887385+0000", 2024-04-19T01:04:29.445 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2843000000000001e-05 2024-04-19T01:04:29.445 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.445 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.448 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:29.448 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.887605+0000", 2024-04-19T01:04:29.448 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021989899999999999 2024-04-19T01:04:29.448 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.448 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.449 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:29.449 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.887631+0000", 2024-04-19T01:04:29.449 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6225999999999999e-05 2024-04-19T01:04:29.449 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.449 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.449 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:29.449 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.895300+0000", 2024-04-19T01:04:29.449 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.007668467 2024-04-19T01:04:29.450 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.450 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.450 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:29.450 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.895390+0000", 2024-04-19T01:04:29.450 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.9894000000000004e-05 2024-04-19T01:04:29.450 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:29.450 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:29.450 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:29.450 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.450 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.450 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 44/31 ECSubWrite(tid=54, reqid=osd.0.0:53, at_version=0'0, trim_to=0'0, roll_forward_to=44'12))", 2024-04-19T01:04:29.450 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:14.896306+0000", 2024-04-19T01:04:29.450 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.487504528000001, 2024-04-19T01:04:29.450 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0034849210000000002, 2024-04-19T01:04:29.451 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:29.451 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:29.451 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:29.451 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:29.451 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.451 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:29.451 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.896306+0000", 2024-04-19T01:04:29.451 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:29.451 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.451 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.451 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:29.451 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.896306+0000", 2024-04-19T01:04:29.451 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:29.451 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.452 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.452 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:29.452 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.896307+0000", 2024-04-19T01:04:29.452 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.08e-06 2024-04-19T01:04:29.452 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.452 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.452 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:29.452 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.896337+0000", 2024-04-19T01:04:29.452 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0108e-05 2024-04-19T01:04:29.452 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.452 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.453 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:29.453 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.896353+0000", 2024-04-19T01:04:29.453 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5282999999999999e-05 2024-04-19T01:04:29.453 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.453 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.453 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:29.453 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.896365+0000", 2024-04-19T01:04:29.453 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1901e-05 2024-04-19T01:04:29.453 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.453 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.453 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:29.453 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.896569+0000", 2024-04-19T01:04:29.453 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00020439500000000001 2024-04-19T01:04:29.453 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.454 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.454 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:29.455 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.896594+0000", 2024-04-19T01:04:29.455 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4987e-05 2024-04-19T01:04:29.455 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.455 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.455 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:29.455 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.899691+0000", 2024-04-19T01:04:29.455 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0030974029999999999 2024-04-19T01:04:29.455 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.455 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.456 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:29.456 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.899791+0000", 2024-04-19T01:04:29.456 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9764000000000002e-05 2024-04-19T01:04:29.456 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:29.456 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:29.456 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:29.456 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.456 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.456 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 47/31 ECSubWrite(tid=61, reqid=client.4604.0:317, at_version=47'16, trim_to=0'0, roll_forward_to=45'14))", 2024-04-19T01:04:29.456 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:17.604486+0000", 2024-04-19T01:04:29.456 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 11.779325282, 2024-04-19T01:04:29.456 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.025312589, 2024-04-19T01:04:29.456 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:29.457 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:29.457 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:29.457 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:29.457 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.457 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:29.457 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.604486+0000", 2024-04-19T01:04:29.457 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:29.457 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.457 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.457 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:29.457 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.604486+0000", 2024-04-19T01:04:29.457 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:29.457 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.457 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.458 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:29.458 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.604486+0000", 2024-04-19T01:04:29.458 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.9999999999999997e-07 2024-04-19T01:04:29.458 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.458 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.458 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:29.458 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.604960+0000", 2024-04-19T01:04:29.458 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00047350300000000001 2024-04-19T01:04:29.458 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.458 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.458 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:29.458 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.605002+0000", 2024-04-19T01:04:29.458 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1949000000000001e-05 2024-04-19T01:04:29.458 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.459 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.460 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:29.460 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.605032+0000", 2024-04-19T01:04:29.460 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9865999999999998e-05 2024-04-19T01:04:29.460 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.460 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.460 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:29.460 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.605289+0000", 2024-04-19T01:04:29.460 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00025766500000000002 2024-04-19T01:04:29.460 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.460 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.460 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:29.460 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.605322+0000", 2024-04-19T01:04:29.460 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2845999999999997e-05 2024-04-19T01:04:29.461 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.461 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.461 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:29.461 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.629741+0000", 2024-04-19T01:04:29.461 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.024419335 2024-04-19T01:04:29.461 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.461 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.461 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:29.461 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.629798+0000", 2024-04-19T01:04:29.461 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6725000000000003e-05 2024-04-19T01:04:29.461 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:29.461 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:29.461 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:29.462 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.462 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.462 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 47/31 ECSubWrite(tid=62, reqid=client.4604.0:318, at_version=47'17, trim_to=0'0, roll_forward_to=45'14))", 2024-04-19T01:04:29.462 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:17.611532+0000", 2024-04-19T01:04:29.463 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 11.772278368, 2024-04-19T01:04:29.463 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.018327896, 2024-04-19T01:04:29.463 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:29.464 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:29.464 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:29.464 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:29.464 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.464 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:29.464 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.611532+0000", 2024-04-19T01:04:29.464 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:29.464 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.464 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.464 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:29.464 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.611532+0000", 2024-04-19T01:04:29.464 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:29.465 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.465 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.465 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:29.465 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.611534+0000", 2024-04-19T01:04:29.465 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.979e-06 2024-04-19T01:04:29.465 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.465 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.465 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:29.465 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.611863+0000", 2024-04-19T01:04:29.465 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00032853100000000002 2024-04-19T01:04:29.465 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.465 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.465 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:29.465 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.611907+0000", 2024-04-19T01:04:29.466 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4425000000000002e-05 2024-04-19T01:04:29.466 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.466 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.466 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:29.466 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.611935+0000", 2024-04-19T01:04:29.466 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7993000000000001e-05 2024-04-19T01:04:29.466 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.466 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.466 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:29.466 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.617071+0000", 2024-04-19T01:04:29.466 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0051354069999999998 2024-04-19T01:04:29.467 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.467 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.467 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:29.467 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.617091+0000", 2024-04-19T01:04:29.467 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0061999999999999e-05 2024-04-19T01:04:29.467 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.467 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.467 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:29.467 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.629827+0000", 2024-04-19T01:04:29.467 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.012735943 2024-04-19T01:04:29.467 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.467 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.467 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:29.468 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.629860+0000", 2024-04-19T01:04:29.468 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3556000000000002e-05 2024-04-19T01:04:29.468 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:29.468 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:29.468 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:29.468 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.468 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.468 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 47/31 ECSubWrite(tid=63, reqid=client.4604.0:319, at_version=47'18, trim_to=0'0, roll_forward_to=45'14))", 2024-04-19T01:04:29.468 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:17.621137+0000", 2024-04-19T01:04:29.468 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 11.76267412, 2024-04-19T01:04:29.468 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0092948300000000005, 2024-04-19T01:04:29.468 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:29.468 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:29.469 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:29.469 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:29.469 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.469 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:29.469 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.621137+0000", 2024-04-19T01:04:29.469 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:29.469 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.470 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.470 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:29.470 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.621137+0000", 2024-04-19T01:04:29.470 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:29.470 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.470 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.471 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:29.471 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.621139+0000", 2024-04-19T01:04:29.471 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3130000000000001e-06 2024-04-19T01:04:29.471 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.471 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.471 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:29.471 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.621276+0000", 2024-04-19T01:04:29.471 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013658400000000001 2024-04-19T01:04:29.471 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.471 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.471 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:29.471 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.621317+0000", 2024-04-19T01:04:29.472 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1152000000000002e-05 2024-04-19T01:04:29.472 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.472 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.472 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:29.472 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.621341+0000", 2024-04-19T01:04:29.472 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4600000000000002e-05 2024-04-19T01:04:29.472 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.472 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.472 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:29.472 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.627256+0000", 2024-04-19T01:04:29.472 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0059141890000000003 2024-04-19T01:04:29.472 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.472 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.473 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:29.473 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.627268+0000", 2024-04-19T01:04:29.473 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2903e-05 2024-04-19T01:04:29.473 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.473 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.473 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:29.473 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.630366+0000", 2024-04-19T01:04:29.473 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0030979879999999999 2024-04-19T01:04:29.473 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.473 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.473 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:29.473 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.630432+0000", 2024-04-19T01:04:29.473 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5100999999999999e-05 2024-04-19T01:04:29.473 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:29.473 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:29.474 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:29.474 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.474 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.474 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 47/31 ECSubWrite(tid=64, reqid=client.4604.0:320, at_version=47'19, trim_to=0'0, roll_forward_to=45'14))", 2024-04-19T01:04:29.474 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:17.623031+0000", 2024-04-19T01:04:29.474 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 11.760779385999999, 2024-04-19T01:04:29.474 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0081197209999999999, 2024-04-19T01:04:29.474 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:29.474 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:29.474 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:29.474 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:29.474 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.474 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:29.475 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.623031+0000", 2024-04-19T01:04:29.475 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:29.475 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.475 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.475 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:29.475 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.623031+0000", 2024-04-19T01:04:29.475 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:29.475 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.475 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.475 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:29.475 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.623033+0000", 2024-04-19T01:04:29.476 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.694e-06 2024-04-19T01:04:29.476 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.476 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.476 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:29.476 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.623067+0000", 2024-04-19T01:04:29.476 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3436999999999999e-05 2024-04-19T01:04:29.476 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.476 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.476 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:29.476 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.623093+0000", 2024-04-19T01:04:29.476 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6469999999999999e-05 2024-04-19T01:04:29.476 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.476 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.477 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:29.477 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.623114+0000", 2024-04-19T01:04:29.477 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1423000000000001e-05 2024-04-19T01:04:29.477 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.477 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.477 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:29.477 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.629946+0000", 2024-04-19T01:04:29.477 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0068313330000000002 2024-04-19T01:04:29.477 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.477 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.477 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:29.478 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.629963+0000", 2024-04-19T01:04:29.478 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7289e-05 2024-04-19T01:04:29.478 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.478 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.478 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:29.478 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.631074+0000", 2024-04-19T01:04:29.478 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011109570000000001 2024-04-19T01:04:29.478 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.478 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.478 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:29.478 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.631151+0000", 2024-04-19T01:04:29.478 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7118000000000003e-05 2024-04-19T01:04:29.479 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:29.479 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:29.479 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:29.479 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.479 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.479 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 48/31 ECSubWrite(tid=75, reqid=osd.0.0:75, at_version=48'21, trim_to=0'0, roll_forward_to=47'19))", 2024-04-19T01:04:29.479 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.528567+0000", 2024-04-19T01:04:29.479 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.855243550000001, 2024-04-19T01:04:29.479 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0061716480000000001, 2024-04-19T01:04:29.479 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:29.479 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:29.480 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:29.480 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:29.480 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.480 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:29.480 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.528567+0000", 2024-04-19T01:04:29.480 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:29.480 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.480 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.480 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:29.480 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.528567+0000", 2024-04-19T01:04:29.480 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:29.480 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.480 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.480 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:29.481 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.528568+0000", 2024-04-19T01:04:29.481 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0999999999999998e-07 2024-04-19T01:04:29.481 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.481 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.481 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:29.481 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.528627+0000", 2024-04-19T01:04:29.481 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8745000000000001e-05 2024-04-19T01:04:29.481 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.481 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.481 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:29.481 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.528665+0000", 2024-04-19T01:04:29.481 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8479000000000002e-05 2024-04-19T01:04:29.482 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.482 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.482 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:29.482 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.528694+0000", 2024-04-19T01:04:29.482 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8770000000000001e-05 2024-04-19T01:04:29.482 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.482 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.482 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:29.482 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.528825+0000", 2024-04-19T01:04:29.482 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013142500000000001 2024-04-19T01:04:29.482 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.482 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.482 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:29.483 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.528840+0000", 2024-04-19T01:04:29.483 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4436999999999999e-05 2024-04-19T01:04:29.483 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.483 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.483 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:29.483 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.534649+0000", 2024-04-19T01:04:29.483 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0058094840000000002 2024-04-19T01:04:29.483 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.483 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.483 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:29.483 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.534739+0000", 2024-04-19T01:04:29.483 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.9598000000000005e-05 2024-04-19T01:04:29.484 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:29.484 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:29.484 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:29.484 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.484 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.484 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 48/31 ECSubWrite(tid=76, reqid=client.4604.0:345, at_version=48'22, trim_to=0'0, roll_forward_to=47'19))", 2024-04-19T01:04:29.484 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.537945+0000", 2024-04-19T01:04:29.484 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.845865392, 2024-04-19T01:04:29.484 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.034986066000000003, 2024-04-19T01:04:29.484 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:29.484 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:29.484 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:29.485 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:29.485 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.485 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:29.485 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.537945+0000", 2024-04-19T01:04:29.485 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:29.485 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.485 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.485 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:29.485 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.537945+0000", 2024-04-19T01:04:29.485 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:29.485 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.485 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.485 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:29.485 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.537948+0000", 2024-04-19T01:04:29.485 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6929999999999999e-06 2024-04-19T01:04:29.486 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.486 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.486 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:29.486 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.538485+0000", 2024-04-19T01:04:29.486 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00053661900000000001 2024-04-19T01:04:29.486 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.486 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.486 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:29.486 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.538530+0000", 2024-04-19T01:04:29.487 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5423000000000002e-05 2024-04-19T01:04:29.487 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.487 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.487 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:29.487 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.538558+0000", 2024-04-19T01:04:29.487 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7569999999999999e-05 2024-04-19T01:04:29.487 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.487 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.487 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:29.487 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.538778+0000", 2024-04-19T01:04:29.487 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000220478 2024-04-19T01:04:29.488 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.488 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.488 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:29.488 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.538806+0000", 2024-04-19T01:04:29.488 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8197000000000001e-05 2024-04-19T01:04:29.488 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.488 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.488 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:29.488 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.572854+0000", 2024-04-19T01:04:29.488 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.034047315000000002 2024-04-19T01:04:29.488 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.489 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.489 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:29.489 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.572931+0000", 2024-04-19T01:04:29.489 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7770999999999998e-05 2024-04-19T01:04:29.489 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:29.489 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:29.489 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:29.489 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.490 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.490 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 48/31 ECSubWrite(tid=78, reqid=client.4604.0:346, at_version=48'23, trim_to=0'0, roll_forward_to=48'21))", 2024-04-19T01:04:29.490 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.548797+0000", 2024-04-19T01:04:29.490 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.835013327, 2024-04-19T01:04:29.490 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.024207922, 2024-04-19T01:04:29.490 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:29.491 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:29.491 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:29.491 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:29.491 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.491 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:29.491 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.548797+0000", 2024-04-19T01:04:29.491 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:29.491 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.491 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.491 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:29.491 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.548797+0000", 2024-04-19T01:04:29.491 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:29.491 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.491 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.492 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:29.492 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.548799+0000", 2024-04-19T01:04:29.492 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.604e-06 2024-04-19T01:04:29.492 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.492 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.492 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:29.492 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.548991+0000", 2024-04-19T01:04:29.492 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019153400000000001 2024-04-19T01:04:29.492 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.492 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.492 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:29.492 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.549011+0000", 2024-04-19T01:04:29.492 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0332000000000001e-05 2024-04-19T01:04:29.492 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.493 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.493 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:29.493 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.549025+0000", 2024-04-19T01:04:29.493 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4260000000000001e-05 2024-04-19T01:04:29.493 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.493 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.493 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:29.493 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.552893+0000", 2024-04-19T01:04:29.493 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0038674460000000001 2024-04-19T01:04:29.493 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.493 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.493 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:29.493 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.552906+0000", 2024-04-19T01:04:29.493 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3105000000000001e-05 2024-04-19T01:04:29.493 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.494 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.494 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:29.494 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.572950+0000", 2024-04-19T01:04:29.494 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.020043787 2024-04-19T01:04:29.494 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.494 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.494 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:29.494 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.573005+0000", 2024-04-19T01:04:29.494 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.5853999999999998e-05 2024-04-19T01:04:29.494 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:29.495 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:29.495 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:29.495 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.495 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.495 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 48/31 ECSubWrite(tid=79, reqid=client.4604.0:347, at_version=48'24, trim_to=0'0, roll_forward_to=48'21))", 2024-04-19T01:04:29.495 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.555132+0000", 2024-04-19T01:04:29.495 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.828678894999999, 2024-04-19T01:04:29.495 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.018366424999999999, 2024-04-19T01:04:29.495 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:29.495 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:29.495 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:29.495 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:29.495 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.495 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:29.496 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.555132+0000", 2024-04-19T01:04:29.496 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:29.496 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.496 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.496 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:29.496 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.555132+0000", 2024-04-19T01:04:29.496 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:29.496 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.496 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.496 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:29.496 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.555133+0000", 2024-04-19T01:04:29.496 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.42e-06 2024-04-19T01:04:29.496 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.496 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.497 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:29.497 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.555208+0000", 2024-04-19T01:04:29.497 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4532999999999999e-05 2024-04-19T01:04:29.497 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.497 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.497 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:29.497 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.555254+0000", 2024-04-19T01:04:29.497 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6031e-05 2024-04-19T01:04:29.497 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.497 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.497 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:29.498 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.555268+0000", 2024-04-19T01:04:29.498 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4068e-05 2024-04-19T01:04:29.498 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.498 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.498 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:29.498 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.571711+0000", 2024-04-19T01:04:29.498 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.016442656999999999 2024-04-19T01:04:29.498 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.498 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.498 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:29.499 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.571720+0000", 2024-04-19T01:04:29.499 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1370000000000008e-06 2024-04-19T01:04:29.499 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.499 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.499 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:29.499 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.573461+0000", 2024-04-19T01:04:29.499 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0017408180000000001 2024-04-19T01:04:29.499 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.499 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.499 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:29.499 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.573498+0000", 2024-04-19T01:04:29.499 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7761e-05 2024-04-19T01:04:29.499 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:29.499 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:29.499 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:29.500 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.500 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.500 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as2 48/31 ECSubWrite(tid=71, reqid=client.4604.0:354, at_version=48'10, trim_to=0'0, roll_forward_to=34'8))", 2024-04-19T01:04:29.500 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.556963+0000", 2024-04-19T01:04:29.500 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.826848177, 2024-04-19T01:04:29.500 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.029734679, 2024-04-19T01:04:29.500 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:29.500 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:29.500 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:29.500 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:29.500 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.500 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:29.500 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.556963+0000", 2024-04-19T01:04:29.500 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:29.500 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.501 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.501 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:29.501 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.556963+0000", 2024-04-19T01:04:29.501 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:29.501 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.501 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.501 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:29.501 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.556963+0000", 2024-04-19T01:04:29.501 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5499999999999998e-07 2024-04-19T01:04:29.501 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.501 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.501 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:29.501 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.558128+0000", 2024-04-19T01:04:29.501 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011650180000000001 2024-04-19T01:04:29.501 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.502 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.502 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:29.502 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.558199+0000", 2024-04-19T01:04:29.502 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0630999999999995e-05 2024-04-19T01:04:29.502 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.502 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.502 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:29.502 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.558221+0000", 2024-04-19T01:04:29.502 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2546999999999999e-05 2024-04-19T01:04:29.502 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.502 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.502 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:29.502 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.558350+0000", 2024-04-19T01:04:29.502 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012838699999999999 2024-04-19T01:04:29.503 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.503 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.503 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:29.503 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.558370+0000", 2024-04-19T01:04:29.503 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0106999999999999e-05 2024-04-19T01:04:29.503 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.503 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.503 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:29.503 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.586643+0000", 2024-04-19T01:04:29.503 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.028273007999999999 2024-04-19T01:04:29.503 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.504 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.504 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:29.504 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.586697+0000", 2024-04-19T01:04:29.504 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4326000000000001e-05 2024-04-19T01:04:29.504 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:29.504 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:29.504 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:29.504 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.504 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.504 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 48/31 ECSubWrite(tid=80, reqid=client.4604.0:348, at_version=48'25, trim_to=0'0, roll_forward_to=48'21))", 2024-04-19T01:04:29.504 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.559574+0000", 2024-04-19T01:04:29.504 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.824236904999999, 2024-04-19T01:04:29.504 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.016989658000000001, 2024-04-19T01:04:29.504 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:29.504 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:29.505 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:29.505 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:29.505 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.505 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:29.505 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.559574+0000", 2024-04-19T01:04:29.505 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:29.505 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.505 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.505 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:29.505 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.559574+0000", 2024-04-19T01:04:29.505 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:29.505 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.505 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.505 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:29.506 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.559574+0000", 2024-04-19T01:04:29.506 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9400000000000001e-07 2024-04-19T01:04:29.506 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.506 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.506 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:29.506 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.559589+0000", 2024-04-19T01:04:29.506 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4873e-05 2024-04-19T01:04:29.506 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.506 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.506 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:29.506 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.559597+0000", 2024-04-19T01:04:29.506 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.1089999999999998e-06 2024-04-19T01:04:29.506 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.506 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.507 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:29.507 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.559605+0000", 2024-04-19T01:04:29.507 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.267e-06 2024-04-19T01:04:29.507 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.507 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.507 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:29.507 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.573085+0000", 2024-04-19T01:04:29.507 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.013480373 2024-04-19T01:04:29.507 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.507 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.507 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:29.507 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.573107+0000", 2024-04-19T01:04:29.507 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2031999999999998e-05 2024-04-19T01:04:29.507 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.507 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.508 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:29.508 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.576518+0000", 2024-04-19T01:04:29.508 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.003411458 2024-04-19T01:04:29.508 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.508 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.508 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:29.508 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.576564+0000", 2024-04-19T01:04:29.508 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5151999999999997e-05 2024-04-19T01:04:29.508 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:29.509 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:29.509 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:29.509 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.509 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.509 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 49/31 ECSubWrite(tid=84, reqid=osd.0.0:84, at_version=49'27, trim_to=0'0, roll_forward_to=48'25))", 2024-04-19T01:04:29.509 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:19.526319+0000", 2024-04-19T01:04:29.509 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 9.85749165, 2024-04-19T01:04:29.509 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0060582550000000002, 2024-04-19T01:04:29.509 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:29.509 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:29.509 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:29.509 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:29.509 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.509 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:29.510 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.526319+0000", 2024-04-19T01:04:29.510 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:29.510 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.510 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.510 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:29.510 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.526319+0000", 2024-04-19T01:04:29.510 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:29.510 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.510 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.510 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:29.510 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.526320+0000", 2024-04-19T01:04:29.510 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2e-07 2024-04-19T01:04:29.510 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.510 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.511 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:29.511 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.526382+0000", 2024-04-19T01:04:29.511 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2181999999999995e-05 2024-04-19T01:04:29.511 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.511 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.511 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:29.511 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.526414+0000", 2024-04-19T01:04:29.511 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1673e-05 2024-04-19T01:04:29.511 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.512 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.512 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:29.512 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.526435+0000", 2024-04-19T01:04:29.512 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1863000000000001e-05 2024-04-19T01:04:29.512 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.512 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.512 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:29.512 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.526666+0000", 2024-04-19T01:04:29.512 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000230392 2024-04-19T01:04:29.512 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.512 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.512 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:29.512 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.526691+0000", 2024-04-19T01:04:29.512 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5239999999999999e-05 2024-04-19T01:04:29.513 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.513 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.513 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:29.513 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.532295+0000", 2024-04-19T01:04:29.513 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0056042289999999996 2024-04-19T01:04:29.513 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.513 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.513 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:29.513 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.532377+0000", 2024-04-19T01:04:29.513 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.2156000000000004e-05 2024-04-19T01:04:29.513 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:29.513 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:29.513 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:29.513 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.514 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.514 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as2 49/31 ECSubWrite(tid=73, reqid=osd.12.0:73, at_version=49'12, trim_to=0'0, roll_forward_to=48'10))", 2024-04-19T01:04:29.514 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:19.530825+0000", 2024-04-19T01:04:29.514 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 9.8529853230000004, 2024-04-19T01:04:29.514 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0042150310000000002, 2024-04-19T01:04:29.514 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:29.514 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:29.514 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:29.514 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:29.514 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.514 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:29.514 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.530825+0000", 2024-04-19T01:04:29.514 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:29.514 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.514 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.515 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:29.515 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.530825+0000", 2024-04-19T01:04:29.515 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:29.515 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.515 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.515 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:29.515 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.530826+0000", 2024-04-19T01:04:29.515 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5600000000000005e-07 2024-04-19T01:04:29.515 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.515 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.515 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:29.515 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.530884+0000", 2024-04-19T01:04:29.515 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7615999999999999e-05 2024-04-19T01:04:29.515 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.515 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.516 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:29.516 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.531047+0000", 2024-04-19T01:04:29.516 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000162776 2024-04-19T01:04:29.516 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.516 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.516 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:29.516 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.531075+0000", 2024-04-19T01:04:29.516 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.794e-05 2024-04-19T01:04:29.516 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.516 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.516 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:29.516 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.531218+0000", 2024-04-19T01:04:29.516 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000143623 2024-04-19T01:04:29.516 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.517 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.517 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:29.517 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.531240+0000", 2024-04-19T01:04:29.517 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1413e-05 2024-04-19T01:04:29.517 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.517 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.517 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:29.517 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.534122+0000", 2024-04-19T01:04:29.517 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0028821129999999999 2024-04-19T01:04:29.517 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.517 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.517 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:29.517 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.535041+0000", 2024-04-19T01:04:29.517 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00091879400000000001 2024-04-19T01:04:29.517 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:29.518 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:29.518 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:29.518 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.518 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.518 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as2 58/31 ECSubWrite(tid=89, reqid=osd.12.0:89, at_version=58'18, trim_to=0'0, roll_forward_to=57'16))", 2024-04-19T01:04:29.518 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:28.662902+0000", 2024-04-19T01:04:29.518 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.72090903200000001, 2024-04-19T01:04:29.519 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0035496159999999998, 2024-04-19T01:04:29.519 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:29.519 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:29.519 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:29.519 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:29.519 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.519 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:29.519 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:28.662902+0000", 2024-04-19T01:04:29.519 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:29.519 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.519 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.519 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:29.519 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:28.662902+0000", 2024-04-19T01:04:29.519 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:29.519 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.520 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.520 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:29.520 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:28.662902+0000", 2024-04-19T01:04:29.520 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3700000000000001e-07 2024-04-19T01:04:29.520 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.520 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.520 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:29.520 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:28.662933+0000", 2024-04-19T01:04:29.520 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0941000000000001e-05 2024-04-19T01:04:29.520 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.521 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.521 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:29.521 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:28.662952+0000", 2024-04-19T01:04:29.521 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8715999999999998e-05 2024-04-19T01:04:29.521 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.521 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.521 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:29.521 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:28.662966+0000", 2024-04-19T01:04:29.521 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3889e-05 2024-04-19T01:04:29.521 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.521 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.521 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:29.521 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:28.663035+0000", 2024-04-19T01:04:29.521 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.9271999999999997e-05 2024-04-19T01:04:29.521 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.522 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.522 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:29.522 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:28.663044+0000", 2024-04-19T01:04:29.522 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.9099999999999994e-06 2024-04-19T01:04:29.522 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.522 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.522 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:29.522 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:28.666415+0000", 2024-04-19T01:04:29.522 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0033710799999999998 2024-04-19T01:04:29.522 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:29.522 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:29.522 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:29.522 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:28.666451+0000", 2024-04-19T01:04:29.522 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6470999999999997e-05 2024-04-19T01:04:29.523 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:29.523 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:29.523 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:29.523 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:29.523 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:29.523 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:04:29.525 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:29.522+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:29.526 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 reset_purged_snaps_last 2024-04-19T01:04:29.526 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2024-04-19T01:04:29.527 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-19T01:04:29.528 INFO:tasks.rados.rados.0.smithi155.stdout:169: done (1 left) 2024-04-19T01:04:29.528 INFO:tasks.rados.rados.0.smithi155.stdout:170: read oid 23 snap -1 2024-04-19T01:04:29.528 INFO:tasks.rados.rados.0.smithi155.stdout:170: expect (ObjNum 15 snap 0 seq_num 15) 2024-04-19T01:04:29.529 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 39 v 50 (ObjNum 49 snap 0 seq_num 8) dirty exists 2024-04-19T01:04:29.529 INFO:tasks.rados.rados.0.smithi155.stdout:168: done (1 left) 2024-04-19T01:04:29.529 INFO:tasks.rados.rados.0.smithi155.stdout:171: rollback oid 47 current snap is 13 2024-04-19T01:04:29.529 INFO:tasks.rados.rados.0.smithi155.stdout:rollback oid 47 to 11 2024-04-19T01:04:29.529 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 37 current snap is 13 2024-04-19T01:04:29.529 INFO:tasks.rados.rados.0.smithi155.stdout:172: seq_num 84 ranges {0=655360,655360=638976,1294336=131072} 2024-04-19T01:04:29.538 INFO:tasks.rados.rados.0.smithi155.stdout:172: writing smithi15528762-37 from 0 to 655360 tid 1 2024-04-19T01:04:29.542 INFO:tasks.rados.rados.0.smithi155.stdout:172: writing smithi15528762-37 from 655360 to 1294336 tid 2 2024-04-19T01:04:29.544 INFO:tasks.rados.rados.0.smithi155.stdout:172: writing smithi15528762-37 from 1294336 to 1425408 tid 3 2024-04-19T01:04:29.545 INFO:tasks.rados.rados.0.smithi155.stdout:173: snap_create 2024-04-19T01:04:29.545 INFO:tasks.rados.rados.0.smithi155.stdout:171: finishing rollback tid 1 to smithi15528762-47 2024-04-19T01:04:29.545 INFO:tasks.rados.rados.0.smithi155.stdout:171: finishing rollback tid 2 to smithi15528762-47 2024-04-19T01:04:29.545 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 47 v 38 (ObjNum 76 snap 8 seq_num 76) dirty exists 2024-04-19T01:04:29.545 INFO:tasks.rados.rados.0.smithi155.stdout:170: expect (ObjNum 15 snap 0 seq_num 15) 2024-04-19T01:04:29.586 INFO:tasks.rados.rados.0.smithi155.stdout:172: finishing write tid 1 to smithi15528762-37 2024-04-19T01:04:29.586 INFO:tasks.rados.rados.0.smithi155.stdout:172: finishing write tid 2 to smithi15528762-37 2024-04-19T01:04:29.587 INFO:tasks.rados.rados.0.smithi155.stdout:172: finishing write tid 3 to smithi15528762-37 2024-04-19T01:04:29.587 INFO:tasks.rados.rados.0.smithi155.stdout:172: finishing write tid 4 to smithi15528762-37 2024-04-19T01:04:29.588 INFO:tasks.rados.rados.0.smithi155.stdout:172: finishing write tid 5 to smithi15528762-37 2024-04-19T01:04:29.588 INFO:tasks.rados.rados.0.smithi155.stdout:172: oid 37 updating version 0 to 54 2024-04-19T01:04:29.588 INFO:tasks.rados.rados.0.smithi155.stdout:172: oid 37 version 54 is already newer than 52 2024-04-19T01:04:29.588 INFO:tasks.rados.rados.0.smithi155.stdout:172: oid 37 version 54 is already newer than 53 2024-04-19T01:04:29.588 INFO:tasks.rados.rados.0.smithi155.stdout:172: oid 37 version 54 is already newer than 51 2024-04-19T01:04:29.588 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 37 v 54 (ObjNum 84 snap 13 seq_num 84) dirty exists 2024-04-19T01:04:29.588 INFO:tasks.rados.rados.0.smithi155.stdout:172: left oid 37 (ObjNum 84 snap 13 seq_num 84) 2024-04-19T01:04:29.626 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:04:29.628+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:04:29.660 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:04:29.660 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:04:29.660 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-04-19T01:04:29.660 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:04:29.674 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-19T01:04:29.703 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 reset_purged_snaps_last 2024-04-19T01:04:29.764 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:04:29.765+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:04:29.827 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:04:29.827+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:04:29.836 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:04:29.836 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:04:29.836 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-04-19T01:04:29.836 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-04-19T01:04:29.836 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:04:29.850 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-19T01:04:29.861 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 reset_purged_snaps_last 2024-04-19T01:04:29.949 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:04:29.951+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:04:30.017 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:04:30.017 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-04-19T01:04:30.017 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-04-19T01:04:30.017 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-04-19T01:04:30.017 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.017 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4260.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-19T01:04:30.017 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:03:50.620310+0000", 2024-04-19T01:04:30.018 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 39.394845302999997, 2024-04-19T01:04:30.018 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.15782495899999999, 2024-04-19T01:04:30.018 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:30.018 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:30.018 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:30.018 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:30.018 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4260", 2024-04-19T01:04:30.018 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/1127060541", 2024-04-19T01:04:30.018 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1 2024-04-19T01:04:30.018 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.018 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:30.018 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.018 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:30.019 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.620310+0000", 2024-04-19T01:04:30.019 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.019 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.019 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.019 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:30.019 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.620310+0000", 2024-04-19T01:04:30.019 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.019 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.019 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.019 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:30.019 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.620313+0000", 2024-04-19T01:04:30.019 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.757e-06 2024-04-19T01:04:30.019 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.020 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.020 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:30.020 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.620327+0000", 2024-04-19T01:04:30.020 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4029000000000001e-05 2024-04-19T01:04:30.020 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.020 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.020 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:30.020 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.620362+0000", 2024-04-19T01:04:30.020 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5201999999999999e-05 2024-04-19T01:04:30.020 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.020 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.020 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:30.021 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.620383+0000", 2024-04-19T01:04:30.021 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1049000000000001e-05 2024-04-19T01:04:30.021 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.021 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.021 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.021 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.620512+0000", 2024-04-19T01:04:30.021 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012820900000000001 2024-04-19T01:04:30.021 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.021 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.021 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for peered", 2024-04-19T01:04:30.021 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.620542+0000", 2024-04-19T01:04:30.021 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0264999999999999e-05 2024-04-19T01:04:30.022 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.022 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.022 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.022 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.774580+0000", 2024-04-19T01:04:30.022 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.15403776799999999 2024-04-19T01:04:30.022 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.022 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.022 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:30.022 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.775052+0000", 2024-04-19T01:04:30.022 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00047277300000000001 2024-04-19T01:04:30.022 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.023 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.023 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 5", 2024-04-19T01:04:30.023 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.775495+0000", 2024-04-19T01:04:30.023 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00044209400000000003 2024-04-19T01:04:30.023 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.023 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.023 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-04-19T01:04:30.023 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.777608+0000", 2024-04-19T01:04:30.023 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0021133649999999999 2024-04-19T01:04:30.023 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.023 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.023 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-04-19T01:04:30.023 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.777957+0000", 2024-04-19T01:04:30.023 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00034961399999999999 2024-04-19T01:04:30.024 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.024 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.024 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:30.024 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.778062+0000", 2024-04-19T01:04:30.024 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000104576 2024-04-19T01:04:30.024 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.024 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.024 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:30.024 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.778135+0000", 2024-04-19T01:04:30.024 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3257999999999996e-05 2024-04-19T01:04:30.024 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.024 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.024 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.024 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.024 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.025 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4260.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-19T01:04:30.025 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:03:51.044749+0000", 2024-04-19T01:04:30.025 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 38.970407082999998, 2024-04-19T01:04:30.025 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.021225126, 2024-04-19T01:04:30.025 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:30.025 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:30.025 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:30.025 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:30.025 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4260", 2024-04-19T01:04:30.025 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/1127060541", 2024-04-19T01:04:30.025 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 114 2024-04-19T01:04:30.026 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.026 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:30.026 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.026 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:30.026 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.044749+0000", 2024-04-19T01:04:30.026 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.027 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.027 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.027 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:30.027 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.044749+0000", 2024-04-19T01:04:30.027 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.027 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.027 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.027 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:30.027 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.044749+0000", 2024-04-19T01:04:30.027 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4000000000000001e-07 2024-04-19T01:04:30.027 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.027 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.027 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:30.027 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.044755+0000", 2024-04-19T01:04:30.028 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6099999999999997e-06 2024-04-19T01:04:30.028 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.028 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.028 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:30.028 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.044765+0000", 2024-04-19T01:04:30.028 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0349e-05 2024-04-19T01:04:30.028 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.028 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.028 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:30.028 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.044773+0000", 2024-04-19T01:04:30.028 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.9500000000000001e-06 2024-04-19T01:04:30.028 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.028 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.028 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.028 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.058797+0000", 2024-04-19T01:04:30.029 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.014023450999999999 2024-04-19T01:04:30.029 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.029 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.029 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:30.029 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.059049+0000", 2024-04-19T01:04:30.029 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00025231500000000003 2024-04-19T01:04:30.029 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.029 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.029 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 5", 2024-04-19T01:04:30.029 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.059446+0000", 2024-04-19T01:04:30.029 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00039740499999999998 2024-04-19T01:04:30.029 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.029 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.029 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-04-19T01:04:30.030 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.063989+0000", 2024-04-19T01:04:30.030 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0045430030000000003 2024-04-19T01:04:30.030 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.030 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.030 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-04-19T01:04:30.030 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.065782+0000", 2024-04-19T01:04:30.030 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0017926999999999999 2024-04-19T01:04:30.030 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.030 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.031 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:30.031 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.065857+0000", 2024-04-19T01:04:30.031 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5375000000000003e-05 2024-04-19T01:04:30.031 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.031 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.031 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:30.031 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.065974+0000", 2024-04-19T01:04:30.031 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000116228 2024-04-19T01:04:30.031 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.032 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.032 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.032 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.032 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.032 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4260.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-19T01:04:30.032 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:03:51.045047+0000", 2024-04-19T01:04:30.032 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 38.970108187999998, 2024-04-19T01:04:30.032 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.021134024000000001, 2024-04-19T01:04:30.032 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:30.032 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:30.032 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:30.032 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:30.032 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4260", 2024-04-19T01:04:30.032 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/1127060541", 2024-04-19T01:04:30.033 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 115 2024-04-19T01:04:30.033 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.033 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:30.033 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.033 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:30.033 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.045047+0000", 2024-04-19T01:04:30.033 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.033 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.033 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.033 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:30.033 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.045047+0000", 2024-04-19T01:04:30.033 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.033 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.033 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.034 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:30.034 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.045049+0000", 2024-04-19T01:04:30.034 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7269999999999999e-06 2024-04-19T01:04:30.034 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.034 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.034 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:30.034 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.045274+0000", 2024-04-19T01:04:30.035 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000225169 2024-04-19T01:04:30.035 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.035 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.035 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:30.035 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.045298+0000", 2024-04-19T01:04:30.035 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4054999999999999e-05 2024-04-19T01:04:30.035 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.036 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.036 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:30.037 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.045316+0000", 2024-04-19T01:04:30.037 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7529000000000001e-05 2024-04-19T01:04:30.037 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.037 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.037 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.037 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.060619+0000", 2024-04-19T01:04:30.037 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.015303192 2024-04-19T01:04:30.037 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.037 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.037 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:30.037 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.060892+0000", 2024-04-19T01:04:30.037 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00027284299999999999 2024-04-19T01:04:30.037 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.038 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.038 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 5", 2024-04-19T01:04:30.038 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.061167+0000", 2024-04-19T01:04:30.038 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000275443 2024-04-19T01:04:30.038 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.038 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.038 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-04-19T01:04:30.038 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.064894+0000", 2024-04-19T01:04:30.038 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0037264030000000001 2024-04-19T01:04:30.038 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.038 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.038 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-04-19T01:04:30.038 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.065987+0000", 2024-04-19T01:04:30.038 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00109366 2024-04-19T01:04:30.039 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.039 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.039 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:30.039 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.066084+0000", 2024-04-19T01:04:30.039 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6866000000000004e-05 2024-04-19T01:04:30.039 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.039 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.039 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:30.039 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.066182+0000", 2024-04-19T01:04:30.039 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7137000000000002e-05 2024-04-19T01:04:30.039 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.039 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.039 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.039 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.039 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.040 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4260.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-19T01:04:30.040 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:03:51.045356+0000", 2024-04-19T01:04:30.040 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 38.969800165000002, 2024-04-19T01:04:30.040 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.021087847999999999, 2024-04-19T01:04:30.040 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:30.040 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:30.040 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:30.040 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:30.040 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4260", 2024-04-19T01:04:30.040 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/1127060541", 2024-04-19T01:04:30.040 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 116 2024-04-19T01:04:30.040 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.040 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:30.040 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.041 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:30.041 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.045356+0000", 2024-04-19T01:04:30.041 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.041 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.041 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.041 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:30.041 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.045356+0000", 2024-04-19T01:04:30.041 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.041 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.041 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.041 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:30.041 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.045357+0000", 2024-04-19T01:04:30.041 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7659999999999999e-06 2024-04-19T01:04:30.041 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.041 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.042 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:30.042 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.045369+0000", 2024-04-19T01:04:30.042 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1469e-05 2024-04-19T01:04:30.042 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.042 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.042 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:30.042 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.045411+0000", 2024-04-19T01:04:30.042 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2246000000000002e-05 2024-04-19T01:04:30.042 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.042 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.042 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:30.042 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.045427+0000", 2024-04-19T01:04:30.042 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5596000000000001e-05 2024-04-19T01:04:30.043 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.043 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.043 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.043 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.062599+0000", 2024-04-19T01:04:30.043 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.017172366000000001 2024-04-19T01:04:30.043 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.043 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.043 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:30.043 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.062821+0000", 2024-04-19T01:04:30.043 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00022237099999999999 2024-04-19T01:04:30.043 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.043 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.043 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 5", 2024-04-19T01:04:30.043 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.063197+0000", 2024-04-19T01:04:30.044 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00037538300000000002 2024-04-19T01:04:30.044 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.044 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.044 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-04-19T01:04:30.044 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.066195+0000", 2024-04-19T01:04:30.044 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0029984529999999999 2024-04-19T01:04:30.044 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.044 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.044 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-04-19T01:04:30.044 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.066291+0000", 2024-04-19T01:04:30.044 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6020999999999994e-05 2024-04-19T01:04:30.044 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.044 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.045 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:30.045 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.066377+0000", 2024-04-19T01:04:30.045 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5582999999999997e-05 2024-04-19T01:04:30.045 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.045 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.045 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:30.045 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.066443+0000", 2024-04-19T01:04:30.045 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6594000000000006e-05 2024-04-19T01:04:30.045 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.045 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.045 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.046 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.046 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.046 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4260.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-19T01:04:30.046 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:03:51.045451+0000", 2024-04-19T01:04:30.046 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 38.969704596, 2024-04-19T01:04:30.046 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.021453949, 2024-04-19T01:04:30.047 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:30.047 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:30.047 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:30.047 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:30.047 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4260", 2024-04-19T01:04:30.047 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/1127060541", 2024-04-19T01:04:30.047 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 117 2024-04-19T01:04:30.047 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.047 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:30.047 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.047 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:30.047 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.045451+0000", 2024-04-19T01:04:30.047 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.047 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.048 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.048 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:30.048 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.045451+0000", 2024-04-19T01:04:30.048 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.048 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.048 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.048 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:30.048 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.045452+0000", 2024-04-19T01:04:30.048 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3400000000000001e-06 2024-04-19T01:04:30.048 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.048 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.048 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:30.048 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.045463+0000", 2024-04-19T01:04:30.048 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0210000000000001e-05 2024-04-19T01:04:30.049 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.049 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.049 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:30.049 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.045487+0000", 2024-04-19T01:04:30.049 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4206000000000002e-05 2024-04-19T01:04:30.049 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.049 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.049 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:30.049 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.045522+0000", 2024-04-19T01:04:30.049 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5407999999999997e-05 2024-04-19T01:04:30.049 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.049 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.049 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.049 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.064995+0000", 2024-04-19T01:04:30.050 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.019472910999999999 2024-04-19T01:04:30.050 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.050 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.050 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:30.050 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.065130+0000", 2024-04-19T01:04:30.050 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013476199999999999 2024-04-19T01:04:30.050 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.050 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.050 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 5", 2024-04-19T01:04:30.050 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.065338+0000", 2024-04-19T01:04:30.050 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00020856 2024-04-19T01:04:30.050 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.050 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.050 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-04-19T01:04:30.051 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.066473+0000", 2024-04-19T01:04:30.051 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011346310000000001 2024-04-19T01:04:30.051 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.051 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.051 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-04-19T01:04:30.051 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.066767+0000", 2024-04-19T01:04:30.051 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000293436 2024-04-19T01:04:30.051 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.051 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.051 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:30.051 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.066849+0000", 2024-04-19T01:04:30.051 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.1966999999999996e-05 2024-04-19T01:04:30.051 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.051 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.052 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:30.052 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.066905+0000", 2024-04-19T01:04:30.052 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6518000000000003e-05 2024-04-19T01:04:30.052 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.052 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.052 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.052 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.052 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.052 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4260.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-19T01:04:30.052 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:03:51.154533+0000", 2024-04-19T01:04:30.052 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 38.860622943999999, 2024-04-19T01:04:30.052 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.022264368, 2024-04-19T01:04:30.052 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:30.052 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:30.052 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:30.053 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:30.053 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4260", 2024-04-19T01:04:30.053 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/1127060541", 2024-04-19T01:04:30.053 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 139 2024-04-19T01:04:30.053 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.053 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:30.053 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.053 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:30.053 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154533+0000", 2024-04-19T01:04:30.053 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.053 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.053 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.053 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:30.053 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154533+0000", 2024-04-19T01:04:30.054 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.054 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.054 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.054 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:30.054 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154533+0000", 2024-04-19T01:04:30.054 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.8999999999999996e-07 2024-04-19T01:04:30.054 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.054 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.054 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:30.054 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154539+0000", 2024-04-19T01:04:30.054 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6999999999999996e-06 2024-04-19T01:04:30.054 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.054 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.054 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:30.055 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154551+0000", 2024-04-19T01:04:30.055 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1565e-05 2024-04-19T01:04:30.055 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.055 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.055 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:30.055 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154558+0000", 2024-04-19T01:04:30.055 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7000000000000008e-06 2024-04-19T01:04:30.055 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.055 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.055 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.055 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.170967+0000", 2024-04-19T01:04:30.055 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.016408323999999998 2024-04-19T01:04:30.055 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.055 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.056 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:30.056 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.171183+0000", 2024-04-19T01:04:30.056 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000216226 2024-04-19T01:04:30.056 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.056 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.056 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 5", 2024-04-19T01:04:30.056 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.171547+0000", 2024-04-19T01:04:30.057 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00036430000000000002 2024-04-19T01:04:30.057 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.057 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.057 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-04-19T01:04:30.057 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.175108+0000", 2024-04-19T01:04:30.058 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.003560325 2024-04-19T01:04:30.058 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.058 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.058 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-04-19T01:04:30.058 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.176560+0000", 2024-04-19T01:04:30.058 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001452577 2024-04-19T01:04:30.058 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.058 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.058 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:30.058 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.176684+0000", 2024-04-19T01:04:30.058 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012410200000000001 2024-04-19T01:04:30.058 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.058 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.058 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:30.059 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.176797+0000", 2024-04-19T01:04:30.059 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000112859 2024-04-19T01:04:30.059 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.059 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.059 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.059 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.059 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.059 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4260.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-19T01:04:30.059 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:03:51.154570+0000", 2024-04-19T01:04:30.059 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 38.860586142000002, 2024-04-19T01:04:30.059 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.022711031999999999, 2024-04-19T01:04:30.059 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:30.059 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:30.059 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:30.060 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:30.060 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4260", 2024-04-19T01:04:30.060 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/1127060541", 2024-04-19T01:04:30.060 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 140 2024-04-19T01:04:30.060 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.060 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:30.060 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.060 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:30.060 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154570+0000", 2024-04-19T01:04:30.060 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.060 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.060 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.060 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:30.060 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154570+0000", 2024-04-19T01:04:30.060 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.061 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.061 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.061 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:30.061 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154570+0000", 2024-04-19T01:04:30.061 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.23e-07 2024-04-19T01:04:30.061 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.061 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.061 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:30.061 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154713+0000", 2024-04-19T01:04:30.061 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014260400000000001 2024-04-19T01:04:30.061 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.061 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.062 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:30.062 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154726+0000", 2024-04-19T01:04:30.062 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3179999999999999e-05 2024-04-19T01:04:30.062 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.062 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.062 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:30.062 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154735+0000", 2024-04-19T01:04:30.062 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.9060000000000008e-06 2024-04-19T01:04:30.062 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.062 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.062 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.062 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.172661+0000", 2024-04-19T01:04:30.062 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.017926464999999999 2024-04-19T01:04:30.062 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.063 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.063 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:30.063 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.172783+0000", 2024-04-19T01:04:30.063 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000121243 2024-04-19T01:04:30.063 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.063 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.063 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 5", 2024-04-19T01:04:30.063 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.172977+0000", 2024-04-19T01:04:30.063 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019476199999999999 2024-04-19T01:04:30.063 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.063 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.063 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-04-19T01:04:30.063 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.176824+0000", 2024-04-19T01:04:30.064 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0038468859999999999 2024-04-19T01:04:30.064 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.064 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.064 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-04-19T01:04:30.064 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.176944+0000", 2024-04-19T01:04:30.064 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000119925 2024-04-19T01:04:30.064 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.064 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.064 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:30.064 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.177103+0000", 2024-04-19T01:04:30.064 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015884499999999999 2024-04-19T01:04:30.064 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.064 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.064 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:30.064 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.177281+0000", 2024-04-19T01:04:30.065 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000177493 2024-04-19T01:04:30.065 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.065 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.065 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.065 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.065 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.065 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4260.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-19T01:04:30.065 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:03:51.154750+0000", 2024-04-19T01:04:30.065 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 38.860405755999999, 2024-04-19T01:04:30.065 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.022800526000000002, 2024-04-19T01:04:30.065 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:30.065 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:30.065 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:30.065 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:30.065 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4260", 2024-04-19T01:04:30.066 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/1127060541", 2024-04-19T01:04:30.066 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 141 2024-04-19T01:04:30.066 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.066 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:30.066 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.066 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:30.066 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154750+0000", 2024-04-19T01:04:30.067 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.067 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.067 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.067 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:30.067 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154750+0000", 2024-04-19T01:04:30.067 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.067 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.067 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.067 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:30.068 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154751+0000", 2024-04-19T01:04:30.068 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.7700000000000003e-07 2024-04-19T01:04:30.068 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.068 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.068 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:30.068 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154757+0000", 2024-04-19T01:04:30.068 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9669999999999998e-06 2024-04-19T01:04:30.068 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.068 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.068 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:30.068 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154767+0000", 2024-04-19T01:04:30.068 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.8349999999999999e-06 2024-04-19T01:04:30.068 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.068 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.069 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:30.069 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.154774+0000", 2024-04-19T01:04:30.069 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7740000000000004e-06 2024-04-19T01:04:30.069 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.069 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.069 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.069 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.175219+0000", 2024-04-19T01:04:30.069 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.020444451999999998 2024-04-19T01:04:30.069 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.069 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.069 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:30.069 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.175431+0000", 2024-04-19T01:04:30.069 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021206900000000001 2024-04-19T01:04:30.069 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.069 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.070 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 5", 2024-04-19T01:04:30.070 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.175804+0000", 2024-04-19T01:04:30.070 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00037343599999999999 2024-04-19T01:04:30.070 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.070 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.070 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-04-19T01:04:30.070 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.177312+0000", 2024-04-19T01:04:30.070 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001507995 2024-04-19T01:04:30.070 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.070 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.070 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-04-19T01:04:30.070 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.177414+0000", 2024-04-19T01:04:30.070 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000101819 2024-04-19T01:04:30.070 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.070 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.071 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:30.071 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.177490+0000", 2024-04-19T01:04:30.071 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5946999999999996e-05 2024-04-19T01:04:30.071 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.071 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.071 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:30.071 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:51.177550+0000", 2024-04-19T01:04:30.071 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0355000000000001e-05 2024-04-19T01:04:30.071 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.071 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.071 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.071 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.071 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.071 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.9s1 39/31 ECSubWrite(tid=32, reqid=client.4604.0:181, at_version=39'16, trim_to=0'0, roll_forward_to=38'14))", 2024-04-19T01:04:30.071 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:09.821971+0000", 2024-04-19T01:04:30.072 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 20.193184798000001, 2024-04-19T01:04:30.072 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.024133847999999999, 2024-04-19T01:04:30.072 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:30.072 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:30.072 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:30.072 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:30.072 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.072 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:30.072 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:09.821971+0000", 2024-04-19T01:04:30.072 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.072 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.072 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.072 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:30.072 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:09.821971+0000", 2024-04-19T01:04:30.073 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.073 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.073 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.073 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:30.073 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:09.821971+0000", 2024-04-19T01:04:30.073 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6000000000000004e-07 2024-04-19T01:04:30.073 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.073 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.073 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:30.073 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:09.822446+0000", 2024-04-19T01:04:30.073 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00047419200000000002 2024-04-19T01:04:30.073 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.073 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.073 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:30.073 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:09.822468+0000", 2024-04-19T01:04:30.073 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2507e-05 2024-04-19T01:04:30.074 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.074 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.074 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:30.074 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:09.822484+0000", 2024-04-19T01:04:30.074 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6368999999999999e-05 2024-04-19T01:04:30.074 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.074 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.074 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.074 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:09.822654+0000", 2024-04-19T01:04:30.074 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000169643 2024-04-19T01:04:30.074 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.074 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.074 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:30.074 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:09.822683+0000", 2024-04-19T01:04:30.075 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9232000000000001e-05 2024-04-19T01:04:30.075 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.075 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.075 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:30.075 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:09.846041+0000", 2024-04-19T01:04:30.075 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.023357902999999999 2024-04-19T01:04:30.075 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.075 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.075 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:30.075 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:09.846105+0000", 2024-04-19T01:04:30.075 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3441999999999993e-05 2024-04-19T01:04:30.075 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.075 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.075 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.075 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.076 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.076 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.9s1 44/31 ECSubWrite(tid=82, reqid=client.4604.0:280, at_version=44'50, trim_to=0'0, roll_forward_to=43'48))", 2024-04-19T01:04:30.076 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:14.883874+0000", 2024-04-19T01:04:30.076 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 15.131281227000001, 2024-04-19T01:04:30.076 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.028027643000000001, 2024-04-19T01:04:30.076 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:30.076 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:30.076 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:30.076 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:30.076 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.076 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:30.076 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.883874+0000", 2024-04-19T01:04:30.076 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.076 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.077 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.077 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:30.077 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.883874+0000", 2024-04-19T01:04:30.077 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.078 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.078 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.078 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:30.078 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.883876+0000", 2024-04-19T01:04:30.078 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7400000000000001e-06 2024-04-19T01:04:30.078 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.078 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.078 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:30.078 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.887310+0000", 2024-04-19T01:04:30.078 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0034333889999999998 2024-04-19T01:04:30.078 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.078 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.078 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:30.079 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.887340+0000", 2024-04-19T01:04:30.079 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0602000000000003e-05 2024-04-19T01:04:30.079 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.079 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.079 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:30.079 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.887361+0000", 2024-04-19T01:04:30.079 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1109999999999999e-05 2024-04-19T01:04:30.079 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.079 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.079 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.079 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.887589+0000", 2024-04-19T01:04:30.079 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000228051 2024-04-19T01:04:30.079 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.079 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.080 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:30.080 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.887614+0000", 2024-04-19T01:04:30.080 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4542999999999999e-05 2024-04-19T01:04:30.080 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.080 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.080 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:30.080 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.911865+0000", 2024-04-19T01:04:30.080 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.024250651000000002 2024-04-19T01:04:30.080 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.080 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.080 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:30.080 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.911902+0000", 2024-04-19T01:04:30.080 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7557e-05 2024-04-19T01:04:30.080 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.080 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.081 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.081 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.081 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.081 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs3 48/31 ECSubWrite(tid=76, reqid=client.4604.0:345, at_version=48'22, trim_to=0'0, roll_forward_to=47'19))", 2024-04-19T01:04:30.081 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.537801+0000", 2024-04-19T01:04:30.081 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 11.477354665, 2024-04-19T01:04:30.081 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.028349493, 2024-04-19T01:04:30.081 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:30.081 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:30.081 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:30.081 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:30.081 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.081 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:30.081 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.537801+0000", 2024-04-19T01:04:30.082 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.082 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.082 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.082 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:30.082 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.537801+0000", 2024-04-19T01:04:30.082 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.082 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.082 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.082 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:30.082 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.537803+0000", 2024-04-19T01:04:30.082 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7099999999999999e-06 2024-04-19T01:04:30.082 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.082 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.082 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:30.082 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.538304+0000", 2024-04-19T01:04:30.083 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00050171999999999999 2024-04-19T01:04:30.083 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.083 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.083 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:30.083 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.538333+0000", 2024-04-19T01:04:30.083 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8725999999999999e-05 2024-04-19T01:04:30.083 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.083 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.083 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:30.083 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.538354+0000", 2024-04-19T01:04:30.083 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.035e-05 2024-04-19T01:04:30.083 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.083 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.083 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.083 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.538557+0000", 2024-04-19T01:04:30.084 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000203766 2024-04-19T01:04:30.084 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.084 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.084 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:30.084 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.538579+0000", 2024-04-19T01:04:30.084 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1792000000000001e-05 2024-04-19T01:04:30.084 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.084 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.084 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:30.084 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.566117+0000", 2024-04-19T01:04:30.084 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.027538045000000001 2024-04-19T01:04:30.084 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.084 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.084 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:30.084 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.566150+0000", 2024-04-19T01:04:30.085 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3383999999999997e-05 2024-04-19T01:04:30.085 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.085 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.085 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.085 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.085 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.085 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:360 3.8s0 3:11aca45f:::smithi15528762-45:head [append 0~573440] snapc 9=[7] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-04-19T01:04:30.085 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:19.532000+0000", 2024-04-19T01:04:30.085 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.483155752, 2024-04-19T01:04:30.085 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.042676382999999998, 2024-04-19T01:04:30.085 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:30.085 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:30.085 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:30.085 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:30.086 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:04:30.086 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:30.086 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 360 2024-04-19T01:04:30.086 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.086 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:30.086 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.086 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:30.086 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.532000+0000", 2024-04-19T01:04:30.086 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.086 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.086 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.086 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:30.086 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.532000+0000", 2024-04-19T01:04:30.086 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.086 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.087 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.087 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:30.087 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.532003+0000", 2024-04-19T01:04:30.087 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3189999999999999e-06 2024-04-19T01:04:30.087 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.087 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.087 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:30.087 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.533430+0000", 2024-04-19T01:04:30.087 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001427069 2024-04-19T01:04:30.087 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.087 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.087 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:30.087 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.533464+0000", 2024-04-19T01:04:30.087 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3708999999999999e-05 2024-04-19T01:04:30.088 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.088 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.088 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:30.088 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.533485+0000", 2024-04-19T01:04:30.088 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0596000000000001e-05 2024-04-19T01:04:30.088 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.088 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.088 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.088 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.535443+0000", 2024-04-19T01:04:30.089 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0019587290000000002 2024-04-19T01:04:30.089 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.089 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.089 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:30.089 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.535618+0000", 2024-04-19T01:04:30.089 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000174518 2024-04-19T01:04:30.089 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.089 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.089 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:30.089 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.537401+0000", 2024-04-19T01:04:30.089 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0017834350000000001 2024-04-19T01:04:30.089 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.089 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.089 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:30.089 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.550350+0000", 2024-04-19T01:04:30.090 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.012948602 2024-04-19T01:04:30.090 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.090 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.090 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:30.090 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.574521+0000", 2024-04-19T01:04:30.090 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.024171392999999999 2024-04-19T01:04:30.090 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.090 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.090 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:30.090 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.574676+0000", 2024-04-19T01:04:30.090 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015501299999999999 2024-04-19T01:04:30.090 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.090 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.091 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.091 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.091 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.091 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:361 3.8s0 3:11aca45f:::smithi15528762-45:head [append 0~425984] snapc 9=[7] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-04-19T01:04:30.091 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:19.540243+0000", 2024-04-19T01:04:30.091 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.474912385, 2024-04-19T01:04:30.091 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.038864792000000002, 2024-04-19T01:04:30.091 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:30.091 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:30.091 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:30.091 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:30.091 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:04:30.091 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:30.091 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 361 2024-04-19T01:04:30.092 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.092 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:30.092 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.092 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:30.092 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540243+0000", 2024-04-19T01:04:30.092 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.092 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.092 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.092 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:30.092 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540243+0000", 2024-04-19T01:04:30.092 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.092 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.092 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.092 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:30.093 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540247+0000", 2024-04-19T01:04:30.093 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.506e-06 2024-04-19T01:04:30.093 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.093 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.093 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:30.093 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.541191+0000", 2024-04-19T01:04:30.093 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00094451099999999998 2024-04-19T01:04:30.093 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.093 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.093 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:30.093 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.541227+0000", 2024-04-19T01:04:30.093 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6053000000000002e-05 2024-04-19T01:04:30.093 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.093 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.093 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:30.094 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.541250+0000", 2024-04-19T01:04:30.094 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2543000000000001e-05 2024-04-19T01:04:30.094 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.094 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.094 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.094 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.550582+0000", 2024-04-19T01:04:30.094 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0093325990000000004 2024-04-19T01:04:30.094 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.094 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.094 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:30.094 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.550838+0000", 2024-04-19T01:04:30.094 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00025546199999999997 2024-04-19T01:04:30.094 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.094 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.094 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:30.095 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.552905+0000", 2024-04-19T01:04:30.095 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0020670699999999998 2024-04-19T01:04:30.095 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.095 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.095 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:30.095 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.574896+0000", 2024-04-19T01:04:30.095 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.021991301000000001 2024-04-19T01:04:30.095 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.095 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.095 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:30.096 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.578996+0000", 2024-04-19T01:04:30.096 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0040997849999999999 2024-04-19T01:04:30.096 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.096 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.096 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:30.096 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.579108+0000", 2024-04-19T01:04:30.096 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000111962 2024-04-19T01:04:30.096 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.096 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.096 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.096 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.096 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.096 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.0s5 49/31 ECSubWrite(tid=39, reqid=osd.1.0:39, at_version=49'35, trim_to=0'0, roll_forward_to=48'33))", 2024-04-19T01:04:30.096 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:19.542188+0000", 2024-04-19T01:04:30.096 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.472967881000001, 2024-04-19T01:04:30.097 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.021965273, 2024-04-19T01:04:30.097 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:30.097 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:30.097 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:30.097 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:30.097 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.097 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:30.097 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.542188+0000", 2024-04-19T01:04:30.097 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.097 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.097 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.097 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:30.097 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.542188+0000", 2024-04-19T01:04:30.097 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.098 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.098 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.098 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:30.098 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.542190+0000", 2024-04-19T01:04:30.098 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.767e-06 2024-04-19T01:04:30.098 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.098 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.098 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:30.099 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.542241+0000", 2024-04-19T01:04:30.099 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.1566000000000001e-05 2024-04-19T01:04:30.099 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.099 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.099 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:30.099 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.542268+0000", 2024-04-19T01:04:30.099 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7146000000000001e-05 2024-04-19T01:04:30.099 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.099 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.099 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:30.099 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.542307+0000", 2024-04-19T01:04:30.099 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9005999999999999e-05 2024-04-19T01:04:30.099 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.099 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.100 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.100 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.550408+0000", 2024-04-19T01:04:30.100 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0081007549999999994 2024-04-19T01:04:30.100 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.100 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.100 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:30.100 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.550427+0000", 2024-04-19T01:04:30.100 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9432999999999998e-05 2024-04-19T01:04:30.100 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.100 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.100 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:30.100 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.564101+0000", 2024-04-19T01:04:30.100 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.013673803 2024-04-19T01:04:30.100 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.101 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.101 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:30.101 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.564153+0000", 2024-04-19T01:04:30.101 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.1796999999999999e-05 2024-04-19T01:04:30.101 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.101 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.101 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.101 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.101 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.101 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:362 3.8s0 3:11aca45f:::smithi15528762-45:head [append 0~344064] snapc 9=[7] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-04-19T01:04:30.101 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:19.547167+0000", 2024-04-19T01:04:30.101 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.467988382, 2024-04-19T01:04:30.101 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.032345455000000002, 2024-04-19T01:04:30.101 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:30.101 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:30.102 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:30.102 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:30.102 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:04:30.102 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:30.102 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 362 2024-04-19T01:04:30.102 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.102 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:30.102 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.102 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:30.102 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547167+0000", 2024-04-19T01:04:30.102 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.102 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.102 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.102 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:30.102 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547167+0000", 2024-04-19T01:04:30.103 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.103 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.103 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.103 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:30.103 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547171+0000", 2024-04-19T01:04:30.103 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.303e-06 2024-04-19T01:04:30.103 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.103 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.103 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:30.103 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547655+0000", 2024-04-19T01:04:30.103 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00048394099999999999 2024-04-19T01:04:30.103 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.103 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.103 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:30.103 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547684+0000", 2024-04-19T01:04:30.104 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9828999999999999e-05 2024-04-19T01:04:30.104 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.104 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.104 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:30.104 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547699+0000", 2024-04-19T01:04:30.104 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4739999999999999e-05 2024-04-19T01:04:30.104 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.104 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.104 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.104 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.564911+0000", 2024-04-19T01:04:30.104 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.017211820999999999 2024-04-19T01:04:30.104 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.104 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.104 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:30.104 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.565200+0000", 2024-04-19T01:04:30.105 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00028915700000000001 2024-04-19T01:04:30.105 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.105 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.105 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:30.105 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.567550+0000", 2024-04-19T01:04:30.105 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0023502869999999999 2024-04-19T01:04:30.105 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.105 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.105 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:30.105 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.574910+0000", 2024-04-19T01:04:30.105 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0073593690000000002 2024-04-19T01:04:30.105 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.105 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.105 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:30.105 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.579412+0000", 2024-04-19T01:04:30.106 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0045022100000000004 2024-04-19T01:04:30.106 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.106 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.106 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:30.106 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.579513+0000", 2024-04-19T01:04:30.106 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010079800000000001 2024-04-19T01:04:30.106 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.106 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.106 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.106 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.106 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.106 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:363 3.8s0 3:11aca45f:::smithi15528762-45:head [setxattr _header (56) in=7b] snapc 9=[7] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-04-19T01:04:30.106 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:19.547729+0000", 2024-04-19T01:04:30.106 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.467426516, 2024-04-19T01:04:30.107 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.033477475, 2024-04-19T01:04:30.107 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:30.107 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:30.107 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:30.107 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:30.107 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:04:30.107 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:30.107 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 363 2024-04-19T01:04:30.107 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.107 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:30.107 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.107 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:30.107 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547729+0000", 2024-04-19T01:04:30.107 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.107 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.108 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.108 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:30.108 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547729+0000", 2024-04-19T01:04:30.108 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.108 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.108 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.108 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:30.108 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547730+0000", 2024-04-19T01:04:30.108 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2160000000000001e-06 2024-04-19T01:04:30.108 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.108 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.108 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:30.108 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547749+0000", 2024-04-19T01:04:30.108 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8312999999999999e-05 2024-04-19T01:04:30.109 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.109 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.109 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:30.109 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547764+0000", 2024-04-19T01:04:30.109 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5279999999999999e-05 2024-04-19T01:04:30.109 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.109 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.109 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:30.109 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547773+0000", 2024-04-19T01:04:30.109 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3300000000000005e-06 2024-04-19T01:04:30.109 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.109 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.109 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.109 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.572682+0000", 2024-04-19T01:04:30.109 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.024908825999999998 2024-04-19T01:04:30.110 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.110 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.110 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:30.110 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.572817+0000", 2024-04-19T01:04:30.110 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000135026 2024-04-19T01:04:30.110 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.110 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.110 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:30.110 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.573699+0000", 2024-04-19T01:04:30.110 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00088153199999999998 2024-04-19T01:04:30.110 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.110 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.110 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:30.110 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.575580+0000", 2024-04-19T01:04:30.110 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0018816589999999999 2024-04-19T01:04:30.111 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.111 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.111 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:30.111 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.581051+0000", 2024-04-19T01:04:30.111 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0054708220000000002 2024-04-19T01:04:30.111 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.111 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.111 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:30.111 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.581207+0000", 2024-04-19T01:04:30.111 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015547099999999999 2024-04-19T01:04:30.111 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.111 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.111 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.111 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.111 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.112 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:364 3.8s0 3:11aca45f:::smithi15528762-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-04-19T01:04:30.112 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:19.547786+0000", 2024-04-19T01:04:30.112 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.467369248000001, 2024-04-19T01:04:30.112 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.037412188999999998, 2024-04-19T01:04:30.112 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:30.112 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:30.112 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-04-19T01:04:30.112 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:30.112 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:04:30.112 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:30.112 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 364 2024-04-19T01:04:30.112 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.112 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:30.112 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.112 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:30.113 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547786+0000", 2024-04-19T01:04:30.113 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.113 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.113 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.113 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:30.113 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547786+0000", 2024-04-19T01:04:30.113 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.113 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.113 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.113 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:30.113 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547787+0000", 2024-04-19T01:04:30.113 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.1399999999999996e-07 2024-04-19T01:04:30.113 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.113 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.113 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:30.114 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547791+0000", 2024-04-19T01:04:30.114 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8299999999999998e-06 2024-04-19T01:04:30.114 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.114 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.114 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:30.114 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547802+0000", 2024-04-19T01:04:30.114 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1372999999999999e-05 2024-04-19T01:04:30.114 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.114 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.114 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:30.114 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547811+0000", 2024-04-19T01:04:30.114 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.0469999999999999e-06 2024-04-19T01:04:30.114 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.114 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.114 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.115 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.575967+0000", 2024-04-19T01:04:30.115 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.028155099999999999 2024-04-19T01:04:30.115 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.115 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.115 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-04-19T01:04:30.115 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.576164+0000", 2024-04-19T01:04:30.115 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019718099999999999 2024-04-19T01:04:30.115 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.115 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.115 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.115 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.581873+0000", 2024-04-19T01:04:30.115 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0057093359999999998 2024-04-19T01:04:30.115 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.115 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.115 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:30.116 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.582089+0000", 2024-04-19T01:04:30.116 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021609099999999999 2024-04-19T01:04:30.116 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.116 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.116 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:30.116 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.585199+0000", 2024-04-19T01:04:30.116 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0031094170000000002 2024-04-19T01:04:30.116 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.116 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.116 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.116 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.116 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.116 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.5s3 59/31 ECSubWrite(tid=66, reqid=client.4604.0:423, at_version=59'51, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:04:30.116 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:29.545169+0000", 2024-04-19T01:04:30.116 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.469986715, 2024-04-19T01:04:30.117 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.030406544000000001, 2024-04-19T01:04:30.117 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:30.117 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:30.117 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:30.117 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:30.117 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.117 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:30.117 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.545169+0000", 2024-04-19T01:04:30.117 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.117 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.117 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.117 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:30.117 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.545169+0000", 2024-04-19T01:04:30.117 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.117 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.117 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.118 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:30.118 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.545169+0000", 2024-04-19T01:04:30.118 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7999999999999996e-07 2024-04-19T01:04:30.118 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.118 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.118 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:30.118 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.545506+0000", 2024-04-19T01:04:30.118 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00033656400000000003 2024-04-19T01:04:30.118 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.118 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.118 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:30.118 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.545535+0000", 2024-04-19T01:04:30.118 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9286e-05 2024-04-19T01:04:30.118 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.119 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.119 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:30.119 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.545558+0000", 2024-04-19T01:04:30.119 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2473e-05 2024-04-19T01:04:30.119 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.119 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.119 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.119 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.545886+0000", 2024-04-19T01:04:30.119 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00032803800000000001 2024-04-19T01:04:30.119 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.119 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.119 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:30.119 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.545912+0000", 2024-04-19T01:04:30.119 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6486e-05 2024-04-19T01:04:30.120 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.120 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.120 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:30.120 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.575499+0000", 2024-04-19T01:04:30.120 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.029586642 2024-04-19T01:04:30.120 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.120 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.120 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:30.120 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.575575+0000", 2024-04-19T01:04:30.120 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6575000000000005e-05 2024-04-19T01:04:30.120 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.120 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.120 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.120 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.121 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.121 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.5s3 59/31 ECSubWrite(tid=67, reqid=client.4604.0:424, at_version=59'52, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:04:30.121 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:29.548409+0000", 2024-04-19T01:04:30.121 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.46674642100000002, 2024-04-19T01:04:30.121 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.027213553000000001, 2024-04-19T01:04:30.121 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:30.121 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:30.121 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:30.121 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:30.121 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.121 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:30.121 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.548409+0000", 2024-04-19T01:04:30.121 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.121 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.122 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.122 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:30.122 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.548409+0000", 2024-04-19T01:04:30.122 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.122 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.122 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.122 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:30.122 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.548411+0000", 2024-04-19T01:04:30.122 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.477e-06 2024-04-19T01:04:30.122 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.122 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.122 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:30.122 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.548794+0000", 2024-04-19T01:04:30.122 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00038350999999999999 2024-04-19T01:04:30.122 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.123 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.123 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:30.123 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.548817+0000", 2024-04-19T01:04:30.123 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2487e-05 2024-04-19T01:04:30.123 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.123 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.123 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:30.123 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.548832+0000", 2024-04-19T01:04:30.123 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5277999999999998e-05 2024-04-19T01:04:30.123 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.123 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.123 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.123 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.561863+0000", 2024-04-19T01:04:30.123 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.013031268 2024-04-19T01:04:30.124 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.124 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.124 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:30.124 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.561891+0000", 2024-04-19T01:04:30.124 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7555999999999998e-05 2024-04-19T01:04:30.124 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.124 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.124 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:30.124 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.575586+0000", 2024-04-19T01:04:30.124 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.013694734 2024-04-19T01:04:30.124 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.124 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.124 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:30.125 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.575623+0000", 2024-04-19T01:04:30.125 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7243000000000002e-05 2024-04-19T01:04:30.125 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.125 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.125 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.125 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.125 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.125 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.5s3 59/31 ECSubWrite(tid=68, reqid=client.4604.0:425, at_version=59'53, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:04:30.125 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:29.555676+0000", 2024-04-19T01:04:30.125 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.45947925099999998, 2024-04-19T01:04:30.125 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.020710566999999999, 2024-04-19T01:04:30.125 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:30.125 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:30.126 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:30.126 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:30.126 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.126 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:30.126 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.555676+0000", 2024-04-19T01:04:30.126 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.126 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.126 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.126 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:30.126 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.555676+0000", 2024-04-19T01:04:30.126 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.126 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.126 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.126 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:30.127 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.555679+0000", 2024-04-19T01:04:30.127 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.717e-06 2024-04-19T01:04:30.127 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.127 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.127 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:30.127 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.555759+0000", 2024-04-19T01:04:30.127 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.9394999999999995e-05 2024-04-19T01:04:30.127 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.127 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.127 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:30.127 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.555787+0000", 2024-04-19T01:04:30.127 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8540000000000001e-05 2024-04-19T01:04:30.127 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.127 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.127 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:30.128 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.555815+0000", 2024-04-19T01:04:30.128 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7628999999999999e-05 2024-04-19T01:04:30.128 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.128 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.128 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.128 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.572188+0000", 2024-04-19T01:04:30.128 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.016372843000000002 2024-04-19T01:04:30.128 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.128 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.128 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:30.128 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.572209+0000", 2024-04-19T01:04:30.128 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1239000000000001e-05 2024-04-19T01:04:30.128 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.128 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.128 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:30.129 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.576333+0000", 2024-04-19T01:04:30.129 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0041239680000000004 2024-04-19T01:04:30.129 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.129 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.129 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:30.129 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.576387+0000", 2024-04-19T01:04:30.129 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4236e-05 2024-04-19T01:04:30.129 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.129 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.129 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.129 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.129 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.129 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:04:30.132 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 reset_purged_snaps_last 2024-04-19T01:04:30.133 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-04-19T01:04:30.133 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:04:30.132+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:04:30.231 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:04:30.230+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:04:30.259 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:04:30.259 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:04:30.259 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-04-19T01:04:30.259 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:04:30.273 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-04-19T01:04:30.332 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:04:30.333+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:04:30.355 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 reset_purged_snaps_last 2024-04-19T01:04:30.391 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:04:30.391 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:04:30.391 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-04-19T01:04:30.391 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-04-19T01:04:30.391 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:04:30.406 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-04-19T01:04:30.433 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:04:30.430+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:04:30.459 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-19T01:04:30.523 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 reset_purged_snaps_last 2024-04-19T01:04:30.525 INFO:tasks.rados.rados.0.smithi155.stdout:170: done (3 left) 2024-04-19T01:04:30.525 INFO:tasks.rados.rados.0.smithi155.stdout:171: done (2 left) 2024-04-19T01:04:30.525 INFO:tasks.rados.rados.0.smithi155.stdout:172: done (1 left) 2024-04-19T01:04:30.525 INFO:tasks.rados.rados.0.smithi155.stdout:173: done (0 left) 2024-04-19T01:04:30.525 INFO:tasks.rados.rados.0.smithi155.stdout:174: setattr oid 42 current snap is 14 2024-04-19T01:04:30.529 INFO:tasks.rados.rados.0.smithi155.stdout:175: read oid 3 snap -1 2024-04-19T01:04:30.529 INFO:tasks.rados.rados.0.smithi155.stdout:175: expect (ObjNum 2 snap 0 seq_num 2) 2024-04-19T01:04:30.529 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 45 current snap is 14 2024-04-19T01:04:30.529 INFO:tasks.rados.rados.0.smithi155.stdout:176: seq_num 85 ranges {1343488=671744,2015232=770048} 2024-04-19T01:04:30.535 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:04:30.533+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:04:30.564 INFO:tasks.rados.rados.0.smithi155.stdout:176: writing smithi15528762-45 from 1343488 to 2015232 tid 1 2024-04-19T01:04:30.577 INFO:tasks.rados.rados.0.smithi155.stdout:176: writing smithi15528762-45 from 2015232 to 2785280 tid 2 2024-04-19T01:04:30.577 INFO:tasks.rados.rados.0.smithi155.stdout:177: rmattr oid 40 current snap is 14 2024-04-19T01:04:30.577 INFO:tasks.rados.rados.0.smithi155.stdout:177: done (3 left) 2024-04-19T01:04:30.578 INFO:tasks.rados.rados.0.smithi155.stdout:178: snap_create 2024-04-19T01:04:30.578 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 42 v 16 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-19T01:04:30.578 INFO:tasks.rados.rados.0.smithi155.stdout:175: expect (ObjNum 2 snap 0 seq_num 2) 2024-04-19T01:04:30.582 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:04:30.582 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-04-19T01:04:30.582 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-04-19T01:04:30.582 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-04-19T01:04:30.582 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.583 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:173 3.3s0 3:c34529b5:::smithi15528762-31:head [append 0~507904] snapc 5=[5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-19T01:04:30.583 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:08.816172+0000", 2024-04-19T01:04:30.583 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 21.764709963000001, 2024-04-19T01:04:30.583 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.024750307999999999, 2024-04-19T01:04:30.583 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:30.583 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:30.583 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:30.583 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:30.583 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:04:30.583 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:30.583 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 173 2024-04-19T01:04:30.583 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.584 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:30.584 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.584 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:30.584 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816172+0000", 2024-04-19T01:04:30.584 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.584 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.584 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.584 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:30.584 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816172+0000", 2024-04-19T01:04:30.584 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.584 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.584 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.585 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:30.585 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816175+0000", 2024-04-19T01:04:30.585 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9589999999999997e-06 2024-04-19T01:04:30.585 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.585 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.585 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:30.585 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816766+0000", 2024-04-19T01:04:30.585 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00059020999999999998 2024-04-19T01:04:30.585 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.585 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.585 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:30.585 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816802+0000", 2024-04-19T01:04:30.586 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6140999999999998e-05 2024-04-19T01:04:30.586 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.586 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.586 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:30.586 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816820+0000", 2024-04-19T01:04:30.586 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7771999999999999e-05 2024-04-19T01:04:30.586 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.586 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.586 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.586 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816990+0000", 2024-04-19T01:04:30.586 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016993299999999999 2024-04-19T01:04:30.586 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.587 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.587 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op must wait for map", 2024-04-19T01:04:30.587 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.817016+0000", 2024-04-19T01:04:30.587 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6857000000000001e-05 2024-04-19T01:04:30.587 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.587 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.587 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.587 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.821660+0000", 2024-04-19T01:04:30.587 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0046440520000000001 2024-04-19T01:04:30.587 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.587 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.587 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:30.587 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.821835+0000", 2024-04-19T01:04:30.588 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000174112 2024-04-19T01:04:30.588 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.588 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.588 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:30.588 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.824020+0000", 2024-04-19T01:04:30.588 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.002185523 2024-04-19T01:04:30.588 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.589 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.589 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:30.589 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.839423+0000", 2024-04-19T01:04:30.589 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.015403261999999999 2024-04-19T01:04:30.589 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.589 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.589 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:30.589 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.840758+0000", 2024-04-19T01:04:30.589 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001335145 2024-04-19T01:04:30.589 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.589 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.589 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:30.589 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.840922+0000", 2024-04-19T01:04:30.590 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000163342 2024-04-19T01:04:30.590 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.590 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.590 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.590 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.590 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.590 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:174 3.3s0 3:c34529b5:::smithi15528762-31:head [append 0~376832] snapc 5=[5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-19T01:04:30.590 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:08.821669+0000", 2024-04-19T01:04:30.591 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 21.759212715, 2024-04-19T01:04:30.591 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.019779560000000002, 2024-04-19T01:04:30.591 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:30.591 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:30.591 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:30.591 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:30.591 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:04:30.591 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:30.591 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 174 2024-04-19T01:04:30.591 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.592 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:30.592 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.592 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:30.592 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.821669+0000", 2024-04-19T01:04:30.592 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.592 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.592 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.592 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:30.592 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.821669+0000", 2024-04-19T01:04:30.592 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.592 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.593 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.593 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:30.593 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.821671+0000", 2024-04-19T01:04:30.593 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2539999999999999e-06 2024-04-19T01:04:30.593 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.593 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.593 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:30.593 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822078+0000", 2024-04-19T01:04:30.593 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00040720600000000002 2024-04-19T01:04:30.593 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.593 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.593 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:30.594 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822097+0000", 2024-04-19T01:04:30.594 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9011000000000001e-05 2024-04-19T01:04:30.594 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.594 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.594 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:30.594 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822111+0000", 2024-04-19T01:04:30.594 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4188e-05 2024-04-19T01:04:30.594 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.594 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.594 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.594 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.830924+0000", 2024-04-19T01:04:30.594 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0088129609999999994 2024-04-19T01:04:30.595 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.595 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.595 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:30.595 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.831084+0000", 2024-04-19T01:04:30.595 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000159889 2024-04-19T01:04:30.595 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.595 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.595 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:30.595 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.832503+0000", 2024-04-19T01:04:30.595 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001418655 2024-04-19T01:04:30.596 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.596 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.596 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:30.596 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.841099+0000", 2024-04-19T01:04:30.596 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0085957340000000007 2024-04-19T01:04:30.596 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.596 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.596 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:30.596 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.841345+0000", 2024-04-19T01:04:30.596 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00024665800000000002 2024-04-19T01:04:30.596 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.597 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.597 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:30.597 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.841448+0000", 2024-04-19T01:04:30.597 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010300400000000001 2024-04-19T01:04:30.597 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.597 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.597 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.597 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.597 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.597 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:175 3.3s0 3:c34529b5:::smithi15528762-31:head [setxattr _header (56) in=7b] snapc 5=[5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-19T01:04:30.597 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:08.822142+0000", 2024-04-19T01:04:30.597 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 21.758739715000001, 2024-04-19T01:04:30.597 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.020224723999999999, 2024-04-19T01:04:30.597 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:30.598 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:30.598 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:30.598 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:30.598 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:04:30.598 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:30.598 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 175 2024-04-19T01:04:30.598 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.598 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:30.598 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.598 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:30.598 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822142+0000", 2024-04-19T01:04:30.598 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.598 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.598 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.598 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:30.599 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822142+0000", 2024-04-19T01:04:30.599 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.599 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.599 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.599 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:30.599 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822143+0000", 2024-04-19T01:04:30.599 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2699999999999999e-06 2024-04-19T01:04:30.599 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.599 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.599 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:30.600 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822151+0000", 2024-04-19T01:04:30.600 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4760000000000001e-06 2024-04-19T01:04:30.600 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.600 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.600 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:30.600 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822165+0000", 2024-04-19T01:04:30.600 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4138000000000001e-05 2024-04-19T01:04:30.600 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.600 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.600 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:30.600 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822178+0000", 2024-04-19T01:04:30.600 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3185e-05 2024-04-19T01:04:30.601 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.601 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.601 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.601 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.839463+0000", 2024-04-19T01:04:30.601 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.017285038999999999 2024-04-19T01:04:30.601 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.601 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.601 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:30.601 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.839580+0000", 2024-04-19T01:04:30.601 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000117601 2024-04-19T01:04:30.601 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.601 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.601 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:30.602 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.840161+0000", 2024-04-19T01:04:30.602 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00058043199999999997 2024-04-19T01:04:30.602 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.602 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.602 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:30.602 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.841469+0000", 2024-04-19T01:04:30.602 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001308065 2024-04-19T01:04:30.602 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.602 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.602 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:30.602 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.842264+0000", 2024-04-19T01:04:30.602 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00079508400000000005 2024-04-19T01:04:30.602 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.602 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.603 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:30.603 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.842366+0000", 2024-04-19T01:04:30.603 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000102434 2024-04-19T01:04:30.603 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.603 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.603 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.603 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.603 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.603 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:176 3.3s0 3:c34529b5:::smithi15528762-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-19T01:04:30.604 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:08.822193+0000", 2024-04-19T01:04:30.604 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 21.758688500000002, 2024-04-19T01:04:30.604 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.023273202, 2024-04-19T01:04:30.604 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:30.604 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:30.604 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-04-19T01:04:30.604 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:30.604 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:04:30.604 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:30.604 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 176 2024-04-19T01:04:30.604 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.604 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:30.604 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.605 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:30.605 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822193+0000", 2024-04-19T01:04:30.605 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.605 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.605 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.605 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:30.605 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822193+0000", 2024-04-19T01:04:30.605 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.605 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.605 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.605 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:30.605 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822194+0000", 2024-04-19T01:04:30.606 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4200000000000004e-07 2024-04-19T01:04:30.606 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.606 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.606 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:30.606 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822198+0000", 2024-04-19T01:04:30.606 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.206e-06 2024-04-19T01:04:30.606 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.606 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.606 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:30.606 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822210+0000", 2024-04-19T01:04:30.606 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.217e-05 2024-04-19T01:04:30.606 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.606 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.607 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:30.607 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822220+0000", 2024-04-19T01:04:30.607 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0059e-05 2024-04-19T01:04:30.607 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.607 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.607 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.607 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.842930+0000", 2024-04-19T01:04:30.607 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.020709701 2024-04-19T01:04:30.607 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.607 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.609 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:30.609 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.843170+0000", 2024-04-19T01:04:30.610 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00024026700000000001 2024-04-19T01:04:30.610 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.610 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.610 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:30.610 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.845466+0000", 2024-04-19T01:04:30.616 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0022958570000000001 2024-04-19T01:04:30.628 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.628 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.628 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.629 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.629 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.629 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 38/31 ECSubWriteReply(tid=13, last_complete=38'9, committed=1, applied=1))", 2024-04-19T01:04:30.629 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:08.829564+0000", 2024-04-19T01:04:30.629 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 21.751317278999998, 2024-04-19T01:04:30.629 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0094823760000000007, 2024-04-19T01:04:30.629 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:30.629 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:30.629 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:30.629 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:30.629 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.629 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:30.629 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.829564+0000", 2024-04-19T01:04:30.630 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.630 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.630 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.630 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:30.630 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.829564+0000", 2024-04-19T01:04:30.630 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.630 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.630 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.630 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:30.630 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.829566+0000", 2024-04-19T01:04:30.630 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1569999999999998e-06 2024-04-19T01:04:30.630 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.630 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.630 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:30.631 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.829576+0000", 2024-04-19T01:04:30.631 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.005e-05 2024-04-19T01:04:30.631 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.631 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.631 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:30.631 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.829605+0000", 2024-04-19T01:04:30.631 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8501999999999999e-05 2024-04-19T01:04:30.631 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.631 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.631 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:30.631 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.829620+0000", 2024-04-19T01:04:30.631 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5017e-05 2024-04-19T01:04:30.631 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.631 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.631 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.632 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.839023+0000", 2024-04-19T01:04:30.632 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0094034410000000002 2024-04-19T01:04:30.632 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.632 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.632 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:30.632 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.839047+0000", 2024-04-19T01:04:30.632 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3209000000000001e-05 2024-04-19T01:04:30.632 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.632 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.632 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.632 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.632 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.633 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 38/31 ECSubWriteReply(tid=13, last_complete=38'9, committed=1, applied=1))", 2024-04-19T01:04:30.633 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:08.830273+0000", 2024-04-19T01:04:30.633 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 21.750608211999999, 2024-04-19T01:04:30.633 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0088936320000000003, 2024-04-19T01:04:30.633 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:30.633 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:30.633 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:30.633 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:30.633 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.633 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:30.633 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.830273+0000", 2024-04-19T01:04:30.633 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.633 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.633 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.634 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:30.634 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.830273+0000", 2024-04-19T01:04:30.634 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.634 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.634 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.634 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:30.634 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.830275+0000", 2024-04-19T01:04:30.634 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4619999999999999e-06 2024-04-19T01:04:30.634 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.634 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.634 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:30.634 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.830283+0000", 2024-04-19T01:04:30.634 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7959999999999996e-06 2024-04-19T01:04:30.635 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.635 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.635 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:30.635 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.830310+0000", 2024-04-19T01:04:30.635 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7036999999999999e-05 2024-04-19T01:04:30.635 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.635 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.635 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:30.635 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.830324+0000", 2024-04-19T01:04:30.635 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4857e-05 2024-04-19T01:04:30.635 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.635 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.636 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.636 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.839152+0000", 2024-04-19T01:04:30.636 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0088272990000000003 2024-04-19T01:04:30.636 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.636 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.636 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:30.636 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.839167+0000", 2024-04-19T01:04:30.636 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5180999999999999e-05 2024-04-19T01:04:30.636 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.636 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.636 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.636 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.636 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.636 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 38/31 ECSubWriteReply(tid=13, last_complete=38'9, committed=1, applied=1))", 2024-04-19T01:04:30.637 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:08.830538+0000", 2024-04-19T01:04:30.637 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 21.750343949000001, 2024-04-19T01:04:30.637 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0085543189999999995, 2024-04-19T01:04:30.637 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:30.637 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:30.637 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:30.637 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:30.637 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.637 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:30.637 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.830538+0000", 2024-04-19T01:04:30.637 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.638 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.638 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.638 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:30.638 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.830538+0000", 2024-04-19T01:04:30.638 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.638 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.638 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.638 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:30.638 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.830540+0000", 2024-04-19T01:04:30.638 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.03e-06 2024-04-19T01:04:30.638 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.638 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.639 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:30.639 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.830549+0000", 2024-04-19T01:04:30.639 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6630000000000005e-06 2024-04-19T01:04:30.639 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.639 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.639 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:30.639 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.830573+0000", 2024-04-19T01:04:30.639 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3512000000000001e-05 2024-04-19T01:04:30.639 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.639 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.639 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:30.639 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.830592+0000", 2024-04-19T01:04:30.639 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8960000000000001e-05 2024-04-19T01:04:30.639 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.640 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.640 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.640 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.839076+0000", 2024-04-19T01:04:30.640 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0084842389999999993 2024-04-19T01:04:30.640 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.640 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.640 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:30.640 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.839092+0000", 2024-04-19T01:04:30.640 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5914999999999999e-05 2024-04-19T01:04:30.640 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.640 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.640 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.641 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.641 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.641 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 37/31 ECSubWriteReply(tid=13, last_complete=38'9, committed=1, applied=1))", 2024-04-19T01:04:30.641 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:08.831207+0000", 2024-04-19T01:04:30.641 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 21.749674182, 2024-04-19T01:04:30.641 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0079223569999999997, 2024-04-19T01:04:30.641 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:30.641 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:30.641 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:30.641 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:30.641 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.641 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:30.641 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.831207+0000", 2024-04-19T01:04:30.642 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.642 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.642 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.642 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:30.642 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.831207+0000", 2024-04-19T01:04:30.642 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.642 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.642 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.642 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:30.642 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.831209+0000", 2024-04-19T01:04:30.642 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0329999999999998e-06 2024-04-19T01:04:30.642 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.642 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.642 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:30.643 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.831219+0000", 2024-04-19T01:04:30.643 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.8430000000000005e-06 2024-04-19T01:04:30.643 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.643 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.643 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:30.643 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.831242+0000", 2024-04-19T01:04:30.643 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3289999999999999e-05 2024-04-19T01:04:30.643 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.643 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.643 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:30.643 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.831257+0000", 2024-04-19T01:04:30.643 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4851999999999999e-05 2024-04-19T01:04:30.643 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.643 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.644 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.644 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.839115+0000", 2024-04-19T01:04:30.644 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0078572959999999997 2024-04-19T01:04:30.644 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.644 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.644 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:30.644 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.839130+0000", 2024-04-19T01:04:30.644 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5043e-05 2024-04-19T01:04:30.644 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.644 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.644 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.645 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.645 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.645 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:244 3.3s0 3:cbbd4e2f:::smithi15528762-23:2 [read 0~376832,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:30.645 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:13.860072+0000", 2024-04-19T01:04:30.645 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 16.720809623000001, 2024-04-19T01:04:30.645 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0076976969999999999, 2024-04-19T01:04:30.645 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:30.645 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:30.645 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-04-19T01:04:30.645 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:30.645 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:04:30.645 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:30.645 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 244 2024-04-19T01:04:30.645 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.645 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:30.646 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.646 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:30.646 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.860072+0000", 2024-04-19T01:04:30.646 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.646 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.646 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.646 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:30.646 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.860072+0000", 2024-04-19T01:04:30.646 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.646 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.646 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.646 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:30.646 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.860072+0000", 2024-04-19T01:04:30.646 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2099999999999996e-07 2024-04-19T01:04:30.647 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.647 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.647 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:30.647 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.860077+0000", 2024-04-19T01:04:30.647 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2359999999999998e-06 2024-04-19T01:04:30.647 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.647 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.647 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:30.647 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.860088+0000", 2024-04-19T01:04:30.647 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1252e-05 2024-04-19T01:04:30.647 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.647 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.648 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:30.648 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.860095+0000", 2024-04-19T01:04:30.648 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.1559999999999998e-06 2024-04-19T01:04:30.648 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.648 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.649 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.649 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.861858+0000", 2024-04-19T01:04:30.649 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0017626600000000001 2024-04-19T01:04:30.649 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.649 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.649 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:30.649 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.861991+0000", 2024-04-19T01:04:30.649 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013294399999999999 2024-04-19T01:04:30.650 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.650 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.650 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:30.650 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.867770+0000", 2024-04-19T01:04:30.650 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0057788279999999997 2024-04-19T01:04:30.650 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.650 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.650 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.650 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.650 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.650 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:343 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [copy-from ver 14] snapc 9=[8,7] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:04:30.650 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.511298+0000", 2024-04-19T01:04:30.651 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 12.069583502, 2024-04-19T01:04:30.651 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.043267551000000001, 2024-04-19T01:04:30.651 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:30.651 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:30.651 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:30.651 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:30.651 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:04:30.651 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:30.651 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 343 2024-04-19T01:04:30.651 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.651 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:30.651 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.652 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:30.652 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511298+0000", 2024-04-19T01:04:30.652 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.652 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.652 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.652 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:30.652 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511298+0000", 2024-04-19T01:04:30.652 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.652 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.652 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.652 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:30.652 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511302+0000", 2024-04-19T01:04:30.652 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1030000000000004e-06 2024-04-19T01:04:30.653 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.653 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.653 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:30.653 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511319+0000", 2024-04-19T01:04:30.653 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7070000000000001e-05 2024-04-19T01:04:30.653 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.653 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.653 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:30.653 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511357+0000", 2024-04-19T01:04:30.653 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8056000000000003e-05 2024-04-19T01:04:30.653 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.653 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.653 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:30.653 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511390+0000", 2024-04-19T01:04:30.654 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3213000000000002e-05 2024-04-19T01:04:30.654 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.654 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.654 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.654 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.516539+0000", 2024-04-19T01:04:30.654 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0051490729999999997 2024-04-19T01:04:30.654 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.654 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.655 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:30.655 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.516748+0000", 2024-04-19T01:04:30.655 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000208099 2024-04-19T01:04:30.655 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.655 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.655 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:30.655 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.541650+0000", 2024-04-19T01:04:30.655 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.024902837000000001 2024-04-19T01:04:30.655 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.655 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.655 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:30.655 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.551197+0000", 2024-04-19T01:04:30.655 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0095463240000000001 2024-04-19T01:04:30.655 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.656 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.656 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:30.656 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.554383+0000", 2024-04-19T01:04:30.656 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0031863830000000001 2024-04-19T01:04:30.656 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.656 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.656 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:30.656 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.554566+0000", 2024-04-19T01:04:30.656 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00018239299999999999 2024-04-19T01:04:30.656 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.656 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.656 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.656 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.657 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.657 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:344 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:04:30.657 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.511452+0000", 2024-04-19T01:04:30.657 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 12.069429252000001, 2024-04-19T01:04:30.657 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.038223957000000003, 2024-04-19T01:04:30.657 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:30.657 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:30.657 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-04-19T01:04:30.657 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:30.667 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:04:30.667 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:30.667 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 344 2024-04-19T01:04:30.667 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.668 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:30.668 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.668 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:30.668 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511452+0000", 2024-04-19T01:04:30.668 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.668 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.668 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.668 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:30.668 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511452+0000", 2024-04-19T01:04:30.668 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.668 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.668 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.668 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:30.669 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511454+0000", 2024-04-19T01:04:30.669 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.39e-06 2024-04-19T01:04:30.669 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.669 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.669 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:30.669 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511460+0000", 2024-04-19T01:04:30.669 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.46e-06 2024-04-19T01:04:30.669 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.669 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.669 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:30.669 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511476+0000", 2024-04-19T01:04:30.669 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6180000000000001e-05 2024-04-19T01:04:30.669 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.670 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.670 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:30.670 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511487+0000", 2024-04-19T01:04:30.670 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.049e-05 2024-04-19T01:04:30.670 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.670 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.670 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.670 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.517203+0000", 2024-04-19T01:04:30.670 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00571595 2024-04-19T01:04:30.670 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.670 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.671 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for blocked object", 2024-04-19T01:04:30.671 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.517341+0000", 2024-04-19T01:04:30.671 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013835099999999999 2024-04-19T01:04:30.671 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.671 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.671 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.671 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.549188+0000", 2024-04-19T01:04:30.671 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.031846589000000002 2024-04-19T01:04:30.671 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.671 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.671 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:30.671 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.549399+0000", 2024-04-19T01:04:30.672 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021149600000000001 2024-04-19T01:04:30.672 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.672 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.672 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:30.672 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.549676+0000", 2024-04-19T01:04:30.672 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000277051 2024-04-19T01:04:30.672 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.672 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.672 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.672 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.672 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.672 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.es6 48/31 ECSubWrite(tid=114, reqid=osd.13.0:113, at_version=0'0, trim_to=0'0, roll_forward_to=48'23))", 2024-04-19T01:04:30.672 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.538316+0000", 2024-04-19T01:04:30.672 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 12.042565602, 2024-04-19T01:04:30.672 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.013029064, 2024-04-19T01:04:30.673 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:30.673 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:30.673 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:30.673 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:30.673 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.673 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:30.673 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.538316+0000", 2024-04-19T01:04:30.673 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.673 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.673 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.673 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:30.673 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.538316+0000", 2024-04-19T01:04:30.673 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.674 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.674 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.674 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:30.674 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.538317+0000", 2024-04-19T01:04:30.674 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6199999999999997e-07 2024-04-19T01:04:30.674 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.674 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.674 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:30.674 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.538335+0000", 2024-04-19T01:04:30.674 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.791e-05 2024-04-19T01:04:30.674 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.674 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.674 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:30.675 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.538349+0000", 2024-04-19T01:04:30.675 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4653e-05 2024-04-19T01:04:30.675 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.675 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.675 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:30.675 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.538362+0000", 2024-04-19T01:04:30.675 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2651e-05 2024-04-19T01:04:30.675 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.675 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.675 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.675 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.538494+0000", 2024-04-19T01:04:30.675 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000132154 2024-04-19T01:04:30.675 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.676 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.676 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:30.676 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.538518+0000", 2024-04-19T01:04:30.676 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4372999999999999e-05 2024-04-19T01:04:30.676 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.676 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.676 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:30.676 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.551270+0000", 2024-04-19T01:04:30.676 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.012751383 2024-04-19T01:04:30.676 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.676 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.676 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:30.676 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.551345+0000", 2024-04-19T01:04:30.677 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5178000000000004e-05 2024-04-19T01:04:30.677 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.677 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.677 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.677 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.677 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.677 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.8s3 49/31 ECSubWrite(tid=201, reqid=osd.11.0:201, at_version=49'24, trim_to=0'0, roll_forward_to=48'20))", 2024-04-19T01:04:30.677 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:19.525178+0000", 2024-04-19T01:04:30.677 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 11.055703233999999, 2024-04-19T01:04:30.677 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0067132609999999999, 2024-04-19T01:04:30.677 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:30.678 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:30.678 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:30.678 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:30.678 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.678 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:30.678 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.525178+0000", 2024-04-19T01:04:30.678 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.678 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.678 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.678 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:30.678 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.525178+0000", 2024-04-19T01:04:30.678 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.678 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.678 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.679 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:30.679 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.525180+0000", 2024-04-19T01:04:30.679 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3880000000000001e-06 2024-04-19T01:04:30.679 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.679 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.679 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:30.679 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.525219+0000", 2024-04-19T01:04:30.679 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9802000000000003e-05 2024-04-19T01:04:30.679 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.679 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.679 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:30.679 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.525251+0000", 2024-04-19T01:04:30.679 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1248000000000003e-05 2024-04-19T01:04:30.679 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.680 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.680 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:30.680 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.525269+0000", 2024-04-19T01:04:30.680 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8127000000000001e-05 2024-04-19T01:04:30.680 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.680 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.680 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.680 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.525413+0000", 2024-04-19T01:04:30.680 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014390099999999999 2024-04-19T01:04:30.680 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.680 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.680 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:30.681 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.525443+0000", 2024-04-19T01:04:30.681 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9884999999999999e-05 2024-04-19T01:04:30.681 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.681 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.681 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:30.681 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.531807+0000", 2024-04-19T01:04:30.681 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0063646629999999996 2024-04-19T01:04:30.681 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.681 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.681 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:30.682 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.531892+0000", 2024-04-19T01:04:30.682 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4246999999999995e-05 2024-04-19T01:04:30.682 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.682 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.682 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.682 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.682 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.682 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.4s4 49/31 ECSubWrite(tid=54, reqid=osd.10.0:54, at_version=49'18, trim_to=0'0, roll_forward_to=48'16))", 2024-04-19T01:04:30.682 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:19.527794+0000", 2024-04-19T01:04:30.682 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 11.05308767, 2024-04-19T01:04:30.683 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0095986419999999992, 2024-04-19T01:04:30.683 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:30.683 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:30.683 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:30.683 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:30.683 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.683 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:30.683 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.527794+0000", 2024-04-19T01:04:30.683 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.683 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.683 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.683 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:30.684 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.527794+0000", 2024-04-19T01:04:30.684 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.695 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.696 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.696 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:30.696 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.527794+0000", 2024-04-19T01:04:30.696 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2399999999999998e-07 2024-04-19T01:04:30.696 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.696 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.696 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:30.696 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.527849+0000", 2024-04-19T01:04:30.696 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4792e-05 2024-04-19T01:04:30.696 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.696 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.696 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:30.696 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.527881+0000", 2024-04-19T01:04:30.696 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1609999999999997e-05 2024-04-19T01:04:30.697 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.697 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.697 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:30.697 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.527904+0000", 2024-04-19T01:04:30.697 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3215000000000001e-05 2024-04-19T01:04:30.697 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.697 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.697 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.697 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.528098+0000", 2024-04-19T01:04:30.697 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000194317 2024-04-19T01:04:30.697 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.697 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.697 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:30.697 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.528126+0000", 2024-04-19T01:04:30.697 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7192999999999999e-05 2024-04-19T01:04:30.698 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.698 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.698 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:30.698 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.537189+0000", 2024-04-19T01:04:30.698 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0090629879999999993 2024-04-19T01:04:30.698 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.698 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.698 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:30.698 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.537392+0000", 2024-04-19T01:04:30.698 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00020390299999999999 2024-04-19T01:04:30.698 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.698 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.698 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.698 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.699 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.699 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.8s3 49/31 ECSubWrite(tid=203, reqid=client.4604.0:360, at_version=49'25, trim_to=0'0, roll_forward_to=49'24))", 2024-04-19T01:04:30.699 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:19.539927+0000", 2024-04-19T01:04:30.699 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 11.040954069, 2024-04-19T01:04:30.699 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.027280209, 2024-04-19T01:04:30.699 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:30.699 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:30.699 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:30.699 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:30.699 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.699 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:30.699 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.539927+0000", 2024-04-19T01:04:30.699 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.699 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.699 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.700 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:30.700 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.539927+0000", 2024-04-19T01:04:30.700 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.700 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.700 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.700 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:30.700 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.539929+0000", 2024-04-19T01:04:30.700 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3829999999999999e-06 2024-04-19T01:04:30.700 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.701 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.701 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:30.701 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540190+0000", 2024-04-19T01:04:30.701 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00026120499999999998 2024-04-19T01:04:30.701 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.701 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.701 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:30.701 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540216+0000", 2024-04-19T01:04:30.701 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5573000000000001e-05 2024-04-19T01:04:30.701 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.701 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.702 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:30.702 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540242+0000", 2024-04-19T01:04:30.702 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5939999999999999e-05 2024-04-19T01:04:30.702 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.702 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.702 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.702 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540387+0000", 2024-04-19T01:04:30.702 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000145983 2024-04-19T01:04:30.702 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.702 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.702 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:30.702 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540412+0000", 2024-04-19T01:04:30.702 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.442e-05 2024-04-19T01:04:30.703 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.703 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.703 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:30.703 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.567114+0000", 2024-04-19T01:04:30.703 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.026701643000000001 2024-04-19T01:04:30.703 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.703 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.703 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:30.703 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.567208+0000", 2024-04-19T01:04:30.703 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4061999999999995e-05 2024-04-19T01:04:30.703 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.703 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.703 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.703 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.703 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.704 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.8s3 49/31 ECSubWrite(tid=204, reqid=client.4604.0:361, at_version=49'26, trim_to=0'0, roll_forward_to=49'24))", 2024-04-19T01:04:30.704 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:19.553252+0000", 2024-04-19T01:04:30.704 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 11.027629150999999, 2024-04-19T01:04:30.704 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.014909534, 2024-04-19T01:04:30.704 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:30.704 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:30.704 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:30.704 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:30.704 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.704 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:30.704 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.553252+0000", 2024-04-19T01:04:30.704 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.704 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.705 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.705 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:30.705 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.553252+0000", 2024-04-19T01:04:30.705 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.705 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.705 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.705 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:30.705 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.553254+0000", 2024-04-19T01:04:30.705 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9369999999999998e-06 2024-04-19T01:04:30.705 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.706 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.706 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:30.706 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.553417+0000", 2024-04-19T01:04:30.706 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016249 2024-04-19T01:04:30.706 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.706 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.706 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:30.706 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.553451+0000", 2024-04-19T01:04:30.706 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4449000000000002e-05 2024-04-19T01:04:30.706 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.706 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.706 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:30.707 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.553472+0000", 2024-04-19T01:04:30.707 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1118999999999999e-05 2024-04-19T01:04:30.707 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.707 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.707 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.707 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.554750+0000", 2024-04-19T01:04:30.707 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0012775219999999999 2024-04-19T01:04:30.707 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.707 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.707 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:30.707 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.554775+0000", 2024-04-19T01:04:30.707 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4712999999999999e-05 2024-04-19T01:04:30.707 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.708 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.708 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:30.708 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.568097+0000", 2024-04-19T01:04:30.708 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.013321995999999999 2024-04-19T01:04:30.708 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.708 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.708 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:30.708 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.568162+0000", 2024-04-19T01:04:30.708 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5308000000000006e-05 2024-04-19T01:04:30.708 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.708 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.708 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.708 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.708 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.709 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.8s3 49/31 ECSubWrite(tid=205, reqid=client.4604.0:362, at_version=49'27, trim_to=0'0, roll_forward_to=49'24))", 2024-04-19T01:04:30.709 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:19.567868+0000", 2024-04-19T01:04:30.709 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 11.013013001999999, 2024-04-19T01:04:30.709 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0085580499999999993, 2024-04-19T01:04:30.709 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:30.709 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:30.709 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:30.709 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:30.709 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.709 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:30.709 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.567868+0000", 2024-04-19T01:04:30.709 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.709 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.709 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.709 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:30.710 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.567868+0000", 2024-04-19T01:04:30.710 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.710 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.710 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.710 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:30.710 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.567870+0000", 2024-04-19T01:04:30.710 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9800000000000001e-06 2024-04-19T01:04:30.710 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.710 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.710 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:30.710 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.568053+0000", 2024-04-19T01:04:30.710 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00018211900000000001 2024-04-19T01:04:30.710 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.711 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.711 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:30.711 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.568080+0000", 2024-04-19T01:04:30.711 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7871000000000001e-05 2024-04-19T01:04:30.711 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.711 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.711 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:30.711 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.568101+0000", 2024-04-19T01:04:30.711 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0460000000000001e-05 2024-04-19T01:04:30.711 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.712 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.712 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.712 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.568252+0000", 2024-04-19T01:04:30.712 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015132500000000001 2024-04-19T01:04:30.712 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.712 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.712 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:30.712 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.568283+0000", 2024-04-19T01:04:30.712 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0582e-05 2024-04-19T01:04:30.712 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.712 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.712 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:30.713 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.576362+0000", 2024-04-19T01:04:30.713 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0080795950000000002 2024-04-19T01:04:30.713 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.713 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.713 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:30.713 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.576427+0000", 2024-04-19T01:04:30.713 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4117999999999998e-05 2024-04-19T01:04:30.713 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.713 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.713 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.713 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.713 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.713 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:418 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [read 0~262144,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e59)", 2024-04-19T01:04:30.713 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:29.527375+0000", 2024-04-19T01:04:30.713 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 1.053506764, 2024-04-19T01:04:30.714 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.012269832, 2024-04-19T01:04:30.714 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:30.714 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:30.714 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-04-19T01:04:30.714 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:30.714 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:04:30.714 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:30.714 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 418 2024-04-19T01:04:30.714 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.714 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:30.714 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.714 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:30.714 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527375+0000", 2024-04-19T01:04:30.714 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.715 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.715 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.715 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:30.715 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527375+0000", 2024-04-19T01:04:30.715 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.715 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.715 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.715 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:30.715 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527378+0000", 2024-04-19T01:04:30.715 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.92e-06 2024-04-19T01:04:30.715 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.715 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.715 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:30.715 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527391+0000", 2024-04-19T01:04:30.715 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3446e-05 2024-04-19T01:04:30.716 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.716 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.716 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:30.716 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527424+0000", 2024-04-19T01:04:30.716 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3386999999999997e-05 2024-04-19T01:04:30.716 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.716 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.716 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:30.717 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527450+0000", 2024-04-19T01:04:30.717 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5406e-05 2024-04-19T01:04:30.717 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.717 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.717 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.717 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527625+0000", 2024-04-19T01:04:30.717 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017486699999999999 2024-04-19T01:04:30.717 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.717 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.717 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op must wait for map", 2024-04-19T01:04:30.717 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527647+0000", 2024-04-19T01:04:30.717 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2592999999999999e-05 2024-04-19T01:04:30.717 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.717 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.717 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.718 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.533053+0000", 2024-04-19T01:04:30.718 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0054056060000000003 2024-04-19T01:04:30.718 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.718 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.718 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:30.718 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.533316+0000", 2024-04-19T01:04:30.718 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00026325200000000001 2024-04-19T01:04:30.718 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.718 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.718 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:30.718 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.539645+0000", 2024-04-19T01:04:30.718 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0063283549999999999 2024-04-19T01:04:30.718 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.718 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.719 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.719 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.719 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.719 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:419 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [read 0~262144,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e59)", 2024-04-19T01:04:30.719 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:29.527509+0000", 2024-04-19T01:04:30.719 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 1.0533720529999999, 2024-04-19T01:04:30.719 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.015116167, 2024-04-19T01:04:30.719 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:30.719 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:30.719 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-04-19T01:04:30.719 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:30.719 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:04:30.719 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:30.719 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 419 2024-04-19T01:04:30.720 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.720 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:30.720 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.720 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:30.720 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527509+0000", 2024-04-19T01:04:30.721 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.721 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.721 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.721 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:30.721 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527509+0000", 2024-04-19T01:04:30.722 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.722 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.722 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.722 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:30.722 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527511+0000", 2024-04-19T01:04:30.722 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4869999999999999e-06 2024-04-19T01:04:30.722 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.722 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.722 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:30.722 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527520+0000", 2024-04-19T01:04:30.722 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3689999999999996e-06 2024-04-19T01:04:30.722 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.722 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.722 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:30.723 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527552+0000", 2024-04-19T01:04:30.723 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1316999999999998e-05 2024-04-19T01:04:30.723 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.723 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.723 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:30.723 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527566+0000", 2024-04-19T01:04:30.723 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4893e-05 2024-04-19T01:04:30.727 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.727 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.727 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.727 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527787+0000", 2024-04-19T01:04:30.727 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00022044999999999999 2024-04-19T01:04:30.727 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.727 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.727 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:04:30.727 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527803+0000", 2024-04-19T01:04:30.728 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6076e-05 2024-04-19T01:04:30.728 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.728 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.728 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.728 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.534575+0000", 2024-04-19T01:04:30.728 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0067724559999999996 2024-04-19T01:04:30.728 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.728 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.728 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:30.728 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.534811+0000", 2024-04-19T01:04:30.728 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00023533000000000001 2024-04-19T01:04:30.729 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.729 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.729 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:30.729 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.542626+0000", 2024-04-19T01:04:30.729 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0078147890000000008 2024-04-19T01:04:30.729 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.729 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.729 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.729 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.729 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.729 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:420 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [read 0~262144,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e59)", 2024-04-19T01:04:30.729 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:29.527610+0000", 2024-04-19T01:04:30.729 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 1.053271488, 2024-04-19T01:04:30.729 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.018555769999999999, 2024-04-19T01:04:30.729 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:30.730 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:30.730 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-04-19T01:04:30.730 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:30.730 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:04:30.730 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:30.730 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 420 2024-04-19T01:04:30.730 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.730 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:30.730 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.730 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:30.730 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527610+0000", 2024-04-19T01:04:30.730 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.730 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.730 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.731 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:30.731 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527610+0000", 2024-04-19T01:04:30.731 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:30.731 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.731 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.731 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:30.731 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527612+0000", 2024-04-19T01:04:30.731 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.967e-06 2024-04-19T01:04:30.731 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.731 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.731 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:30.732 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527642+0000", 2024-04-19T01:04:30.732 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9589000000000001e-05 2024-04-19T01:04:30.732 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.732 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.732 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:30.732 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527664+0000", 2024-04-19T01:04:30.732 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2143000000000001e-05 2024-04-19T01:04:30.732 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.732 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.732 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:30.732 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527679+0000", 2024-04-19T01:04:30.732 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5e-05 2024-04-19T01:04:30.732 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.732 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.733 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.733 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527873+0000", 2024-04-19T01:04:30.733 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000194537 2024-04-19T01:04:30.733 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.733 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.733 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:04:30.733 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527889+0000", 2024-04-19T01:04:30.733 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5603e-05 2024-04-19T01:04:30.733 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.733 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.733 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:30.733 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.535626+0000", 2024-04-19T01:04:30.733 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0077369099999999996 2024-04-19T01:04:30.733 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.733 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.734 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:30.734 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.535845+0000", 2024-04-19T01:04:30.734 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000218827 2024-04-19T01:04:30.734 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:30.734 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:30.734 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:30.734 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.546166+0000", 2024-04-19T01:04:30.734 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.010321194000000001 2024-04-19T01:04:30.734 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.734 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.734 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.734 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:30.734 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:30.734 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:04:30.736 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-04-19T01:04:30.736 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T01:04:30.737 INFO:tasks.rados.rados.0.smithi155.stdout:176: finishing write tid 1 to smithi15528762-45 2024-04-19T01:04:30.737 INFO:tasks.rados.rados.0.smithi155.stdout:176: finishing write tid 2 to smithi15528762-45 2024-04-19T01:04:30.737 INFO:tasks.rados.rados.0.smithi155.stdout:176: finishing write tid 3 to smithi15528762-45 2024-04-19T01:04:30.737 INFO:tasks.rados.rados.0.smithi155.stdout:176: finishing write tid 4 to smithi15528762-45 2024-04-19T01:04:30.737 INFO:tasks.rados.rados.0.smithi155.stdout:176: oid 45 updating version 0 to 43 2024-04-19T01:04:30.737 INFO:tasks.rados.rados.0.smithi155.stdout:176: oid 45 updating version 43 to 44 2024-04-19T01:04:30.737 INFO:tasks.rados.rados.0.smithi155.stdout:176: oid 45 updating version 44 to 45 2024-04-19T01:04:30.737 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 45 v 45 (ObjNum 85 snap 14 seq_num 85) dirty exists 2024-04-19T01:04:30.737 INFO:tasks.rados.rados.0.smithi155.stdout:176: left oid 45 (ObjNum 85 snap 14 seq_num 85) 2024-04-19T01:04:30.737 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-04-19T01:04:30.738 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T01:04:30.738 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-04-19T01:04:30.739 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T01:04:30.740 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-19T01:04:30.740 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-04-19T01:04:30.741 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T01:04:30.742 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:04:30.734+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:04:30.742 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 reset_purged_snaps_last 2024-04-19T01:04:30.743 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-04-19T01:04:30.743 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T01:04:30.743 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-04-19T01:04:30.744 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T01:04:30.744 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-04-19T01:04:30.744 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T01:04:30.752 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-04-19T01:04:30.752 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T01:04:30.768 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2024-04-19T01:04:30.769 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T01:04:30.792 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2024-04-19T01:04:30.792 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T01:04:30.815 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2024-04-19T01:04:30.815 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T01:04:30.837 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:04:30.837+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:04:30.838 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2024-04-19T01:04:30.838 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T01:04:30.859 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2024-04-19T01:04:30.859 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-19T01:04:30.869 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:30.870 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:30.884 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2024-04-19T01:04:30.884 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-19T01:04:30.886 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-19T01:04:30.906 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2024-04-19T01:04:30.907 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-19T01:04:30.914 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 reset_purged_snaps_last 2024-04-19T01:04:30.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:04:30.927+0000 7f1ed5a85640 0 -- 172.21.15.33:0/3770360835 >> [v2:172.21.15.155:6807/741132602,v1:172.21.15.155:6810/741132602] conn(0x7f1ed02252f0 msgr2=0x7f1ed02276e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-19T01:04:30.934 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2024-04-19T01:04:30.938 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:30.938+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:31.024 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:31.025 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:31.039 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:04:31.041+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:04:31.042 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-19T01:04:31.093 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 reset_purged_snaps_last 2024-04-19T01:04:31.140 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:04:31.141+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:04:31.160 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:31.160 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:31.175 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-19T01:04:31.241 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:31.240+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:31.258 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 reset_purged_snaps_last 2024-04-19T01:04:31.296 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:31.296 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:31.311 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-19T01:04:31.342 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:04:31.341+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:04:31.423 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 reset_purged_snaps_last 2024-04-19T01:04:31.443 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:04:31.442+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:04:31.473 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:31.473 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:31.488 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-19T01:04:31.506 INFO:tasks.rados.rados.0.smithi155.stdout:174: done (3 left) 2024-04-19T01:04:31.506 INFO:tasks.rados.rados.0.smithi155.stdout:175: done (2 left) 2024-04-19T01:04:31.506 INFO:tasks.rados.rados.0.smithi155.stdout:176: done (1 left) 2024-04-19T01:04:31.506 INFO:tasks.rados.rados.0.smithi155.stdout:178: done (0 left) 2024-04-19T01:04:31.506 INFO:tasks.rados.rados.0.smithi155.stdout:179: read oid 39 snap -1 2024-04-19T01:04:31.506 INFO:tasks.rados.rados.0.smithi155.stdout:179: expect (ObjNum 0 snap 0 seq_num 0) 2024-04-19T01:04:31.506 INFO:tasks.rados.rados.0.smithi155.stdout:180: copy_from oid 28 from oid 43 current snap is 15 2024-04-19T01:04:31.506 INFO:tasks.rados.rados.0.smithi155.stdout:181: snap_remove snap 13 2024-04-19T01:04:31.544 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:31.542+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:31.547 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-19T01:04:31.645 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:04:31.645+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:04:31.667 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:31.667 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:31.683 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-19T01:04:31.746 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:04:31.747+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:04:31.802 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:31.802 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:31.816 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-19T01:04:31.847 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:04:31.848+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:04:31.948 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:04:31.946+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:04:31.992 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:31.992 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:32.006 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-19T01:04:32.049 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:04:32.048+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:04:32.134 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:32.134 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:32.149 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-04-19T01:04:32.150 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:04:32.150+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:04:32.251 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:04:32.251+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:04:32.274 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:32.274 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:32.289 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-04-19T01:04:32.352 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:04:32.352+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:04:32.453 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:32.456+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:32.460 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:32.461 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:32.475 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-04-19T01:04:32.501 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2024-04-19T01:04:32.508 INFO:tasks.rados.rados.0.smithi155.stdout:181: done (2 left) 2024-04-19T01:04:32.508 INFO:tasks.rados.rados.0.smithi155.stdout:182: rollback oid 29 current snap is 15 2024-04-19T01:04:32.508 INFO:tasks.rados.rados.0.smithi155.stdout:rollback oid 29 to 11 2024-04-19T01:04:32.508 INFO:tasks.rados.rados.0.smithi155.stdout:179: expect (ObjNum 0 snap 0 seq_num 0) 2024-04-19T01:04:32.525 INFO:tasks.rados.rados.0.smithi155.stdout:180: finishing copy_from to smithi15528762-28 2024-04-19T01:04:32.526 INFO:tasks.rados.rados.0.smithi155.stdout:180: got expected ENOENT (src dne) 2024-04-19T01:04:32.526 INFO:tasks.rados.rados.0.smithi155.stdout:180: finishing copy_from racing read to smithi15528762-28 2024-04-19T01:04:32.526 INFO:tasks.rados.rados.0.smithi155.stdout:182: finishing rollback tid 0 to smithi15528762-29 2024-04-19T01:04:32.526 INFO:tasks.rados.rados.0.smithi155.stdout:182: finishing rollback tid 1 to smithi15528762-29 2024-04-19T01:04:32.526 INFO:tasks.rados.rados.0.smithi155.stdout:182: finishing rollback tid 2 to smithi15528762-29 2024-04-19T01:04:32.526 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 29 v 35 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-19T01:04:32.526 INFO:tasks.rados.rados.0.smithi155.stdout:179: done (2 left) 2024-04-19T01:04:32.526 INFO:tasks.rados.rados.0.smithi155.stdout:180: done (1 left) 2024-04-19T01:04:32.526 INFO:tasks.rados.rados.0.smithi155.stdout:182: done (0 left) 2024-04-19T01:04:32.526 INFO:tasks.rados.rados.0.smithi155.stdout:183: copy_from oid 43 from oid 13 current snap is 15 2024-04-19T01:04:32.526 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 2 current snap is 15 2024-04-19T01:04:32.526 INFO:tasks.rados.rados.0.smithi155.stdout:184: seq_num 86 ranges {1474560=688128,2162688=540672,2703360=229376} 2024-04-19T01:04:32.559 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:32.556+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:32.562 INFO:tasks.rados.rados.0.smithi155.stdout:184: writing smithi15528762-2 from 1474560 to 2162688 tid 1 2024-04-19T01:04:32.571 INFO:tasks.rados.rados.0.smithi155.stdout:184: writing smithi15528762-2 from 2162688 to 2703360 tid 2 2024-04-19T01:04:32.574 INFO:tasks.rados.rados.0.smithi155.stdout:184: writing smithi15528762-2 from 2703360 to 2932736 tid 3 2024-04-19T01:04:32.575 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 30 current snap is 15 2024-04-19T01:04:32.575 INFO:tasks.rados.rados.0.smithi155.stdout:183: finishing copy_from racing read to smithi15528762-43 2024-04-19T01:04:32.575 INFO:tasks.rados.rados.0.smithi155.stdout:183: finishing copy_from to smithi15528762-43 2024-04-19T01:04:32.575 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 43 v 4 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-04-19T01:04:32.575 INFO:tasks.rados.rados.0.smithi155.stdout:185: seq_num 87 ranges {491520=704512,1196032=688128,1884160=81920} 2024-04-19T01:04:32.589 INFO:tasks.rados.rados.0.smithi155.stdout:185: writing smithi15528762-30 from 491520 to 1196032 tid 1 2024-04-19T01:04:32.599 INFO:tasks.rados.rados.0.smithi155.stdout:185: writing smithi15528762-30 from 1196032 to 1884160 tid 2 2024-04-19T01:04:32.600 INFO:tasks.rados.rados.0.smithi155.stdout:185: writing smithi15528762-30 from 1884160 to 1966080 tid 3 2024-04-19T01:04:32.600 INFO:tasks.rados.rados.0.smithi155.stdout:183: done (2 left) 2024-04-19T01:04:32.600 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 35 current snap is 15 2024-04-19T01:04:32.600 INFO:tasks.rados.rados.0.smithi155.stdout:186: seq_num 88 ranges {1589248=720896,2310144=442368,2752512=327680} 2024-04-19T01:04:32.625 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:32.625 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:32.625 INFO:tasks.rados.rados.0.smithi155.stdout:186: writing smithi15528762-35 from 1589248 to 2310144 tid 1 2024-04-19T01:04:32.629 INFO:tasks.rados.rados.0.smithi155.stdout:186: writing smithi15528762-35 from 2310144 to 2752512 tid 2 2024-04-19T01:04:32.632 INFO:tasks.rados.rados.0.smithi155.stdout:186: writing smithi15528762-35 from 2752512 to 3080192 tid 3 2024-04-19T01:04:32.632 INFO:tasks.rados.rados.0.smithi155.stdout:187: snap_create 2024-04-19T01:04:32.633 INFO:tasks.rados.rados.0.smithi155.stdout:184: finishing write tid 1 to smithi15528762-2 2024-04-19T01:04:32.633 INFO:tasks.rados.rados.0.smithi155.stdout:184: finishing write tid 2 to smithi15528762-2 2024-04-19T01:04:32.633 INFO:tasks.rados.rados.0.smithi155.stdout:184: finishing write tid 3 to smithi15528762-2 2024-04-19T01:04:32.633 INFO:tasks.rados.rados.0.smithi155.stdout:184: finishing write tid 4 to smithi15528762-2 2024-04-19T01:04:32.633 INFO:tasks.rados.rados.0.smithi155.stdout:184: finishing write tid 5 to smithi15528762-2 2024-04-19T01:04:32.633 INFO:tasks.rados.rados.0.smithi155.stdout:184: oid 2 updating version 0 to 52 2024-04-19T01:04:32.633 INFO:tasks.rados.rados.0.smithi155.stdout:184: oid 2 version 52 is already newer than 49 2024-04-19T01:04:32.633 INFO:tasks.rados.rados.0.smithi155.stdout:184: oid 2 version 52 is already newer than 50 2024-04-19T01:04:32.633 INFO:tasks.rados.rados.0.smithi155.stdout:184: oid 2 version 52 is already newer than 51 2024-04-19T01:04:32.633 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 2 v 52 (ObjNum 86 snap 15 seq_num 86) dirty exists 2024-04-19T01:04:32.633 INFO:tasks.rados.rados.0.smithi155.stdout:184: left oid 2 (ObjNum 86 snap 15 seq_num 86) 2024-04-19T01:04:32.633 INFO:tasks.rados.rados.0.smithi155.stdout:185: finishing write tid 1 to smithi15528762-30 2024-04-19T01:04:32.633 INFO:tasks.rados.rados.0.smithi155.stdout:185: finishing write tid 2 to smithi15528762-30 2024-04-19T01:04:32.633 INFO:tasks.rados.rados.0.smithi155.stdout:185: finishing write tid 3 to smithi15528762-30 2024-04-19T01:04:32.634 INFO:tasks.rados.rados.0.smithi155.stdout:185: finishing write tid 4 to smithi15528762-30 2024-04-19T01:04:32.639 INFO:tasks.rados.rados.0.smithi155.stdout:185: finishing write tid 5 to smithi15528762-30 2024-04-19T01:04:32.639 INFO:tasks.rados.rados.0.smithi155.stdout:185: oid 30 updating version 0 to 48 2024-04-19T01:04:32.639 INFO:tasks.rados.rados.0.smithi155.stdout:185: oid 30 version 48 is already newer than 47 2024-04-19T01:04:32.639 INFO:tasks.rados.rados.0.smithi155.stdout:185: oid 30 updating version 48 to 49 2024-04-19T01:04:32.640 INFO:tasks.rados.rados.0.smithi155.stdout:185: oid 30 updating version 49 to 50 2024-04-19T01:04:32.640 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 30 v 50 (ObjNum 87 snap 15 seq_num 87) dirty exists 2024-04-19T01:04:32.640 INFO:tasks.rados.rados.0.smithi155.stdout:185: left oid 30 (ObjNum 87 snap 15 seq_num 87) 2024-04-19T01:04:32.641 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-19T01:04:32.656 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:04:32.654+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:04:32.663 INFO:tasks.rados.rados.0.smithi155.stdout:186: finishing write tid 1 to smithi15528762-35 2024-04-19T01:04:32.666 INFO:tasks.rados.rados.0.smithi155.stdout:186: finishing write tid 2 to smithi15528762-35 2024-04-19T01:04:32.666 INFO:tasks.rados.rados.0.smithi155.stdout:186: finishing write tid 3 to smithi15528762-35 2024-04-19T01:04:32.666 INFO:tasks.rados.rados.0.smithi155.stdout:186: finishing write tid 4 to smithi15528762-35 2024-04-19T01:04:32.670 INFO:tasks.rados.rados.0.smithi155.stdout:186: finishing write tid 5 to smithi15528762-35 2024-04-19T01:04:32.670 INFO:tasks.rados.rados.0.smithi155.stdout:186: oid 35 updating version 0 to 14 2024-04-19T01:04:32.670 INFO:tasks.rados.rados.0.smithi155.stdout:186: oid 35 updating version 14 to 17 2024-04-19T01:04:32.671 INFO:tasks.rados.rados.0.smithi155.stdout:186: oid 35 version 17 is already newer than 15 2024-04-19T01:04:32.671 INFO:tasks.rados.rados.0.smithi155.stdout:186: oid 35 version 17 is already newer than 16 2024-04-19T01:04:32.671 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 35 v 17 (ObjNum 88 snap 15 seq_num 88) dirty exists 2024-04-19T01:04:32.671 INFO:tasks.rados.rados.0.smithi155.stdout:186: left oid 35 (ObjNum 88 snap 15 seq_num 88) 2024-04-19T01:04:32.758 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:04:32.758+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:04:32.773 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:32.774 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:32.788 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-19T01:04:32.860 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:04:32.858+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:04:32.910 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:32.910 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:32.924 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-19T01:04:32.956 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-19T01:04:32.974 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:04:32.974+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:04:33.041 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:33.041 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:33.055 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-19T01:04:33.062 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:04:33.061+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:04:33.116 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-04-19T01:04:33.116 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T01:04:33.134 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-04-19T01:04:33.134 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T01:04:33.156 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-04-19T01:04:33.156 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T01:04:33.163 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:04:33.162+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:04:33.178 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-04-19T01:04:33.178 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T01:04:33.199 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-04-19T01:04:33.200 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T01:04:33.217 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:33.217 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:33.221 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-04-19T01:04:33.222 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T01:04:33.231 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-19T01:04:33.245 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-04-19T01:04:33.246 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T01:04:33.263 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:04:33.262+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:04:33.270 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-04-19T01:04:33.270 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T01:04:33.289 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2024-04-19T01:04:33.289 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T01:04:33.310 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2024-04-19T01:04:33.310 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T01:04:33.332 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2024-04-19T01:04:33.332 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T01:04:33.353 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2024-04-19T01:04:33.353 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T01:04:33.364 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:04:33.362+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:04:33.370 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2024-04-19T01:04:33.371 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-19T01:04:33.393 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2024-04-19T01:04:33.393 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-19T01:04:33.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:04:33.394+0000 7f844effd640 0 -- 172.21.15.33:0/2833268193 >> [v2:172.21.15.118:6800/506416463,v1:172.21.15.118:6801/506416463] conn(0x7f8450223c30 msgr2=0x7f8450226020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-19T01:04:33.401 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:33.401 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [], 2024-04-19T01:04:33.401 INFO:teuthology.orchestra.run.smithi110.stdout: "complaint_time": 30, 2024-04-19T01:04:33.401 INFO:teuthology.orchestra.run.smithi110.stdout: "num_blocked_ops": 0 2024-04-19T01:04:33.401 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:33.413 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-19T01:04:33.465 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:33.464+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:33.539 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:33.540 INFO:teuthology.orchestra.run.smithi110.stdout: "size": 20, 2024-04-19T01:04:33.540 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 600, 2024-04-19T01:04:33.540 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [ 2024-04-19T01:04:33.540 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.540 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.5s6 43/31 ECSubWrite(tid=43, reqid=client.4604.0:251, at_version=43'29, trim_to=0'0, roll_forward_to=43'27))", 2024-04-19T01:04:33.540 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.985342+0000", 2024-04-19T01:04:33.540 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 19.553409526999999, 2024-04-19T01:04:33.540 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.016484163, 2024-04-19T01:04:33.540 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:33.540 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:33.540 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:33.540 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:33.541 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.541 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:33.541 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.985342+0000", 2024-04-19T01:04:33.541 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:33.541 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.541 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.541 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:33.541 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.985342+0000", 2024-04-19T01:04:33.541 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:33.541 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.541 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.541 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:33.542 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.985344+0000", 2024-04-19T01:04:33.542 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.317e-06 2024-04-19T01:04:33.542 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.542 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.542 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:33.542 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.985967+0000", 2024-04-19T01:04:33.542 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000622851 2024-04-19T01:04:33.542 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.542 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.542 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:33.542 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.986000+0000", 2024-04-19T01:04:33.542 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.2829e-05 2024-04-19T01:04:33.543 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.543 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.543 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:33.543 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.986019+0000", 2024-04-19T01:04:33.543 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9845000000000001e-05 2024-04-19T01:04:33.543 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.543 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.543 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:33.543 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.986179+0000", 2024-04-19T01:04:33.543 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015946600000000001 2024-04-19T01:04:33.543 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.543 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.543 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:33.544 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.986199+0000", 2024-04-19T01:04:33.544 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0353e-05 2024-04-19T01:04:33.544 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.544 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.544 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:33.544 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.001743+0000", 2024-04-19T01:04:33.544 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.015543519 2024-04-19T01:04:33.544 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.544 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.544 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:33.544 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.001826+0000", 2024-04-19T01:04:33.545 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.2983000000000002e-05 2024-04-19T01:04:33.545 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:33.545 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:33.545 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:33.545 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.545 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.545 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs2 43/31 ECSubWrite(tid=47, reqid=client.4604.0:263, at_version=43'8, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:04:33.545 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.995028+0000", 2024-04-19T01:04:33.545 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 19.543723307, 2024-04-19T01:04:33.545 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.038142937000000002, 2024-04-19T01:04:33.545 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:33.545 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:33.545 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:33.545 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:33.546 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.546 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:33.546 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995028+0000", 2024-04-19T01:04:33.546 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:33.547 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.547 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.547 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:33.547 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995028+0000", 2024-04-19T01:04:33.547 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:33.547 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.547 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.547 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:33.547 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995029+0000", 2024-04-19T01:04:33.547 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.47e-07 2024-04-19T01:04:33.547 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.548 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.548 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:33.548 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995397+0000", 2024-04-19T01:04:33.548 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00036862599999999999 2024-04-19T01:04:33.548 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.548 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.549 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:33.549 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995437+0000", 2024-04-19T01:04:33.549 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.9564999999999998e-05 2024-04-19T01:04:33.549 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.549 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.549 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:33.549 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995461+0000", 2024-04-19T01:04:33.549 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4261999999999999e-05 2024-04-19T01:04:33.549 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.549 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.549 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:33.549 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995661+0000", 2024-04-19T01:04:33.549 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00019937500000000001 2024-04-19T01:04:33.549 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.550 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.550 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:33.550 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995689+0000", 2024-04-19T01:04:33.550 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.8282000000000001e-05 2024-04-19T01:04:33.550 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.550 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.550 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:33.550 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.033059+0000", 2024-04-19T01:04:33.550 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.037370288000000002 2024-04-19T01:04:33.550 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.550 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.550 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:33.550 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.033171+0000", 2024-04-19T01:04:33.550 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000111692 2024-04-19T01:04:33.551 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:33.551 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:33.551 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:33.551 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.551 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.551 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs2 43/31 ECSubWrite(tid=48, reqid=client.4604.0:264, at_version=43'9, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:04:33.551 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.005925+0000", 2024-04-19T01:04:33.551 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 19.532826651000001, 2024-04-19T01:04:33.551 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.028007243000000001, 2024-04-19T01:04:33.551 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:33.551 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:33.551 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:33.551 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:33.551 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.551 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:33.552 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.005925+0000", 2024-04-19T01:04:33.552 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:33.552 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.552 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.552 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:33.552 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.005925+0000", 2024-04-19T01:04:33.552 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:33.552 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.552 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.552 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:33.552 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.005926+0000", 2024-04-19T01:04:33.552 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9700000000000002e-06 2024-04-19T01:04:33.552 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.552 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.553 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:33.553 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006164+0000", 2024-04-19T01:04:33.553 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00023784199999999999 2024-04-19T01:04:33.553 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.554 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.554 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:33.554 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006203+0000", 2024-04-19T01:04:33.554 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.9171999999999997e-05 2024-04-19T01:04:33.554 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.554 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.554 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:33.554 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006235+0000", 2024-04-19T01:04:33.555 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.1937999999999998e-05 2024-04-19T01:04:33.555 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.555 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.555 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:33.555 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.015013+0000", 2024-04-19T01:04:33.555 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0087775909999999995 2024-04-19T01:04:33.555 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.555 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.555 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:33.555 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.015043+0000", 2024-04-19T01:04:33.555 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.9739000000000001e-05 2024-04-19T01:04:33.555 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.555 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.556 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:33.556 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.033841+0000", 2024-04-19T01:04:33.556 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.018797861999999999 2024-04-19T01:04:33.556 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.556 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.556 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:33.556 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.033932+0000", 2024-04-19T01:04:33.556 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.1129000000000001e-05 2024-04-19T01:04:33.556 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:33.557 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:33.557 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:33.557 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.557 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.557 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs2 43/31 ECSubWrite(tid=49, reqid=client.4604.0:265, at_version=43'10, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:04:33.557 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.006295+0000", 2024-04-19T01:04:33.557 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 19.532455881000001, 2024-04-19T01:04:33.557 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.02899026, 2024-04-19T01:04:33.557 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:33.557 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:33.557 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:33.557 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:33.557 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.558 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:33.558 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006295+0000", 2024-04-19T01:04:33.558 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:33.558 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.558 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.558 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:33.558 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006295+0000", 2024-04-19T01:04:33.558 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:33.558 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.558 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.558 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:33.558 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006296+0000", 2024-04-19T01:04:33.558 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.1230000000000001e-06 2024-04-19T01:04:33.559 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.559 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.559 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:33.559 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006329+0000", 2024-04-19T01:04:33.559 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.2465000000000002e-05 2024-04-19T01:04:33.559 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.559 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.559 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:33.559 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006362+0000", 2024-04-19T01:04:33.559 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.3118999999999999e-05 2024-04-19T01:04:33.559 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.559 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.559 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:33.559 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006401+0000", 2024-04-19T01:04:33.560 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.9347999999999997e-05 2024-04-19T01:04:33.560 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.560 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.560 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:33.560 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.033284+0000", 2024-04-19T01:04:33.560 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.026882553999999999 2024-04-19T01:04:33.560 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.560 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.560 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:33.560 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.033308+0000", 2024-04-19T01:04:33.560 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4295999999999999e-05 2024-04-19T01:04:33.560 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.561 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.561 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:33.561 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.035213+0000", 2024-04-19T01:04:33.561 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0019049749999999999 2024-04-19T01:04:33.561 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.561 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.561 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:33.561 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.035286+0000", 2024-04-19T01:04:33.561 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.2379999999999995e-05 2024-04-19T01:04:33.561 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:33.562 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:33.562 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:33.562 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.562 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.562 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs2 48/31 ECSubWrite(tid=76, reqid=client.4604.0:345, at_version=48'22, trim_to=0'0, roll_forward_to=47'19))", 2024-04-19T01:04:33.562 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.538229+0000", 2024-04-19T01:04:33.562 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 15.000521674, 2024-04-19T01:04:33.562 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.041621538999999999, 2024-04-19T01:04:33.562 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:33.562 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:33.562 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:33.562 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:33.562 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.563 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:33.563 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538229+0000", 2024-04-19T01:04:33.563 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:33.563 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.563 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.563 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:33.563 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538229+0000", 2024-04-19T01:04:33.563 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:33.563 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.563 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.564 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:33.564 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538232+0000", 2024-04-19T01:04:33.564 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1730000000000002e-06 2024-04-19T01:04:33.564 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.565 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.565 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:33.565 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538714+0000", 2024-04-19T01:04:33.565 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00048223299999999999 2024-04-19T01:04:33.565 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.565 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.565 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:33.565 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538755+0000", 2024-04-19T01:04:33.565 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.1044999999999998e-05 2024-04-19T01:04:33.565 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.565 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.565 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:33.565 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538779+0000", 2024-04-19T01:04:33.566 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3828999999999998e-05 2024-04-19T01:04:33.566 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.566 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.566 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:33.566 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.539007+0000", 2024-04-19T01:04:33.566 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00022818400000000001 2024-04-19T01:04:33.566 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.566 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.566 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:33.567 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.539033+0000", 2024-04-19T01:04:33.567 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6455999999999999e-05 2024-04-19T01:04:33.567 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.567 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.567 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:33.567 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.579777+0000", 2024-04-19T01:04:33.567 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.040743133000000001 2024-04-19T01:04:33.567 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.567 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.567 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:33.567 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.579851+0000", 2024-04-19T01:04:33.567 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.4486000000000005e-05 2024-04-19T01:04:33.567 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:33.567 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:33.568 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:33.568 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.568 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.568 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs2 48/31 ECSubWrite(tid=78, reqid=client.4604.0:346, at_version=48'23, trim_to=0'0, roll_forward_to=48'21))", 2024-04-19T01:04:33.568 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.549034+0000", 2024-04-19T01:04:33.568 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 14.989717631, 2024-04-19T01:04:33.568 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.030907911999999999, 2024-04-19T01:04:33.568 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:33.568 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:33.568 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:33.568 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:33.568 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.569 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:33.569 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.549034+0000", 2024-04-19T01:04:33.569 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:33.569 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.569 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.569 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:33.569 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.549034+0000", 2024-04-19T01:04:33.569 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:33.569 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.569 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.569 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:33.569 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.549035+0000", 2024-04-19T01:04:33.569 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9429999999999999e-06 2024-04-19T01:04:33.569 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.570 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.570 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:33.570 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.549509+0000", 2024-04-19T01:04:33.570 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00047399999999999997 2024-04-19T01:04:33.570 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.570 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.570 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:33.570 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.549541+0000", 2024-04-19T01:04:33.570 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.1843e-05 2024-04-19T01:04:33.570 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.570 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.570 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:33.570 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.549563+0000", 2024-04-19T01:04:33.571 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2045e-05 2024-04-19T01:04:33.571 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.571 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.571 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:33.571 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.558487+0000", 2024-04-19T01:04:33.571 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0089238470000000004 2024-04-19T01:04:33.571 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.571 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.571 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:33.571 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.558508+0000", 2024-04-19T01:04:33.571 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1236000000000001e-05 2024-04-19T01:04:33.571 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.572 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.572 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:33.572 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.579870+0000", 2024-04-19T01:04:33.572 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.021361601000000001 2024-04-19T01:04:33.572 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.572 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.572 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:33.572 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.579941+0000", 2024-04-19T01:04:33.572 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.1396999999999994e-05 2024-04-19T01:04:33.572 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:33.572 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:33.572 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:33.572 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.572 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.573 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs2 48/31 ECSubWrite(tid=79, reqid=client.4604.0:347, at_version=48'24, trim_to=0'0, roll_forward_to=48'21))", 2024-04-19T01:04:33.573 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.555772+0000", 2024-04-19T01:04:33.573 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 14.982978805, 2024-04-19T01:04:33.573 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.024949988999999999, 2024-04-19T01:04:33.573 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:33.573 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:33.573 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:33.573 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:33.573 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.573 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:33.573 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.555772+0000", 2024-04-19T01:04:33.573 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:33.573 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.574 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.574 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:33.574 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.555772+0000", 2024-04-19T01:04:33.574 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:33.574 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.574 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.574 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:33.575 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.555774+0000", 2024-04-19T01:04:33.575 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.734e-06 2024-04-19T01:04:33.575 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.575 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.575 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:33.575 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.555841+0000", 2024-04-19T01:04:33.575 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.6434999999999997e-05 2024-04-19T01:04:33.575 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.575 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.575 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:33.575 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.555877+0000", 2024-04-19T01:04:33.575 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.6544000000000001e-05 2024-04-19T01:04:33.575 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.576 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.576 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:33.576 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.555895+0000", 2024-04-19T01:04:33.576 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7558999999999999e-05 2024-04-19T01:04:33.576 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.576 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.576 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:33.576 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.576874+0000", 2024-04-19T01:04:33.576 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.020979564999999999 2024-04-19T01:04:33.576 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.576 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.576 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:33.576 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.576898+0000", 2024-04-19T01:04:33.576 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3787000000000001e-05 2024-04-19T01:04:33.576 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.577 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.577 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:33.577 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.580616+0000", 2024-04-19T01:04:33.577 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0037177289999999999 2024-04-19T01:04:33.577 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.577 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.577 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:33.577 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.580722+0000", 2024-04-19T01:04:33.577 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000106636 2024-04-19T01:04:33.577 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:33.577 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:33.578 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:33.578 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.578 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.578 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.8s6 48/31 ECSubWrite(tid=199, reqid=client.4604.0:353, at_version=0'0, trim_to=0'0, roll_forward_to=48'20))", 2024-04-19T01:04:33.578 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.557241+0000", 2024-04-19T01:04:33.578 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 14.98151047, 2024-04-19T01:04:33.578 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.020337572000000002, 2024-04-19T01:04:33.578 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:33.578 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:33.578 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:33.578 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:33.578 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.579 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:33.579 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557241+0000", 2024-04-19T01:04:33.579 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:33.579 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.579 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.579 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:33.579 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557241+0000", 2024-04-19T01:04:33.579 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:33.579 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.579 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.579 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:33.579 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557242+0000", 2024-04-19T01:04:33.579 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.4449999999999999e-06 2024-04-19T01:04:33.579 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.580 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.580 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:33.580 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557262+0000", 2024-04-19T01:04:33.580 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0103999999999999e-05 2024-04-19T01:04:33.580 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.580 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.580 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:33.580 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557287+0000", 2024-04-19T01:04:33.580 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4556e-05 2024-04-19T01:04:33.580 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.580 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.580 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:33.580 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557304+0000", 2024-04-19T01:04:33.581 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7592e-05 2024-04-19T01:04:33.581 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.581 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.581 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:33.581 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557935+0000", 2024-04-19T01:04:33.581 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00063064499999999999 2024-04-19T01:04:33.581 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.581 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.581 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:33.581 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557959+0000", 2024-04-19T01:04:33.581 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3736000000000001e-05 2024-04-19T01:04:33.581 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.581 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.581 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:33.582 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.577484+0000", 2024-04-19T01:04:33.582 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.019525415000000001 2024-04-19T01:04:33.582 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.582 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.582 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:33.582 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.577578+0000", 2024-04-19T01:04:33.582 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.4079000000000005e-05 2024-04-19T01:04:33.582 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:33.582 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:33.582 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:33.582 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.582 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.582 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.es2 48/31 ECSubWrite(tid=118, reqid=client.4604.0:357, at_version=48'25, trim_to=0'0, roll_forward_to=48'23))", 2024-04-19T01:04:33.583 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.557362+0000", 2024-04-19T01:04:33.583 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 14.981389383, 2024-04-19T01:04:33.583 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.020223954999999998, 2024-04-19T01:04:33.583 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:33.583 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:33.583 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:33.583 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:33.583 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.583 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:33.583 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557362+0000", 2024-04-19T01:04:33.583 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:33.583 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.583 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.583 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:33.584 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557362+0000", 2024-04-19T01:04:33.584 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:33.584 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.584 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.584 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:33.584 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557363+0000", 2024-04-19T01:04:33.584 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.7499999999999999e-07 2024-04-19T01:04:33.584 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.584 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.584 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:33.584 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557396+0000", 2024-04-19T01:04:33.584 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.3878999999999999e-05 2024-04-19T01:04:33.584 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.584 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.585 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:33.585 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557421+0000", 2024-04-19T01:04:33.585 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4627999999999999e-05 2024-04-19T01:04:33.585 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.585 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.585 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:33.585 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557435+0000", 2024-04-19T01:04:33.585 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.4185000000000001e-05 2024-04-19T01:04:33.586 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.586 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.586 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:33.586 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.558839+0000", 2024-04-19T01:04:33.586 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0014035860000000001 2024-04-19T01:04:33.586 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.586 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.586 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:33.586 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.558863+0000", 2024-04-19T01:04:33.586 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4434e-05 2024-04-19T01:04:33.586 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.586 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.586 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:33.587 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.577527+0000", 2024-04-19T01:04:33.587 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.018663800000000001 2024-04-19T01:04:33.587 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.587 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.587 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:33.587 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.577586+0000", 2024-04-19T01:04:33.587 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.8668000000000001e-05 2024-04-19T01:04:33.587 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:33.587 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:33.587 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:33.587 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.587 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.587 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.es2 48/31 ECSubWrite(tid=120, reqid=client.4604.0:358, at_version=48'26, trim_to=0'0, roll_forward_to=48'23))", 2024-04-19T01:04:33.587 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.558551+0000", 2024-04-19T01:04:33.587 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 14.980200432, 2024-04-19T01:04:33.588 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.019082335999999998, 2024-04-19T01:04:33.588 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:33.588 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:33.588 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:33.588 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:33.588 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.588 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:33.588 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.558551+0000", 2024-04-19T01:04:33.588 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:33.588 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.588 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.588 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:33.588 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.558551+0000", 2024-04-19T01:04:33.588 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:33.588 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.589 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.589 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:33.589 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.558552+0000", 2024-04-19T01:04:33.589 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.4929999999999999e-06 2024-04-19T01:04:33.589 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.589 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.589 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:33.589 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.558588+0000", 2024-04-19T01:04:33.589 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.6125999999999999e-05 2024-04-19T01:04:33.589 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.589 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.589 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:33.589 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.558611+0000", 2024-04-19T01:04:33.589 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2959000000000002e-05 2024-04-19T01:04:33.590 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.590 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.590 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:33.590 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.558633+0000", 2024-04-19T01:04:33.590 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1815e-05 2024-04-19T01:04:33.590 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.590 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.590 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:33.590 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.565266+0000", 2024-04-19T01:04:33.590 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0066330060000000003 2024-04-19T01:04:33.590 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.590 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.590 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:33.590 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.565300+0000", 2024-04-19T01:04:33.591 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.3671999999999999e-05 2024-04-19T01:04:33.591 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.591 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.591 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:33.591 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.577592+0000", 2024-04-19T01:04:33.591 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.012291905000000001 2024-04-19T01:04:33.591 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.591 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.591 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:33.591 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.577633+0000", 2024-04-19T01:04:33.591 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.1359999999999997e-05 2024-04-19T01:04:33.591 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:33.591 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:33.591 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:33.591 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.592 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.592 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs2 48/31 ECSubWrite(tid=80, reqid=client.4604.0:348, at_version=48'25, trim_to=0'0, roll_forward_to=48'21))", 2024-04-19T01:04:33.592 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.561184+0000", 2024-04-19T01:04:33.592 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 14.977567433000001, 2024-04-19T01:04:33.592 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.021208286, 2024-04-19T01:04:33.592 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:33.592 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:33.592 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:33.592 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:33.592 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.592 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:33.592 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.561184+0000", 2024-04-19T01:04:33.592 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:33.592 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.592 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.593 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:33.593 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.561184+0000", 2024-04-19T01:04:33.593 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:33.593 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.593 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.593 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:33.593 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.561185+0000", 2024-04-19T01:04:33.593 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.685e-06 2024-04-19T01:04:33.593 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.593 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.593 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:33.593 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.561230+0000", 2024-04-19T01:04:33.593 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.4867000000000003e-05 2024-04-19T01:04:33.593 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.594 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.594 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:33.594 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.561282+0000", 2024-04-19T01:04:33.594 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.1712999999999998e-05 2024-04-19T01:04:33.594 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.594 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.594 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:33.594 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.561304+0000", 2024-04-19T01:04:33.594 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2209999999999999e-05 2024-04-19T01:04:33.594 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.594 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.594 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:33.594 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.580080+0000", 2024-04-19T01:04:33.594 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.018775475999999999 2024-04-19T01:04:33.595 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.595 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.595 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:33.595 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.580193+0000", 2024-04-19T01:04:33.595 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000113244 2024-04-19T01:04:33.595 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.595 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.595 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:33.595 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.582324+0000", 2024-04-19T01:04:33.596 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0021313790000000001 2024-04-19T01:04:33.596 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.596 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.596 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:33.596 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.582392+0000", 2024-04-19T01:04:33.596 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.7712000000000005e-05 2024-04-19T01:04:33.596 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:33.596 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:33.596 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:33.596 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.596 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.596 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.8s6 49/31 ECSubWrite(tid=203, reqid=client.4604.0:360, at_version=49'25, trim_to=0'0, roll_forward_to=49'24))", 2024-04-19T01:04:33.596 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:19.538676+0000", 2024-04-19T01:04:33.596 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 14.000075415, 2024-04-19T01:04:33.597 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.019513513, 2024-04-19T01:04:33.597 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:33.597 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:33.597 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:33.597 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:33.597 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.597 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:33.597 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.538676+0000", 2024-04-19T01:04:33.597 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:33.597 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.597 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.598 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:33.598 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.538676+0000", 2024-04-19T01:04:33.598 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:33.598 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.598 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.598 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:33.598 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.538678+0000", 2024-04-19T01:04:33.598 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0260000000000002e-06 2024-04-19T01:04:33.598 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.598 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.598 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:33.598 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.538968+0000", 2024-04-19T01:04:33.598 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000290172 2024-04-19T01:04:33.598 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.599 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.599 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:33.599 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.539364+0000", 2024-04-19T01:04:33.599 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00039640399999999999 2024-04-19T01:04:33.599 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.599 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.599 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:33.599 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.539393+0000", 2024-04-19T01:04:33.599 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.8912e-05 2024-04-19T01:04:33.599 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.599 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.599 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:33.599 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.539604+0000", 2024-04-19T01:04:33.599 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00021122900000000001 2024-04-19T01:04:33.599 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.600 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.600 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:33.600 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.539630+0000", 2024-04-19T01:04:33.600 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5151999999999999e-05 2024-04-19T01:04:33.600 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.600 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.600 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:33.600 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.558098+0000", 2024-04-19T01:04:33.600 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.018468232000000001 2024-04-19T01:04:33.600 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.600 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.600 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:33.600 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.558189+0000", 2024-04-19T01:04:33.600 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.1385999999999995e-05 2024-04-19T01:04:33.600 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:33.601 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:33.601 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:33.601 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.601 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.601 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.8s6 49/31 ECSubWrite(tid=204, reqid=client.4604.0:361, at_version=49'26, trim_to=0'0, roll_forward_to=49'24))", 2024-04-19T01:04:33.601 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:19.560727+0000", 2024-04-19T01:04:33.601 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 13.978023788, 2024-04-19T01:04:33.601 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.018075543999999999, 2024-04-19T01:04:33.601 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:33.601 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:33.601 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:33.601 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:33.601 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.601 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:33.602 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.560727+0000", 2024-04-19T01:04:33.602 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:33.602 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.602 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.602 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:33.602 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.560727+0000", 2024-04-19T01:04:33.602 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:33.602 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.602 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.602 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:33.602 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.560730+0000", 2024-04-19T01:04:33.602 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.9299999999999999e-06 2024-04-19T01:04:33.603 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.603 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.603 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:33.603 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.560956+0000", 2024-04-19T01:04:33.603 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00022553100000000001 2024-04-19T01:04:33.603 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.603 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.603 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:33.603 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.560989+0000", 2024-04-19T01:04:33.603 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.3646000000000003e-05 2024-04-19T01:04:33.603 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.603 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.603 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:33.604 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.561012+0000", 2024-04-19T01:04:33.604 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2742e-05 2024-04-19T01:04:33.604 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.604 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.604 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:33.604 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.561272+0000", 2024-04-19T01:04:33.604 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000260067 2024-04-19T01:04:33.604 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.604 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.604 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:33.604 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.561298+0000", 2024-04-19T01:04:33.604 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5862000000000001e-05 2024-04-19T01:04:33.605 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.605 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.605 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:33.605 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.578729+0000", 2024-04-19T01:04:33.605 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.017430977 2024-04-19T01:04:33.605 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.605 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.605 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:33.605 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.578803+0000", 2024-04-19T01:04:33.605 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.3788999999999994e-05 2024-04-19T01:04:33.606 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:33.606 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:33.606 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:33.606 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.606 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.606 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.8s6 49/31 ECSubWrite(tid=205, reqid=client.4604.0:362, at_version=49'27, trim_to=0'0, roll_forward_to=49'24))", 2024-04-19T01:04:33.606 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:19.568233+0000", 2024-04-19T01:04:33.606 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 13.970518179999999, 2024-04-19T01:04:33.606 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.01137372, 2024-04-19T01:04:33.606 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:33.606 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:33.607 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:33.607 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:33.607 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.607 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:33.607 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.568233+0000", 2024-04-19T01:04:33.607 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:33.607 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.607 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.607 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:33.607 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.568233+0000", 2024-04-19T01:04:33.607 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:33.607 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.607 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.607 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:33.608 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.568235+0000", 2024-04-19T01:04:33.608 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0200000000000001e-06 2024-04-19T01:04:33.608 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.608 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.608 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:33.608 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.568401+0000", 2024-04-19T01:04:33.608 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00016551000000000001 2024-04-19T01:04:33.608 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.608 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.608 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:33.608 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.568444+0000", 2024-04-19T01:04:33.608 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.3427999999999998e-05 2024-04-19T01:04:33.608 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.608 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.609 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:33.609 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.568468+0000", 2024-04-19T01:04:33.609 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4263000000000001e-05 2024-04-19T01:04:33.609 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.609 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.609 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:33.609 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.569893+0000", 2024-04-19T01:04:33.609 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0014248939999999999 2024-04-19T01:04:33.609 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.609 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.609 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:33.609 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.569924+0000", 2024-04-19T01:04:33.609 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.0925e-05 2024-04-19T01:04:33.609 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.609 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.610 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:33.610 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.579539+0000", 2024-04-19T01:04:33.610 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0096151299999999995 2024-04-19T01:04:33.610 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.610 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.610 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:33.610 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.579607+0000", 2024-04-19T01:04:33.610 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.7550000000000002e-05 2024-04-19T01:04:33.610 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:33.610 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:33.610 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:33.610 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.610 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.610 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.5s6 59/31 ECSubWrite(tid=66, reqid=client.4604.0:423, at_version=59'51, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:04:33.611 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:29.540851+0000", 2024-04-19T01:04:33.611 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 3.99790014, 2024-04-19T01:04:33.611 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.030615177, 2024-04-19T01:04:33.611 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:33.611 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:33.611 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:33.611 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:33.611 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.611 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:33.611 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.540851+0000", 2024-04-19T01:04:33.611 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:33.611 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.611 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.611 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:33.611 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.540851+0000", 2024-04-19T01:04:33.612 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:33.612 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.612 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.612 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:33.612 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.540852+0000", 2024-04-19T01:04:33.612 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.6400000000000002e-07 2024-04-19T01:04:33.612 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.612 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.612 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:33.612 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.541398+0000", 2024-04-19T01:04:33.612 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00054620600000000004 2024-04-19T01:04:33.612 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.612 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.612 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:33.613 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.541433+0000", 2024-04-19T01:04:33.613 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.4853e-05 2024-04-19T01:04:33.613 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.613 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.613 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:33.613 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.541457+0000", 2024-04-19T01:04:33.613 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4641999999999999e-05 2024-04-19T01:04:33.613 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.613 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.613 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:33.613 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.541639+0000", 2024-04-19T01:04:33.613 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00018139100000000001 2024-04-19T01:04:33.613 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.613 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.614 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:33.614 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.541667+0000", 2024-04-19T01:04:33.614 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.7943e-05 2024-04-19T01:04:33.614 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.614 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.614 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:33.614 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.571413+0000", 2024-04-19T01:04:33.614 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0297462 2024-04-19T01:04:33.614 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.614 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.614 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:33.614 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.571466+0000", 2024-04-19T01:04:33.614 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.3378000000000003e-05 2024-04-19T01:04:33.614 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:33.615 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:33.615 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:33.615 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.615 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.615 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.5s6 59/31 ECSubWrite(tid=67, reqid=client.4604.0:424, at_version=59'52, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:04:33.615 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:29.548369+0000", 2024-04-19T01:04:33.615 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 3.9903823960000002, 2024-04-19T01:04:33.615 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.023140674999999999, 2024-04-19T01:04:33.615 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:33.615 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:33.615 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:33.615 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:33.615 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.615 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:33.616 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.548369+0000", 2024-04-19T01:04:33.616 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:33.616 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.616 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.616 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:33.616 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.548369+0000", 2024-04-19T01:04:33.616 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:33.617 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.617 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.617 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:33.617 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.548370+0000", 2024-04-19T01:04:33.617 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.066e-06 2024-04-19T01:04:33.617 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.617 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.617 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:33.617 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.548631+0000", 2024-04-19T01:04:33.617 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00026157799999999999 2024-04-19T01:04:33.617 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.617 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.617 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:33.617 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.548648+0000", 2024-04-19T01:04:33.618 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7079e-05 2024-04-19T01:04:33.618 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.618 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.618 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:33.618 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.548657+0000", 2024-04-19T01:04:33.618 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.9500000000000007e-06 2024-04-19T01:04:33.618 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.618 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.618 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:33.618 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.555857+0000", 2024-04-19T01:04:33.618 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0071992569999999997 2024-04-19T01:04:33.618 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.618 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.619 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:33.619 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.555883+0000", 2024-04-19T01:04:33.619 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6522000000000001e-05 2024-04-19T01:04:33.619 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.619 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.619 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:33.619 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.571475+0000", 2024-04-19T01:04:33.619 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.015591727999999999 2024-04-19T01:04:33.619 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.619 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.619 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:33.619 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.571509+0000", 2024-04-19T01:04:33.619 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.4495000000000001e-05 2024-04-19T01:04:33.619 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:33.620 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:33.620 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:33.620 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.620 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.620 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.5s6 59/31 ECSubWrite(tid=68, reqid=client.4604.0:425, at_version=59'53, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:04:33.620 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:29.555995+0000", 2024-04-19T01:04:33.620 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 3.9827561579999999, 2024-04-19T01:04:33.620 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.015894385, 2024-04-19T01:04:33.620 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:33.620 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:33.620 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:33.620 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:33.620 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.621 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:33.621 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.555995+0000", 2024-04-19T01:04:33.621 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:33.621 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.621 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.621 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:33.621 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.555995+0000", 2024-04-19T01:04:33.621 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:33.621 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.621 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.621 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:33.621 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.555996+0000", 2024-04-19T01:04:33.621 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.2640000000000001e-06 2024-04-19T01:04:33.622 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.622 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.622 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:33.622 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.556050+0000", 2024-04-19T01:04:33.622 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.3804999999999997e-05 2024-04-19T01:04:33.622 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.622 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.622 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:33.622 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.556068+0000", 2024-04-19T01:04:33.622 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7467999999999999e-05 2024-04-19T01:04:33.622 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.622 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.622 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:33.623 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.556081+0000", 2024-04-19T01:04:33.623 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.3081000000000001e-05 2024-04-19T01:04:33.623 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.623 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.623 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:33.623 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.568917+0000", 2024-04-19T01:04:33.623 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.012836596 2024-04-19T01:04:33.623 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.623 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.623 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:33.623 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.568932+0000", 2024-04-19T01:04:33.623 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5228999999999999e-05 2024-04-19T01:04:33.623 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.623 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.623 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:33.624 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.571819+0000", 2024-04-19T01:04:33.624 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0028863679999999998 2024-04-19T01:04:33.624 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.624 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.624 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:33.624 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.571889+0000", 2024-04-19T01:04:33.624 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.0574000000000006e-05 2024-04-19T01:04:33.624 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:33.624 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:33.624 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:33.624 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.624 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.624 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.5s6 59/31 ECSubWrite(tid=69, reqid=client.4604.0:426, at_version=59'54, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:04:33.624 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:29.557509+0000", 2024-04-19T01:04:33.624 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 3.981241781, 2024-04-19T01:04:33.625 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.015146768999999999, 2024-04-19T01:04:33.625 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:33.625 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:33.625 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:33.625 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:33.625 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.625 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:33.625 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.557509+0000", 2024-04-19T01:04:33.625 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:33.625 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.625 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.625 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:33.625 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.557509+0000", 2024-04-19T01:04:33.625 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:33.626 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.626 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.626 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:33.626 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.557510+0000", 2024-04-19T01:04:33.626 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.057e-06 2024-04-19T01:04:33.626 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.626 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.626 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:33.626 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.557535+0000", 2024-04-19T01:04:33.626 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4700000000000001e-05 2024-04-19T01:04:33.626 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.626 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.626 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:33.626 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.557556+0000", 2024-04-19T01:04:33.627 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1107999999999998e-05 2024-04-19T01:04:33.627 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.627 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.627 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:33.627 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.557569+0000", 2024-04-19T01:04:33.627 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.2889e-05 2024-04-19T01:04:33.627 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.627 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.627 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:33.627 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.571117+0000", 2024-04-19T01:04:33.627 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.013548248000000001 2024-04-19T01:04:33.627 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.627 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.627 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:33.627 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.571133+0000", 2024-04-19T01:04:33.628 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5943e-05 2024-04-19T01:04:33.628 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.628 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.628 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:33.628 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.572561+0000", 2024-04-19T01:04:33.628 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001428157 2024-04-19T01:04:33.628 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.628 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.628 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:33.628 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.572656+0000", 2024-04-19T01:04:33.628 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.4666999999999993e-05 2024-04-19T01:04:33.628 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:33.628 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:33.628 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:33.628 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.629 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.629 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.8s6 60/31 ECSubWrite(tid=225, reqid=client.4604.0:433, at_version=60'43, trim_to=0'0, roll_forward_to=59'41))", 2024-04-19T01:04:33.629 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:30.569927+0000", 2024-04-19T01:04:33.629 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 2.9688241, 2024-04-19T01:04:33.629 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.027044315999999999, 2024-04-19T01:04:33.629 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:33.629 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:33.629 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:33.629 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:33.629 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.630 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:33.630 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.569927+0000", 2024-04-19T01:04:33.630 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:33.630 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.630 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.630 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:33.630 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.569927+0000", 2024-04-19T01:04:33.630 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:33.630 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.630 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.630 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:33.630 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.569927+0000", 2024-04-19T01:04:33.630 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.34e-07 2024-04-19T01:04:33.630 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.631 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.631 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:33.631 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.570430+0000", 2024-04-19T01:04:33.631 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00050269400000000004 2024-04-19T01:04:33.631 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.631 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.631 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:33.631 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.570465+0000", 2024-04-19T01:04:33.631 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.4898999999999999e-05 2024-04-19T01:04:33.631 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.631 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.631 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:33.631 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.570490+0000", 2024-04-19T01:04:33.631 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4805999999999999e-05 2024-04-19T01:04:33.632 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.632 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.632 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:33.632 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.570639+0000", 2024-04-19T01:04:33.632 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00014890899999999999 2024-04-19T01:04:33.632 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.632 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.632 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:33.632 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.570665+0000", 2024-04-19T01:04:33.632 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5825000000000001e-05 2024-04-19T01:04:33.632 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.632 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.632 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:33.632 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.596871+0000", 2024-04-19T01:04:33.633 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.026206291999999999 2024-04-19T01:04:33.633 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.633 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.633 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:33.633 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.596971+0000", 2024-04-19T01:04:33.633 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000100457 2024-04-19T01:04:33.633 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:33.633 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:33.633 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:33.633 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.633 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.633 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.8s6 60/31 ECSubWrite(tid=226, reqid=client.4604.0:434, at_version=60'44, trim_to=0'0, roll_forward_to=59'41))", 2024-04-19T01:04:33.633 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:30.582166+0000", 2024-04-19T01:04:33.633 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 2.9565852619999999, 2024-04-19T01:04:33.634 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.015613482999999999, 2024-04-19T01:04:33.634 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:33.634 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:33.634 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:33.634 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:33.634 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.634 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:33.634 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.582166+0000", 2024-04-19T01:04:33.634 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:33.634 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.634 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.634 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:33.634 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.582166+0000", 2024-04-19T01:04:33.634 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:33.635 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.635 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.635 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:33.635 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.582168+0000", 2024-04-19T01:04:33.635 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.996e-06 2024-04-19T01:04:33.635 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.635 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.635 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:33.635 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.582695+0000", 2024-04-19T01:04:33.635 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00052661099999999996 2024-04-19T01:04:33.635 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.635 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.635 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:33.635 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.582731+0000", 2024-04-19T01:04:33.635 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.6548999999999999e-05 2024-04-19T01:04:33.635 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.636 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.636 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:33.636 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.582754+0000", 2024-04-19T01:04:33.636 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2785999999999998e-05 2024-04-19T01:04:33.636 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.636 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.636 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:33.636 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.583399+0000", 2024-04-19T01:04:33.636 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00064540599999999995 2024-04-19T01:04:33.636 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.636 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.636 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:33.636 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.583424+0000", 2024-04-19T01:04:33.636 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4596e-05 2024-04-19T01:04:33.636 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.637 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.637 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:33.637 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.597688+0000", 2024-04-19T01:04:33.637 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.014264354999999999 2024-04-19T01:04:33.637 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:33.637 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:33.637 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:33.637 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.597779+0000", 2024-04-19T01:04:33.637 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.1184e-05 2024-04-19T01:04:33.637 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:33.637 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:33.637 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:33.637 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:33.637 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:33.637 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:33.639 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2024-04-19T01:04:33.639 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-19T01:04:33.640 INFO:tasks.rados.rados.0.smithi155.stdout:184: done (3 left) 2024-04-19T01:04:33.640 INFO:tasks.rados.rados.0.smithi155.stdout:185: done (2 left) 2024-04-19T01:04:33.640 INFO:tasks.rados.rados.0.smithi155.stdout:186: done (1 left) 2024-04-19T01:04:33.640 INFO:tasks.rados.rados.0.smithi155.stdout:187: done (0 left) 2024-04-19T01:04:33.640 INFO:tasks.rados.rados.0.smithi155.stdout:188: copy_from oid 32 from oid 50 current snap is 16 2024-04-19T01:04:33.640 INFO:tasks.rados.rados.0.smithi155.stdout:189: snap_remove snap 11 2024-04-19T01:04:33.641 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-19T01:04:33.641 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:04:33.638+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:04:33.740 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:04:33.740+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:04:33.809 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:33.809 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [], 2024-04-19T01:04:33.809 INFO:teuthology.orchestra.run.smithi110.stdout: "num_ops": 0 2024-04-19T01:04:33.809 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:33.823 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-19T01:04:33.858 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2024-04-19T01:04:33.859 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-19T01:04:33.870 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:04:33.870+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:04:33.941 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:04:33.942+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:04:33.944 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:33.945 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [], 2024-04-19T01:04:33.945 INFO:teuthology.orchestra.run.smithi110.stdout: "complaint_time": 30, 2024-04-19T01:04:33.945 INFO:teuthology.orchestra.run.smithi110.stdout: "num_blocked_ops": 0 2024-04-19T01:04:33.945 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:33.959 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-19T01:04:34.046 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:04:34.046+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:04:34.128 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:34.128 INFO:teuthology.orchestra.run.smithi110.stdout: "size": 20, 2024-04-19T01:04:34.128 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 600, 2024-04-19T01:04:34.129 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [ 2024-04-19T01:04:34.129 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.129 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 39/31 ECSubWrite(tid=32, reqid=client.4604.0:181, at_version=39'16, trim_to=0'0, roll_forward_to=38'14))", 2024-04-19T01:04:34.129 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:09.820805+0000", 2024-04-19T01:04:34.129 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 24.308636034999999, 2024-04-19T01:04:34.129 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.013829808000000001, 2024-04-19T01:04:34.129 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:34.129 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:34.129 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:34.129 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:34.129 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.129 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:34.129 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.820805+0000", 2024-04-19T01:04:34.129 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.130 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.130 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.130 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:34.130 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.820805+0000", 2024-04-19T01:04:34.130 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.130 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.130 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.130 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:34.130 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.820806+0000", 2024-04-19T01:04:34.130 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.2699999999999999e-07 2024-04-19T01:04:34.130 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.131 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.131 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:34.131 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.821061+0000", 2024-04-19T01:04:34.131 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000255133 2024-04-19T01:04:34.131 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.131 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.131 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:34.131 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.821256+0000", 2024-04-19T01:04:34.131 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00019562400000000001 2024-04-19T01:04:34.131 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.132 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.132 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:34.132 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.821283+0000", 2024-04-19T01:04:34.132 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6712000000000001e-05 2024-04-19T01:04:34.132 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.132 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.132 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.132 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.821504+0000", 2024-04-19T01:04:34.132 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00022120199999999999 2024-04-19T01:04:34.132 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.132 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.132 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:34.132 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.821557+0000", 2024-04-19T01:04:34.133 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.2534000000000002e-05 2024-04-19T01:04:34.133 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.133 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.133 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:34.133 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.834585+0000", 2024-04-19T01:04:34.133 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.013027617999999999 2024-04-19T01:04:34.133 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.133 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.133 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:34.133 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.834635+0000", 2024-04-19T01:04:34.133 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.0257999999999999e-05 2024-04-19T01:04:34.133 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.133 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.133 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.134 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.134 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.134 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 43/31 ECSubWrite(tid=70, reqid=client.4604.0:235, at_version=43'43, trim_to=0'0, roll_forward_to=41'37))", 2024-04-19T01:04:34.134 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.870038+0000", 2024-04-19T01:04:34.134 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 20.259402673, 2024-04-19T01:04:34.134 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.020219719000000001, 2024-04-19T01:04:34.134 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:34.135 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:34.135 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:34.135 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:34.135 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.135 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:34.135 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.870038+0000", 2024-04-19T01:04:34.135 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.136 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.136 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.136 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:34.136 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.870038+0000", 2024-04-19T01:04:34.136 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.136 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.136 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.136 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:34.136 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.870040+0000", 2024-04-19T01:04:34.136 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.81e-06 2024-04-19T01:04:34.136 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.136 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.136 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:34.136 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.870453+0000", 2024-04-19T01:04:34.137 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000412965 2024-04-19T01:04:34.137 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.137 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.137 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:34.137 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.870494+0000", 2024-04-19T01:04:34.137 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.0938e-05 2024-04-19T01:04:34.137 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.137 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.137 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:34.137 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.870510+0000", 2024-04-19T01:04:34.137 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5458999999999999e-05 2024-04-19T01:04:34.137 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.137 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.137 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.137 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.870799+0000", 2024-04-19T01:04:34.138 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00028974699999999998 2024-04-19T01:04:34.138 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.138 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.138 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:34.138 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.870832+0000", 2024-04-19T01:04:34.138 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.2434999999999997e-05 2024-04-19T01:04:34.138 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.138 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.138 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:34.138 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.890185+0000", 2024-04-19T01:04:34.139 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.019353467999999999 2024-04-19T01:04:34.139 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.139 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.139 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:34.139 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.890258+0000", 2024-04-19T01:04:34.139 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.2897000000000004e-05 2024-04-19T01:04:34.139 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.139 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.139 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.139 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.139 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.139 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 43/31 ECSubWrite(tid=72, reqid=client.4604.0:236, at_version=43'44, trim_to=0'0, roll_forward_to=41'37))", 2024-04-19T01:04:34.139 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.890428+0000", 2024-04-19T01:04:34.139 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 20.239012768999999, 2024-04-19T01:04:34.140 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0076663950000000003, 2024-04-19T01:04:34.140 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:34.140 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:34.140 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:34.140 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:34.140 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.140 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:34.140 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.890428+0000", 2024-04-19T01:04:34.140 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.140 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.140 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.141 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:34.144 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.890428+0000", 2024-04-19T01:04:34.144 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.144 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.144 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.144 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:34.144 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.890430+0000", 2024-04-19T01:04:34.144 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0040000000000002e-06 2024-04-19T01:04:34.144 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.144 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.144 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:34.145 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.890673+0000", 2024-04-19T01:04:34.145 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000242425 2024-04-19T01:04:34.145 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.145 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.145 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:34.145 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.890711+0000", 2024-04-19T01:04:34.145 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.8717999999999998e-05 2024-04-19T01:04:34.145 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.145 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.145 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:34.146 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.890735+0000", 2024-04-19T01:04:34.146 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4006e-05 2024-04-19T01:04:34.146 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.146 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.146 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.146 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.890909+0000", 2024-04-19T01:04:34.146 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00017384200000000001 2024-04-19T01:04:34.146 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.146 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.147 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:34.147 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.890938+0000", 2024-04-19T01:04:34.147 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.8958999999999998e-05 2024-04-19T01:04:34.147 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.147 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.147 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:34.147 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.898014+0000", 2024-04-19T01:04:34.147 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0070753470000000001 2024-04-19T01:04:34.147 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.147 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.147 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:34.147 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.898095+0000", 2024-04-19T01:04:34.147 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.1094000000000007e-05 2024-04-19T01:04:34.148 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.148 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.148 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.148 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.148 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.148 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.ds2 43/31 ECSubWrite(tid=38, reqid=client.4604.0:253, at_version=43'14, trim_to=0'0, roll_forward_to=43'12))", 2024-04-19T01:04:34.148 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.992478+0000", 2024-04-19T01:04:34.148 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 20.136963468000001, 2024-04-19T01:04:34.148 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.025708154, 2024-04-19T01:04:34.148 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:34.148 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:34.148 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:34.149 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:34.149 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.149 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:34.149 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.992478+0000", 2024-04-19T01:04:34.149 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.149 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.149 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.149 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:34.149 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.992478+0000", 2024-04-19T01:04:34.149 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.149 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.149 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.150 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:34.150 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.992480+0000", 2024-04-19T01:04:34.150 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9870000000000002e-06 2024-04-19T01:04:34.150 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.150 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.150 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:34.150 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.992860+0000", 2024-04-19T01:04:34.150 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00038019600000000002 2024-04-19T01:04:34.150 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.150 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.150 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:34.150 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.992891+0000", 2024-04-19T01:04:34.150 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.0982000000000003e-05 2024-04-19T01:04:34.151 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.151 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.151 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:34.151 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.992906+0000", 2024-04-19T01:04:34.151 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5503e-05 2024-04-19T01:04:34.151 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.151 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.151 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.151 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.993064+0000", 2024-04-19T01:04:34.151 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015782900000000001 2024-04-19T01:04:34.151 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.151 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.151 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:34.152 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.993111+0000", 2024-04-19T01:04:34.152 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.7138000000000002e-05 2024-04-19T01:04:34.152 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.152 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.152 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:34.152 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.018092+0000", 2024-04-19T01:04:34.152 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.024980822999999999 2024-04-19T01:04:34.152 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.152 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.152 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:34.152 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.018186+0000", 2024-04-19T01:04:34.152 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.3696000000000006e-05 2024-04-19T01:04:34.153 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.153 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.153 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.153 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.153 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.153 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.ds2 43/31 ECSubWrite(tid=39, reqid=client.4604.0:254, at_version=43'15, trim_to=0'0, roll_forward_to=43'12))", 2024-04-19T01:04:34.153 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.996882+0000", 2024-04-19T01:04:34.153 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 20.132559349000001, 2024-04-19T01:04:34.153 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.032356653999999999, 2024-04-19T01:04:34.153 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:34.153 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:34.153 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:34.153 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:34.153 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.154 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:34.154 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.996882+0000", 2024-04-19T01:04:34.154 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.154 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.154 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.154 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:34.154 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.996882+0000", 2024-04-19T01:04:34.154 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.154 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.154 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.154 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:34.154 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.996884+0000", 2024-04-19T01:04:34.154 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2029999999999999e-06 2024-04-19T01:04:34.155 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.155 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.155 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:34.155 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.997205+0000", 2024-04-19T01:04:34.155 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00032076899999999999 2024-04-19T01:04:34.155 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.155 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.155 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:34.155 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.997249+0000", 2024-04-19T01:04:34.155 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.4654000000000003e-05 2024-04-19T01:04:34.155 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.155 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.155 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:34.155 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.997273+0000", 2024-04-19T01:04:34.156 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3589000000000001e-05 2024-04-19T01:04:34.156 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.156 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.156 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.156 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.007691+0000", 2024-04-19T01:04:34.156 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.010418495999999999 2024-04-19T01:04:34.156 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.156 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.156 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:34.156 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.007717+0000", 2024-04-19T01:04:34.156 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5876000000000001e-05 2024-04-19T01:04:34.156 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.156 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.156 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:34.157 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.029151+0000", 2024-04-19T01:04:34.157 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.021433651000000001 2024-04-19T01:04:34.157 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.157 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.157 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:34.157 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.029238+0000", 2024-04-19T01:04:34.157 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.7416000000000005e-05 2024-04-19T01:04:34.157 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.157 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.157 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.157 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.157 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.158 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.ds2 43/31 ECSubWrite(tid=40, reqid=client.4604.0:255, at_version=43'16, trim_to=0'0, roll_forward_to=43'12))", 2024-04-19T01:04:34.158 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.002736+0000", 2024-04-19T01:04:34.158 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 20.12670511, 2024-04-19T01:04:34.158 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.026570379000000002, 2024-04-19T01:04:34.158 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:34.158 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:34.158 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:34.158 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:34.158 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.158 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:34.158 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.002736+0000", 2024-04-19T01:04:34.158 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.159 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.159 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.159 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:34.159 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.002736+0000", 2024-04-19T01:04:34.159 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.159 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.159 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.159 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:34.159 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.002738+0000", 2024-04-19T01:04:34.160 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0899999999999999e-06 2024-04-19T01:04:34.160 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.160 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.160 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:34.160 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.002782+0000", 2024-04-19T01:04:34.160 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.3611e-05 2024-04-19T01:04:34.160 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.160 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.160 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:34.160 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.002823+0000", 2024-04-19T01:04:34.160 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.0954999999999997e-05 2024-04-19T01:04:34.160 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.161 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.161 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:34.161 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.002845+0000", 2024-04-19T01:04:34.161 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2775000000000001e-05 2024-04-19T01:04:34.161 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.161 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.161 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.161 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.019866+0000", 2024-04-19T01:04:34.161 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.017021162999999999 2024-04-19T01:04:34.161 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.161 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.161 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:34.162 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.019893+0000", 2024-04-19T01:04:34.162 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6176000000000001e-05 2024-04-19T01:04:34.162 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.162 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.162 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:34.162 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.029258+0000", 2024-04-19T01:04:34.162 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0093652990000000005 2024-04-19T01:04:34.162 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.162 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.162 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:34.162 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.029306+0000", 2024-04-19T01:04:34.162 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.8310000000000003e-05 2024-04-19T01:04:34.162 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.163 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.163 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.163 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.163 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.163 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.ds2 43/31 ECSubWrite(tid=42, reqid=client.4604.0:271, at_version=43'18, trim_to=0'0, roll_forward_to=43'14))", 2024-04-19T01:04:34.163 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.019519+0000", 2024-04-19T01:04:34.163 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 20.109922403999999, 2024-04-19T01:04:34.164 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.010349381, 2024-04-19T01:04:34.164 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:34.164 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:34.164 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:34.164 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:34.164 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.165 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:34.165 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.019519+0000", 2024-04-19T01:04:34.165 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.165 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.165 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.165 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:34.165 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.019519+0000", 2024-04-19T01:04:34.165 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.165 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.165 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.165 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:34.165 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.019520+0000", 2024-04-19T01:04:34.166 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.424e-06 2024-04-19T01:04:34.166 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.166 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.166 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:34.166 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.019678+0000", 2024-04-19T01:04:34.166 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015818599999999999 2024-04-19T01:04:34.166 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.166 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.166 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:34.166 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.019702+0000", 2024-04-19T01:04:34.166 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3561999999999999e-05 2024-04-19T01:04:34.166 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.166 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.167 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:34.167 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.019717+0000", 2024-04-19T01:04:34.167 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.4793e-05 2024-04-19T01:04:34.167 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.167 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.167 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.167 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.021792+0000", 2024-04-19T01:04:34.167 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0020751739999999999 2024-04-19T01:04:34.167 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.167 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.167 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:34.167 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.021855+0000", 2024-04-19T01:04:34.167 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.3436999999999996e-05 2024-04-19T01:04:34.167 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.168 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.168 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:34.168 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.029768+0000", 2024-04-19T01:04:34.168 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0079128429999999993 2024-04-19T01:04:34.168 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.168 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.168 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:34.168 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.029868+0000", 2024-04-19T01:04:34.168 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.9962000000000003e-05 2024-04-19T01:04:34.168 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.168 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.168 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.168 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.168 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.169 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 43/31 ECSubWrite(tid=76, reqid=client.4604.0:267, at_version=43'46, trim_to=0'0, roll_forward_to=43'45))", 2024-04-19T01:04:34.169 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.020527+0000", 2024-04-19T01:04:34.169 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 20.108914102, 2024-04-19T01:04:34.169 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.016212788999999998, 2024-04-19T01:04:34.169 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:34.169 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:34.169 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:34.169 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:34.169 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.169 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:34.169 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.020527+0000", 2024-04-19T01:04:34.169 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.169 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.170 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.170 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:34.170 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.020527+0000", 2024-04-19T01:04:34.170 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.170 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.170 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.170 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:34.170 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.020529+0000", 2024-04-19T01:04:34.170 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.046e-06 2024-04-19T01:04:34.170 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.170 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.170 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:34.170 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.020873+0000", 2024-04-19T01:04:34.170 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00034371800000000002 2024-04-19T01:04:34.171 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.171 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.171 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:34.171 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.020902+0000", 2024-04-19T01:04:34.171 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.8903000000000001e-05 2024-04-19T01:04:34.171 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.171 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.171 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:34.171 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.020924+0000", 2024-04-19T01:04:34.171 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2892e-05 2024-04-19T01:04:34.171 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.171 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.171 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.171 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.021071+0000", 2024-04-19T01:04:34.171 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00014612300000000001 2024-04-19T01:04:34.172 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.172 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.172 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:34.172 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.021113+0000", 2024-04-19T01:04:34.172 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.2573e-05 2024-04-19T01:04:34.172 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.172 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.172 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:34.172 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.036641+0000", 2024-04-19T01:04:34.172 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.015527501000000001 2024-04-19T01:04:34.172 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.172 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.172 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:34.173 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.036740+0000", 2024-04-19T01:04:34.173 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.9032999999999999e-05 2024-04-19T01:04:34.173 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.173 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.173 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.173 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.173 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.173 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.1s4 44/31 ECSubWrite(tid=15, reqid=client.4604.0:289, at_version=44'8, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:04:34.173 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.875457+0000", 2024-04-19T01:04:34.173 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 19.253983784999999, 2024-04-19T01:04:34.173 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.036567641999999997, 2024-04-19T01:04:34.173 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:34.173 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:34.173 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:34.174 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:34.174 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.174 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:34.174 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875457+0000", 2024-04-19T01:04:34.174 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.174 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.174 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.174 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:34.174 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875457+0000", 2024-04-19T01:04:34.174 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.174 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.174 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.175 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:34.175 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875458+0000", 2024-04-19T01:04:34.175 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.8400000000000004e-07 2024-04-19T01:04:34.175 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.175 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.175 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:34.175 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875519+0000", 2024-04-19T01:04:34.175 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.1377000000000005e-05 2024-04-19T01:04:34.175 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.175 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.175 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:34.175 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875588+0000", 2024-04-19T01:04:34.175 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.9099999999999999e-05 2024-04-19T01:04:34.175 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.176 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.176 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:34.176 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875615+0000", 2024-04-19T01:04:34.176 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6902000000000001e-05 2024-04-19T01:04:34.176 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.176 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.176 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.176 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875846+0000", 2024-04-19T01:04:34.176 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00023076699999999999 2024-04-19T01:04:34.176 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.176 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.177 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:34.177 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875874+0000", 2024-04-19T01:04:34.177 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.7725e-05 2024-04-19T01:04:34.177 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.177 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.178 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:34.178 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.911826+0000", 2024-04-19T01:04:34.178 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.035952348000000002 2024-04-19T01:04:34.178 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.178 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.178 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:34.178 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.912025+0000", 2024-04-19T01:04:34.178 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000198739 2024-04-19T01:04:34.178 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.178 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.178 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.179 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.179 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.179 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 44/31 ECSubWrite(tid=82, reqid=client.4604.0:280, at_version=44'50, trim_to=0'0, roll_forward_to=43'48))", 2024-04-19T01:04:34.179 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.880504+0000", 2024-04-19T01:04:34.179 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 19.248936691000001, 2024-04-19T01:04:34.179 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.041548157000000002, 2024-04-19T01:04:34.179 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:34.179 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:34.179 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:34.179 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:34.179 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.179 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:34.179 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.880504+0000", 2024-04-19T01:04:34.180 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.180 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.180 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.180 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:34.180 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.880504+0000", 2024-04-19T01:04:34.180 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.180 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.180 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.180 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:34.180 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.880506+0000", 2024-04-19T01:04:34.180 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9439999999999999e-06 2024-04-19T01:04:34.180 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.180 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.181 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:34.181 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.880844+0000", 2024-04-19T01:04:34.181 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00033782799999999998 2024-04-19T01:04:34.181 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.181 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.181 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:34.181 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.880877+0000", 2024-04-19T01:04:34.181 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.2524999999999998e-05 2024-04-19T01:04:34.181 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.181 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.181 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:34.182 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.880899+0000", 2024-04-19T01:04:34.182 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2656000000000001e-05 2024-04-19T01:04:34.182 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.182 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.182 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.182 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.881129+0000", 2024-04-19T01:04:34.182 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00022929600000000001 2024-04-19T01:04:34.182 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.182 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.182 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:34.182 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.881155+0000", 2024-04-19T01:04:34.182 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6729000000000001e-05 2024-04-19T01:04:34.182 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.183 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.183 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:34.183 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.921810+0000", 2024-04-19T01:04:34.183 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.040654807000000001 2024-04-19T01:04:34.183 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.183 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.183 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:34.183 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.922052+0000", 2024-04-19T01:04:34.183 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000242372 2024-04-19T01:04:34.183 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.183 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.183 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.184 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.184 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.184 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.es5 44/31 ECSubWrite(tid=83, reqid=osd.13.0:81, at_version=0'0, trim_to=0'0, roll_forward_to=44'14))", 2024-04-19T01:04:34.184 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.887116+0000", 2024-04-19T01:04:34.184 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 19.242325329, 2024-04-19T01:04:34.184 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0081527319999999993, 2024-04-19T01:04:34.184 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:34.184 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:34.184 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:34.184 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:34.184 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.184 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:34.184 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.887116+0000", 2024-04-19T01:04:34.185 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.185 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.185 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.185 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:34.185 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.887116+0000", 2024-04-19T01:04:34.185 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.185 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.185 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.185 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:34.185 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.887118+0000", 2024-04-19T01:04:34.185 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1560000000000002e-06 2024-04-19T01:04:34.185 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.185 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.185 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:34.186 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.887144+0000", 2024-04-19T01:04:34.186 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6495999999999998e-05 2024-04-19T01:04:34.186 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.186 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.186 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:34.186 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.887178+0000", 2024-04-19T01:04:34.186 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.3241999999999998e-05 2024-04-19T01:04:34.186 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.186 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.186 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:34.186 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.887200+0000", 2024-04-19T01:04:34.186 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2665000000000001e-05 2024-04-19T01:04:34.187 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.187 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.187 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.187 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.887400+0000", 2024-04-19T01:04:34.187 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000199278 2024-04-19T01:04:34.187 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.187 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.187 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:34.187 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.887424+0000", 2024-04-19T01:04:34.187 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4545999999999998e-05 2024-04-19T01:04:34.187 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.187 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.187 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:34.188 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.895177+0000", 2024-04-19T01:04:34.188 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.007752643 2024-04-19T01:04:34.188 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.188 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.188 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:34.188 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.895268+0000", 2024-04-19T01:04:34.188 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.1706000000000006e-05 2024-04-19T01:04:34.188 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.188 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.188 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.188 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.188 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.188 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.ds2 44/31 ECSubWrite(tid=45, reqid=osd.12.0:45, at_version=44'20, trim_to=0'0, roll_forward_to=43'18))", 2024-04-19T01:04:34.188 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.888380+0000", 2024-04-19T01:04:34.189 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 19.241060618999999, 2024-04-19T01:04:34.189 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0076145199999999996, 2024-04-19T01:04:34.189 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:34.189 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:34.189 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:34.189 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:34.189 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.189 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:34.189 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.888380+0000", 2024-04-19T01:04:34.189 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.189 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.189 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.189 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:34.189 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.888380+0000", 2024-04-19T01:04:34.190 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.190 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.190 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.190 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:34.190 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.888381+0000", 2024-04-19T01:04:34.190 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.23e-07 2024-04-19T01:04:34.190 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.190 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.191 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:34.191 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.888477+0000", 2024-04-19T01:04:34.191 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.6095999999999996e-05 2024-04-19T01:04:34.191 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.191 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.191 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:34.191 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.888511+0000", 2024-04-19T01:04:34.191 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.4184999999999999e-05 2024-04-19T01:04:34.191 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.191 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.191 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:34.191 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.888537+0000", 2024-04-19T01:04:34.191 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5202e-05 2024-04-19T01:04:34.192 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.192 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.192 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.192 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.888786+0000", 2024-04-19T01:04:34.192 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00024980899999999998 2024-04-19T01:04:34.192 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.192 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.192 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:34.192 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.888812+0000", 2024-04-19T01:04:34.192 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5629000000000001e-05 2024-04-19T01:04:34.192 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.192 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.192 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:34.192 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.895898+0000", 2024-04-19T01:04:34.192 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0070860430000000002 2024-04-19T01:04:34.193 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.193 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.193 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:34.193 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.895995+0000", 2024-04-19T01:04:34.193 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.6732999999999997e-05 2024-04-19T01:04:34.193 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.193 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.193 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.193 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.193 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.193 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.cs4 44/31 ECSubWrite(tid=52, reqid=osd.0.0:52, at_version=44'25, trim_to=0'0, roll_forward_to=43'23))", 2024-04-19T01:04:34.193 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.889007+0000", 2024-04-19T01:04:34.193 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 19.240433770999999, 2024-04-19T01:04:34.194 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0074716929999999997, 2024-04-19T01:04:34.194 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:34.194 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:34.194 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:34.194 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:34.194 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.194 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:34.194 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.889007+0000", 2024-04-19T01:04:34.194 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.194 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.195 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.195 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:34.195 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.889007+0000", 2024-04-19T01:04:34.195 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.195 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.195 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.195 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:34.195 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.889008+0000", 2024-04-19T01:04:34.195 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.6000000000000003e-07 2024-04-19T01:04:34.195 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.195 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.195 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:34.195 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.889062+0000", 2024-04-19T01:04:34.196 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.4112e-05 2024-04-19T01:04:34.196 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.196 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.196 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:34.196 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.889103+0000", 2024-04-19T01:04:34.196 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.1134000000000003e-05 2024-04-19T01:04:34.196 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.196 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.196 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:34.196 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.889125+0000", 2024-04-19T01:04:34.196 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2232999999999999e-05 2024-04-19T01:04:34.196 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.197 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.197 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.197 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.889497+0000", 2024-04-19T01:04:34.197 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00037114599999999999 2024-04-19T01:04:34.197 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.197 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.197 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:34.197 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.889522+0000", 2024-04-19T01:04:34.197 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5352000000000001e-05 2024-04-19T01:04:34.197 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.197 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.197 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:34.197 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.896386+0000", 2024-04-19T01:04:34.197 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0068645499999999996 2024-04-19T01:04:34.198 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.198 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.198 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:34.198 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.896479+0000", 2024-04-19T01:04:34.198 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.2505999999999998e-05 2024-04-19T01:04:34.198 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.198 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.198 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.198 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.198 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.198 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 44/31 ECSubWrite(tid=84, reqid=client.4604.0:281, at_version=44'51, trim_to=0'0, roll_forward_to=43'48))", 2024-04-19T01:04:34.198 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.906686+0000", 2024-04-19T01:04:34.198 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 19.222755293999999, 2024-04-19T01:04:34.199 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.018013254999999999, 2024-04-19T01:04:34.199 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:34.199 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:34.199 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:34.199 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:34.199 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.199 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:34.199 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.906686+0000", 2024-04-19T01:04:34.199 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.199 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.199 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.199 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:34.199 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.906686+0000", 2024-04-19T01:04:34.199 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.200 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.200 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.200 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:34.200 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.906688+0000", 2024-04-19T01:04:34.200 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1569999999999998e-06 2024-04-19T01:04:34.200 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.200 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.200 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:34.200 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.906789+0000", 2024-04-19T01:04:34.200 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000101488 2024-04-19T01:04:34.200 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.200 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.200 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:34.200 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.906831+0000", 2024-04-19T01:04:34.201 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.1488e-05 2024-04-19T01:04:34.201 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.201 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.201 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:34.201 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.906855+0000", 2024-04-19T01:04:34.201 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4269000000000001e-05 2024-04-19T01:04:34.201 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.201 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.201 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.201 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.913161+0000", 2024-04-19T01:04:34.201 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0063058070000000001 2024-04-19T01:04:34.201 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.201 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.202 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:34.202 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.913193+0000", 2024-04-19T01:04:34.202 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.2007e-05 2024-04-19T01:04:34.202 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.202 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.202 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:34.202 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.924624+0000", 2024-04-19T01:04:34.202 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.011430879999999999 2024-04-19T01:04:34.203 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.203 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.203 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:34.203 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.924699+0000", 2024-04-19T01:04:34.203 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.5159000000000003e-05 2024-04-19T01:04:34.203 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.203 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.203 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.203 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.203 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.205 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 44/31 ECSubWrite(tid=85, reqid=client.4604.0:282, at_version=44'52, trim_to=0'0, roll_forward_to=43'48))", 2024-04-19T01:04:34.205 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.916938+0000", 2024-04-19T01:04:34.205 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 19.212503019, 2024-04-19T01:04:34.205 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0078204020000000006, 2024-04-19T01:04:34.205 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:34.206 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:34.206 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:34.206 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:34.206 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.206 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:34.206 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.916938+0000", 2024-04-19T01:04:34.206 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.206 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.206 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.206 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:34.206 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.916938+0000", 2024-04-19T01:04:34.206 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.206 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.207 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.207 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:34.207 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.916940+0000", 2024-04-19T01:04:34.207 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0159999999999998e-06 2024-04-19T01:04:34.207 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.207 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.207 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:34.207 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.916992+0000", 2024-04-19T01:04:34.207 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.2494999999999998e-05 2024-04-19T01:04:34.207 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.207 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.207 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:34.207 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.917033+0000", 2024-04-19T01:04:34.207 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.0908000000000002e-05 2024-04-19T01:04:34.207 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.208 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.208 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:34.208 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.917057+0000", 2024-04-19T01:04:34.208 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4042000000000001e-05 2024-04-19T01:04:34.208 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.208 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.208 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.208 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.922174+0000", 2024-04-19T01:04:34.208 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.005116308 2024-04-19T01:04:34.208 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.208 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.208 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:34.208 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.922203+0000", 2024-04-19T01:04:34.209 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.9615e-05 2024-04-19T01:04:34.209 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.209 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.209 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:34.209 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.924709+0000", 2024-04-19T01:04:34.209 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0025060299999999998 2024-04-19T01:04:34.209 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.209 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.209 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:34.209 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.924758+0000", 2024-04-19T01:04:34.209 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.8987999999999999e-05 2024-04-19T01:04:34.209 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.209 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.209 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.210 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.210 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.210 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.0s1 44/31 ECSubWrite(tid=25, reqid=client.4604.0:284, at_version=44'22, trim_to=0'0, roll_forward_to=44'20))", 2024-04-19T01:04:34.210 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.922536+0000", 2024-04-19T01:04:34.210 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 19.206905016, 2024-04-19T01:04:34.210 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.029007871000000001, 2024-04-19T01:04:34.210 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:34.210 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:34.210 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:34.210 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:34.210 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.210 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:34.210 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.922536+0000", 2024-04-19T01:04:34.210 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.211 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.211 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.211 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:34.211 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.922536+0000", 2024-04-19T01:04:34.211 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.211 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.211 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.211 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:34.211 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.922538+0000", 2024-04-19T01:04:34.211 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0829999999999998e-06 2024-04-19T01:04:34.211 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.211 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.211 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:34.211 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.923282+0000", 2024-04-19T01:04:34.211 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000743554 2024-04-19T01:04:34.212 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.212 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.212 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:34.212 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.923311+0000", 2024-04-19T01:04:34.212 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.9147999999999999e-05 2024-04-19T01:04:34.212 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.212 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.212 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:34.212 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.923337+0000", 2024-04-19T01:04:34.212 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6648999999999998e-05 2024-04-19T01:04:34.212 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.212 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.212 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.212 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.923466+0000", 2024-04-19T01:04:34.213 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00012821100000000001 2024-04-19T01:04:34.213 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.213 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.213 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:34.213 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.923478+0000", 2024-04-19T01:04:34.213 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.2291e-05 2024-04-19T01:04:34.213 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.213 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.213 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:34.213 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.951479+0000", 2024-04-19T01:04:34.213 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.02800066 2024-04-19T01:04:34.213 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.213 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.214 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:34.214 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.951544+0000", 2024-04-19T01:04:34.214 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.5275000000000001e-05 2024-04-19T01:04:34.214 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.214 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.214 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.214 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.214 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.214 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.ds2 47/31 ECSubWrite(tid=61, reqid=client.4604.0:330, at_version=47'32, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:04:34.214 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:17.677293+0000", 2024-04-19T01:04:34.214 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 16.452147886999999, 2024-04-19T01:04:34.214 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0087219389999999997, 2024-04-19T01:04:34.214 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:34.215 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:34.215 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:34.215 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:34.215 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.215 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:34.215 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.677293+0000", 2024-04-19T01:04:34.215 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.215 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.215 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.215 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:34.215 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.677293+0000", 2024-04-19T01:04:34.215 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.215 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.215 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.215 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:34.216 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.677294+0000", 2024-04-19T01:04:34.216 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.1289999999999999e-06 2024-04-19T01:04:34.216 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.216 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.216 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:34.216 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.677592+0000", 2024-04-19T01:04:34.216 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00029753199999999998 2024-04-19T01:04:34.216 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.216 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.216 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:34.216 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.677607+0000", 2024-04-19T01:04:34.216 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5367999999999999e-05 2024-04-19T01:04:34.216 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.217 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.217 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:34.217 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.677618+0000", 2024-04-19T01:04:34.217 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.1201e-05 2024-04-19T01:04:34.217 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.217 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.217 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.217 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.677706+0000", 2024-04-19T01:04:34.217 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.7769000000000006e-05 2024-04-19T01:04:34.217 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.218 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.218 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:34.218 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.677713+0000", 2024-04-19T01:04:34.218 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.8399999999999997e-06 2024-04-19T01:04:34.218 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.218 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.218 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:34.218 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.685970+0000", 2024-04-19T01:04:34.218 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0082570330000000004 2024-04-19T01:04:34.218 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.218 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.218 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:34.218 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.686015+0000", 2024-04-19T01:04:34.218 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.5067000000000001e-05 2024-04-19T01:04:34.219 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.219 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.219 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.219 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.219 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.219 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 48/31 ECSubWrite(tid=115, reqid=osd.13.0:112, at_version=0'0, trim_to=0'0, roll_forward_to=48'69))", 2024-04-19T01:04:34.219 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.539886+0000", 2024-04-19T01:04:34.219 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 15.589554722999999, 2024-04-19T01:04:34.219 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.011017575, 2024-04-19T01:04:34.219 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:34.219 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:34.220 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:34.220 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:34.220 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.220 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:34.220 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.539886+0000", 2024-04-19T01:04:34.220 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.220 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.220 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.220 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:34.220 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.539886+0000", 2024-04-19T01:04:34.220 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.220 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.220 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.221 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:34.221 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.539888+0000", 2024-04-19T01:04:34.221 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.24e-06 2024-04-19T01:04:34.221 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.221 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.221 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:34.221 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.539911+0000", 2024-04-19T01:04:34.221 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3785999999999999e-05 2024-04-19T01:04:34.221 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.221 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.221 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:34.221 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.539937+0000", 2024-04-19T01:04:34.221 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5252000000000001e-05 2024-04-19T01:04:34.221 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.222 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.222 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:34.222 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.539958+0000", 2024-04-19T01:04:34.222 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1223e-05 2024-04-19T01:04:34.222 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.222 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.222 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.222 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.540206+0000", 2024-04-19T01:04:34.222 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00024783599999999998 2024-04-19T01:04:34.222 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.222 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.222 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:34.222 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.540231+0000", 2024-04-19T01:04:34.222 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5318999999999999e-05 2024-04-19T01:04:34.223 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.223 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.223 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:34.223 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.550811+0000", 2024-04-19T01:04:34.223 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.010580042 2024-04-19T01:04:34.223 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.223 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.223 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:34.223 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.550904+0000", 2024-04-19T01:04:34.223 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.2876999999999999e-05 2024-04-19T01:04:34.223 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.223 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.223 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.224 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.224 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.224 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 48/31 ECSubWrite(tid=116, reqid=client.4604.0:350, at_version=48'71, trim_to=0'0, roll_forward_to=48'69))", 2024-04-19T01:04:34.224 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.545276+0000", 2024-04-19T01:04:34.224 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 15.584164545, 2024-04-19T01:04:34.224 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0076773500000000003, 2024-04-19T01:04:34.224 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:34.224 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:34.224 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:34.224 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:34.224 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.224 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:34.224 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.545276+0000", 2024-04-19T01:04:34.224 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.224 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.225 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.225 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:34.225 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.545276+0000", 2024-04-19T01:04:34.225 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.225 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.225 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.225 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:34.225 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.545278+0000", 2024-04-19T01:04:34.225 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.8560000000000001e-06 2024-04-19T01:04:34.225 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.225 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.225 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:34.226 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.545330+0000", 2024-04-19T01:04:34.226 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.1907999999999998e-05 2024-04-19T01:04:34.226 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.226 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.226 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:34.226 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.545393+0000", 2024-04-19T01:04:34.226 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.2830999999999995e-05 2024-04-19T01:04:34.226 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.226 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.226 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:34.226 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.545416+0000", 2024-04-19T01:04:34.226 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3255e-05 2024-04-19T01:04:34.226 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.226 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.227 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.227 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.545553+0000", 2024-04-19T01:04:34.227 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00013673599999999999 2024-04-19T01:04:34.227 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.227 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.227 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:34.227 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.545578+0000", 2024-04-19T01:04:34.227 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4759000000000001e-05 2024-04-19T01:04:34.227 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.227 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.227 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:34.227 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.552869+0000", 2024-04-19T01:04:34.227 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0072913880000000002 2024-04-19T01:04:34.227 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.228 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.228 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:34.228 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.552954+0000", 2024-04-19T01:04:34.228 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.4617000000000007e-05 2024-04-19T01:04:34.228 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.228 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.228 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.228 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.228 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.228 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 63/31 ECSubWrite(tid=152, reqid=client.4604.0:465, at_version=63'100, trim_to=0'0, roll_forward_to=62'98))", 2024-04-19T01:04:34.228 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:33.566417+0000", 2024-04-19T01:04:34.228 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 0.56302387700000001, 2024-04-19T01:04:34.228 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.010107771, 2024-04-19T01:04:34.228 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:34.229 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:34.229 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:34.229 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:34.229 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.229 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:34.229 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.566417+0000", 2024-04-19T01:04:34.229 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.229 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.229 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.229 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:34.229 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.566417+0000", 2024-04-19T01:04:34.229 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.229 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.229 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.229 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:34.230 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.566417+0000", 2024-04-19T01:04:34.230 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.3000000000000002e-07 2024-04-19T01:04:34.230 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.230 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.230 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:34.230 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.566642+0000", 2024-04-19T01:04:34.230 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00022461900000000001 2024-04-19T01:04:34.230 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.230 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.230 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:34.230 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.566656+0000", 2024-04-19T01:04:34.230 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.3740999999999999e-05 2024-04-19T01:04:34.230 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.231 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.231 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:34.231 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.566669+0000", 2024-04-19T01:04:34.231 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.2806000000000001e-05 2024-04-19T01:04:34.231 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.231 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.231 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.231 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.566764+0000", 2024-04-19T01:04:34.231 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.5223999999999995e-05 2024-04-19T01:04:34.231 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.231 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.235 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:34.235 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.566779+0000", 2024-04-19T01:04:34.235 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5081e-05 2024-04-19T01:04:34.236 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.236 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.236 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:34.237 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.576481+0000", 2024-04-19T01:04:34.237 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0097020909999999995 2024-04-19T01:04:34.237 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.237 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.237 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:34.237 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.576525+0000", 2024-04-19T01:04:34.237 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.3878999999999998e-05 2024-04-19T01:04:34.238 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.238 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.238 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.238 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.238 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.239 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:34.241 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-04-19T01:04:34.241 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:04:34.241+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:04:34.341 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:04:34.342+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:04:34.409 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:34.409 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [], 2024-04-19T01:04:34.409 INFO:teuthology.orchestra.run.smithi110.stdout: "num_ops": 0 2024-04-19T01:04:34.409 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:34.423 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-04-19T01:04:34.442 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:04:34.442+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:04:34.531 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-19T01:04:34.540 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:34.540 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [], 2024-04-19T01:04:34.540 INFO:teuthology.orchestra.run.smithi110.stdout: "complaint_time": 30, 2024-04-19T01:04:34.540 INFO:teuthology.orchestra.run.smithi110.stdout: "num_blocked_ops": 0 2024-04-19T01:04:34.540 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:34.551 INFO:tasks.rados.rados.0.smithi155.stdout:189: done (1 left) 2024-04-19T01:04:34.551 INFO:tasks.rados.rados.0.smithi155.stdout:190: setattr oid 29 current snap is 16 2024-04-19T01:04:34.551 INFO:tasks.rados.rados.0.smithi155.stdout:188: finishing copy_from racing read to smithi15528762-32 2024-04-19T01:04:34.551 INFO:tasks.rados.rados.0.smithi155.stdout:188: finishing copy_from to smithi15528762-32 2024-04-19T01:04:34.551 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 32 v 9 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-04-19T01:04:34.553 INFO:tasks.rados.rados.0.smithi155.stdout:188: done (1 left) 2024-04-19T01:04:34.553 INFO:tasks.rados.rados.0.smithi155.stdout:191: snap_create 2024-04-19T01:04:34.556 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-04-19T01:04:34.567 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 29 v 39 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-19T01:04:34.582 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:04:34.582+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:04:34.643 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:04:34.642+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:04:34.685 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:34.685 INFO:teuthology.orchestra.run.smithi110.stdout: "size": 20, 2024-04-19T01:04:34.685 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 600, 2024-04-19T01:04:34.686 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [ 2024-04-19T01:04:34.686 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.686 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:181 3.9s0 3:93d266e5:::smithi15528762-34:head [append 0~524288] snapc 6=[6,5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-04-19T01:04:34.686 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:09.817970+0000", 2024-04-19T01:04:34.686 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 24.866110385999999, 2024-04-19T01:04:34.686 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.030101993, 2024-04-19T01:04:34.686 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:34.686 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:34.687 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:34.687 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:34.687 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:34.687 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:34.688 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 181 2024-04-19T01:04:34.688 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.688 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:34.688 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.688 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:34.688 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.817970+0000", 2024-04-19T01:04:34.688 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.688 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.688 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.688 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:34.689 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.817970+0000", 2024-04-19T01:04:34.689 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.689 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.689 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.689 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:34.689 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.817974+0000", 2024-04-19T01:04:34.689 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.546e-06 2024-04-19T01:04:34.689 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.689 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.690 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:34.690 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.819093+0000", 2024-04-19T01:04:34.690 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001119054 2024-04-19T01:04:34.690 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.690 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.691 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:34.691 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.819129+0000", 2024-04-19T01:04:34.691 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.6613999999999999e-05 2024-04-19T01:04:34.691 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.691 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.691 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:34.691 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.819155+0000", 2024-04-19T01:04:34.691 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6089e-05 2024-04-19T01:04:34.691 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.691 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.691 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.692 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.819284+0000", 2024-04-19T01:04:34.692 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000128085 2024-04-19T01:04:34.692 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.692 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.692 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:34.692 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.819408+0000", 2024-04-19T01:04:34.692 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000124058 2024-04-19T01:04:34.692 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.692 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.692 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:34.692 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.820674+0000", 2024-04-19T01:04:34.692 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0012667819999999999 2024-04-19T01:04:34.692 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.693 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.693 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:34.693 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.836303+0000", 2024-04-19T01:04:34.693 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.015628607999999999 2024-04-19T01:04:34.693 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.693 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.693 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:34.693 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.847942+0000", 2024-04-19T01:04:34.693 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.011638895999999999 2024-04-19T01:04:34.693 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.693 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.693 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:34.694 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.848072+0000", 2024-04-19T01:04:34.694 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00013026100000000001 2024-04-19T01:04:34.694 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.694 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.694 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.694 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.694 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.694 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:235 3.9s0 3:93d266e5:::smithi15528762-34:head [append 0~688128] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:34.694 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.857683+0000", 2024-04-19T01:04:34.694 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 20.82639708, 2024-04-19T01:04:34.694 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.044416514999999997, 2024-04-19T01:04:34.694 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:34.695 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:34.695 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:34.695 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:34.695 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:34.695 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:34.695 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 235 2024-04-19T01:04:34.695 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.695 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:34.695 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.695 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:34.695 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.857683+0000", 2024-04-19T01:04:34.695 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.695 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.695 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.696 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:34.696 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.857683+0000", 2024-04-19T01:04:34.696 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.696 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.696 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.696 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:34.696 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.857687+0000", 2024-04-19T01:04:34.697 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.1130000000000001e-06 2024-04-19T01:04:34.697 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.697 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.697 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:34.697 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.859118+0000", 2024-04-19T01:04:34.697 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001431114 2024-04-19T01:04:34.697 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.697 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.697 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:34.697 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.859150+0000", 2024-04-19T01:04:34.697 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.2589000000000003e-05 2024-04-19T01:04:34.697 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.698 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.698 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:34.698 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.860377+0000", 2024-04-19T01:04:34.698 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001226878 2024-04-19T01:04:34.698 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.699 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.699 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.699 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.866898+0000", 2024-04-19T01:04:34.699 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0065209250000000003 2024-04-19T01:04:34.699 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.699 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.699 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:34.699 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.867163+0000", 2024-04-19T01:04:34.699 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000264901 2024-04-19T01:04:34.699 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.699 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.700 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:34.700 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.869905+0000", 2024-04-19T01:04:34.700 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0027423550000000001 2024-04-19T01:04:34.700 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.701 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.701 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:34.701 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.897422+0000", 2024-04-19T01:04:34.701 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.027516445 2024-04-19T01:04:34.701 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.701 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.701 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:34.701 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.901877+0000", 2024-04-19T01:04:34.701 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.004454965 2024-04-19T01:04:34.701 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.701 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.701 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:34.701 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.902100+0000", 2024-04-19T01:04:34.702 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00022322999999999999 2024-04-19T01:04:34.702 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.702 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.702 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.702 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.702 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.702 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:236 3.9s0 3:93d266e5:::smithi15528762-34:head [append 0~376832] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:34.702 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.860599+0000", 2024-04-19T01:04:34.702 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 20.823481059999999, 2024-04-19T01:04:34.702 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.042094899999999998, 2024-04-19T01:04:34.702 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:34.702 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:34.703 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:34.703 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:34.703 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:34.703 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:34.703 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 236 2024-04-19T01:04:34.703 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.703 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:34.703 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.703 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:34.703 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.860599+0000", 2024-04-19T01:04:34.703 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.703 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.703 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.703 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:34.704 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.860599+0000", 2024-04-19T01:04:34.704 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.704 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.704 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.704 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:34.704 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.860602+0000", 2024-04-19T01:04:34.704 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4899999999999999e-06 2024-04-19T01:04:34.704 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.704 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.704 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:34.704 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861320+0000", 2024-04-19T01:04:34.704 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00071774399999999998 2024-04-19T01:04:34.704 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.704 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.704 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:34.705 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861368+0000", 2024-04-19T01:04:34.705 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.8016999999999998e-05 2024-04-19T01:04:34.705 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.705 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.705 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:34.705 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861389+0000", 2024-04-19T01:04:34.705 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1175999999999999e-05 2024-04-19T01:04:34.705 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.705 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.706 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.706 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.888682+0000", 2024-04-19T01:04:34.706 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.027292739 2024-04-19T01:04:34.706 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.706 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.706 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:34.706 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.888867+0000", 2024-04-19T01:04:34.706 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00018496499999999999 2024-04-19T01:04:34.706 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.706 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.706 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:34.707 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.890430+0000", 2024-04-19T01:04:34.707 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001563069 2024-04-19T01:04:34.707 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.707 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.707 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:34.707 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.898524+0000", 2024-04-19T01:04:34.707 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0080939800000000006 2024-04-19T01:04:34.708 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.708 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.708 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:34.708 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.902524+0000", 2024-04-19T01:04:34.708 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0039999930000000003 2024-04-19T01:04:34.708 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.708 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.708 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:34.708 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.902694+0000", 2024-04-19T01:04:34.708 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00017072699999999999 2024-04-19T01:04:34.708 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.708 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.709 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.709 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.709 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.709 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:237 3.9s0 3:93d266e5:::smithi15528762-34:head [setxattr _header (56) in=7b] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:34.709 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.861460+0000", 2024-04-19T01:04:34.709 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 20.822620503, 2024-04-19T01:04:34.709 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.042252307000000003, 2024-04-19T01:04:34.709 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:34.709 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:34.709 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:34.709 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:34.709 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:34.709 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:34.709 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 237 2024-04-19T01:04:34.710 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.710 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:34.710 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.710 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:34.710 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861460+0000", 2024-04-19T01:04:34.710 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.710 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.710 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.710 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:34.714 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861460+0000", 2024-04-19T01:04:34.714 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.714 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.714 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.714 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:34.714 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861462+0000", 2024-04-19T01:04:34.714 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9929999999999998e-06 2024-04-19T01:04:34.714 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.715 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.715 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:34.715 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861472+0000", 2024-04-19T01:04:34.715 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.9070000000000002e-06 2024-04-19T01:04:34.715 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.715 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.715 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:34.715 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861493+0000", 2024-04-19T01:04:34.715 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1386000000000002e-05 2024-04-19T01:04:34.715 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.715 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.715 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:34.715 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861510+0000", 2024-04-19T01:04:34.715 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7198999999999999e-05 2024-04-19T01:04:34.716 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.716 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.716 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.716 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.899576+0000", 2024-04-19T01:04:34.716 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.038065486000000003 2024-04-19T01:04:34.716 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.716 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.716 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:34.716 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.899753+0000", 2024-04-19T01:04:34.716 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000176522 2024-04-19T01:04:34.716 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.716 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.716 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:34.716 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.900579+0000", 2024-04-19T01:04:34.717 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00082676899999999998 2024-04-19T01:04:34.717 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.717 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.717 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:34.717 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.902843+0000", 2024-04-19T01:04:34.717 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0022635329999999999 2024-04-19T01:04:34.717 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.717 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.717 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:34.717 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.903504+0000", 2024-04-19T01:04:34.717 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00066085099999999995 2024-04-19T01:04:34.717 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.717 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.717 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:34.717 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.903712+0000", 2024-04-19T01:04:34.718 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00020866099999999999 2024-04-19T01:04:34.718 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.718 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.718 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.718 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.718 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.718 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:238 3.9s0 3:93d266e5:::smithi15528762-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:34.718 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.861534+0000", 2024-04-19T01:04:34.718 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 20.822546316, 2024-04-19T01:04:34.718 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.046352598000000002, 2024-04-19T01:04:34.718 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:34.718 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:34.718 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:04:34.719 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:34.719 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:34.719 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:34.719 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 238 2024-04-19T01:04:34.719 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.719 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:34.719 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.719 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:34.719 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861534+0000", 2024-04-19T01:04:34.719 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.719 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.719 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.719 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:34.720 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861534+0000", 2024-04-19T01:04:34.720 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.720 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.720 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.720 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:34.720 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861536+0000", 2024-04-19T01:04:34.720 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.607e-06 2024-04-19T01:04:34.720 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.720 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.720 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:34.720 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861542+0000", 2024-04-19T01:04:34.720 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.1689999999999997e-06 2024-04-19T01:04:34.720 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.720 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.721 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:34.721 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861562+0000", 2024-04-19T01:04:34.721 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0043000000000001e-05 2024-04-19T01:04:34.721 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.721 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.721 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:34.721 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861882+0000", 2024-04-19T01:04:34.721 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000320398 2024-04-19T01:04:34.721 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.721 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.721 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.721 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.901266+0000", 2024-04-19T01:04:34.721 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.039383373999999999 2024-04-19T01:04:34.721 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.722 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.722 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting for rw locks", 2024-04-19T01:04:34.722 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.901474+0000", 2024-04-19T01:04:34.722 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00020846999999999999 2024-04-19T01:04:34.722 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.722 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.722 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.722 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.904551+0000", 2024-04-19T01:04:34.722 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0030766690000000002 2024-04-19T01:04:34.722 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.722 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.722 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:34.722 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.904806+0000", 2024-04-19T01:04:34.723 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00025481899999999999 2024-04-19T01:04:34.723 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.723 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.723 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:34.723 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.907887+0000", 2024-04-19T01:04:34.723 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0030810490000000002 2024-04-19T01:04:34.723 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.723 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.723 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.723 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.723 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.723 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 43/31 ECSubWriteReply(tid=67, last_complete=43'39, committed=1, applied=1))", 2024-04-19T01:04:34.724 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.867911+0000", 2024-04-19T01:04:34.724 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 20.816169346999999, 2024-04-19T01:04:34.724 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.029356884, 2024-04-19T01:04:34.724 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:34.724 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:34.724 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:34.724 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:34.724 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.724 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:34.724 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.867911+0000", 2024-04-19T01:04:34.724 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.724 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.724 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.725 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:34.725 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.867911+0000", 2024-04-19T01:04:34.725 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.725 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.725 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.725 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:34.725 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.867912+0000", 2024-04-19T01:04:34.725 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.9300000000000006e-07 2024-04-19T01:04:34.725 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.725 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.725 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:34.725 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.867919+0000", 2024-04-19T01:04:34.725 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.2799999999999998e-06 2024-04-19T01:04:34.725 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.726 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.726 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:34.726 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.867940+0000", 2024-04-19T01:04:34.726 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0576000000000001e-05 2024-04-19T01:04:34.726 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.726 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.726 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:34.726 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.867956+0000", 2024-04-19T01:04:34.726 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5971999999999999e-05 2024-04-19T01:04:34.726 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.726 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.727 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.727 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.897221+0000", 2024-04-19T01:04:34.727 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.029264935999999998 2024-04-19T01:04:34.727 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.727 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.727 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:34.727 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.897268+0000", 2024-04-19T01:04:34.727 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.7126999999999998e-05 2024-04-19T01:04:34.727 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.727 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.727 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.727 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.727 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.728 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs1 43/31 ECSubWrite(tid=47, reqid=client.4604.0:263, at_version=43'8, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:04:34.728 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.994106+0000", 2024-04-19T01:04:34.728 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 20.689974463999999, 2024-04-19T01:04:34.728 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.031552210999999997, 2024-04-19T01:04:34.728 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:34.728 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:34.728 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:34.728 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:34.728 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.728 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:34.728 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.994106+0000", 2024-04-19T01:04:34.728 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.728 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.728 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.729 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:34.729 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.994106+0000", 2024-04-19T01:04:34.729 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.729 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.729 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.729 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:34.729 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.994107+0000", 2024-04-19T01:04:34.729 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.8000000000000004e-07 2024-04-19T01:04:34.729 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.729 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.729 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:34.729 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995101+0000", 2024-04-19T01:04:34.729 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00099370800000000005 2024-04-19T01:04:34.729 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.730 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.730 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:34.730 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995142+0000", 2024-04-19T01:04:34.730 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.0982000000000002e-05 2024-04-19T01:04:34.730 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.730 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.731 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:34.731 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995166+0000", 2024-04-19T01:04:34.731 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4485e-05 2024-04-19T01:04:34.731 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.731 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.731 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.731 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995315+0000", 2024-04-19T01:04:34.731 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00014939 2024-04-19T01:04:34.731 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.731 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.731 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:34.731 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995332+0000", 2024-04-19T01:04:34.731 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6469000000000001e-05 2024-04-19T01:04:34.731 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.732 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.732 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:34.732 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.025589+0000", 2024-04-19T01:04:34.732 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.03025707 2024-04-19T01:04:34.732 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.732 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.732 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:34.732 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.025658+0000", 2024-04-19T01:04:34.732 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.9227000000000007e-05 2024-04-19T01:04:34.732 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.732 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.732 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.732 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.732 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.733 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:267 3.9s0 3:93d266e5:::smithi15528762-34:head [append 0~737280] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:34.733 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.013235+0000", 2024-04-19T01:04:34.733 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 20.670845194999998, 2024-04-19T01:04:34.733 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.043578804999999998, 2024-04-19T01:04:34.733 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:34.733 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:34.733 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:34.733 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:34.733 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:34.733 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:34.733 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 267 2024-04-19T01:04:34.733 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.733 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:34.734 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.734 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:34.734 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.013235+0000", 2024-04-19T01:04:34.734 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.734 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.734 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.734 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:34.734 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.013235+0000", 2024-04-19T01:04:34.734 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.734 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.734 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.734 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:34.734 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.013239+0000", 2024-04-19T01:04:34.735 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.6200000000000001e-06 2024-04-19T01:04:34.735 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.735 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.735 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:34.735 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.014549+0000", 2024-04-19T01:04:34.735 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001310075 2024-04-19T01:04:34.735 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.735 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.735 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:34.735 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.014585+0000", 2024-04-19T01:04:34.735 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.5494999999999998e-05 2024-04-19T01:04:34.735 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.735 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.735 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:34.736 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.014607+0000", 2024-04-19T01:04:34.736 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2371999999999999e-05 2024-04-19T01:04:34.736 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.736 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.736 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.736 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.014744+0000", 2024-04-19T01:04:34.736 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000136988 2024-04-19T01:04:34.736 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.736 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.736 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:34.736 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.015097+0000", 2024-04-19T01:04:34.736 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00035269400000000002 2024-04-19T01:04:34.736 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.737 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.737 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:34.737 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.020447+0000", 2024-04-19T01:04:34.737 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0053503739999999998 2024-04-19T01:04:34.737 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.737 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.737 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:34.737 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.047175+0000", 2024-04-19T01:04:34.737 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.026728372 2024-04-19T01:04:34.737 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.737 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.737 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:34.737 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.056595+0000", 2024-04-19T01:04:34.737 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0094197320000000001 2024-04-19T01:04:34.738 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.738 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.738 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:34.738 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.056814+0000", 2024-04-19T01:04:34.738 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00021908299999999999 2024-04-19T01:04:34.738 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.738 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.738 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.738 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.738 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.738 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:268 3.9s0 3:93d266e5:::smithi15528762-34:head [append 0~376832] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:34.738 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.015660+0000", 2024-04-19T01:04:34.738 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 20.66842016, 2024-04-19T01:04:34.739 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.041602957000000003, 2024-04-19T01:04:34.739 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:34.739 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:34.742 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:34.743 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:34.743 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:34.743 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:34.743 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 268 2024-04-19T01:04:34.743 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.743 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:34.743 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.743 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:34.743 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.015660+0000", 2024-04-19T01:04:34.744 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.744 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.744 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.744 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:34.744 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.015660+0000", 2024-04-19T01:04:34.744 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.744 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.744 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.744 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:34.744 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.015663+0000", 2024-04-19T01:04:34.744 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.0299999999999998e-06 2024-04-19T01:04:34.744 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.745 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.745 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:34.745 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016328+0000", 2024-04-19T01:04:34.745 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00066432000000000004 2024-04-19T01:04:34.745 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.745 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.745 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:34.745 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016362+0000", 2024-04-19T01:04:34.745 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.4709000000000003e-05 2024-04-19T01:04:34.746 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.746 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.746 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:34.746 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016384+0000", 2024-04-19T01:04:34.746 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1172999999999999e-05 2024-04-19T01:04:34.746 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.746 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.746 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.746 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.038612+0000", 2024-04-19T01:04:34.746 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.022227925999999999 2024-04-19T01:04:34.746 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.747 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.747 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:34.747 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.038788+0000", 2024-04-19T01:04:34.747 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000176095 2024-04-19T01:04:34.747 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.747 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.747 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:34.747 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.040120+0000", 2024-04-19T01:04:34.747 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001331917 2024-04-19T01:04:34.747 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.747 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.748 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:34.748 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.047665+0000", 2024-04-19T01:04:34.748 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0075450789999999997 2024-04-19T01:04:34.748 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.748 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.748 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:34.748 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.057085+0000", 2024-04-19T01:04:34.748 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0094201930000000003 2024-04-19T01:04:34.748 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.749 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.749 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:34.749 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.057263+0000", 2024-04-19T01:04:34.749 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00017851499999999999 2024-04-19T01:04:34.749 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.749 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.749 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.750 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.750 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.750 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:269 3.9s0 3:93d266e5:::smithi15528762-34:head [setxattr _header (56) in=7b] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:34.750 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.016434+0000", 2024-04-19T01:04:34.760 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 20.667646198, 2024-04-19T01:04:34.760 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.041381567000000001, 2024-04-19T01:04:34.761 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:34.761 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:34.761 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:34.761 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:34.761 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:34.762 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:34.762 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 269 2024-04-19T01:04:34.762 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.762 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:34.762 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.762 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:34.763 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016434+0000", 2024-04-19T01:04:34.763 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.763 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.763 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.763 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:34.763 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016434+0000", 2024-04-19T01:04:34.763 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.763 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.763 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.764 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:34.764 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016436+0000", 2024-04-19T01:04:34.764 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5939999999999999e-06 2024-04-19T01:04:34.764 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.764 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.764 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:34.764 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016445+0000", 2024-04-19T01:04:34.764 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.9889999999999998e-06 2024-04-19T01:04:34.764 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.764 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.764 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:34.764 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016465+0000", 2024-04-19T01:04:34.765 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0403000000000001e-05 2024-04-19T01:04:34.765 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.765 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.765 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:34.765 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016481+0000", 2024-04-19T01:04:34.765 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5772000000000001e-05 2024-04-19T01:04:34.765 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.765 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.765 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.765 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.047733+0000", 2024-04-19T01:04:34.765 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.031251689999999999 2024-04-19T01:04:34.765 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.765 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.766 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:34.766 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.047885+0000", 2024-04-19T01:04:34.766 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015215300000000001 2024-04-19T01:04:34.766 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.766 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.766 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:34.766 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.048690+0000", 2024-04-19T01:04:34.766 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00080468100000000002 2024-04-19T01:04:34.766 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.766 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.766 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:34.766 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.049873+0000", 2024-04-19T01:04:34.766 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0011833200000000001 2024-04-19T01:04:34.767 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.767 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.767 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:34.767 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.057577+0000", 2024-04-19T01:04:34.767 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0077039860000000003 2024-04-19T01:04:34.767 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.767 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.767 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:34.767 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.057816+0000", 2024-04-19T01:04:34.767 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000238979 2024-04-19T01:04:34.767 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.767 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.768 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.768 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.768 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.768 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:270 3.9s0 3:93d266e5:::smithi15528762-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:34.768 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.016503+0000", 2024-04-19T01:04:34.768 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 20.667577031, 2024-04-19T01:04:34.768 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.044494194000000001, 2024-04-19T01:04:34.768 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:34.768 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:34.768 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:04:34.769 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:34.769 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:34.769 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:34.769 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 270 2024-04-19T01:04:34.769 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.769 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:34.769 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.769 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:34.769 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016503+0000", 2024-04-19T01:04:34.769 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.769 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.769 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.770 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:34.770 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016503+0000", 2024-04-19T01:04:34.770 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.770 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.770 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.770 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:34.771 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016505+0000", 2024-04-19T01:04:34.771 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.2729999999999999e-06 2024-04-19T01:04:34.771 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.771 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.771 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:34.771 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016511+0000", 2024-04-19T01:04:34.771 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.7570000000000001e-06 2024-04-19T01:04:34.771 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.771 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.771 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:34.771 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016530+0000", 2024-04-19T01:04:34.771 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9565999999999999e-05 2024-04-19T01:04:34.772 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.772 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.772 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:34.772 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016545+0000", 2024-04-19T01:04:34.772 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.4963000000000001e-05 2024-04-19T01:04:34.772 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.772 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.772 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.773 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.049233+0000", 2024-04-19T01:04:34.773 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.032687815000000002 2024-04-19T01:04:34.773 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.773 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.773 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting for rw locks", 2024-04-19T01:04:34.773 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.049365+0000", 2024-04-19T01:04:34.773 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00013258399999999999 2024-04-19T01:04:34.773 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.773 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.773 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.774 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.058814+0000", 2024-04-19T01:04:34.774 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0094481080000000002 2024-04-19T01:04:34.774 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.774 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.774 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:34.774 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.059010+0000", 2024-04-19T01:04:34.774 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00019622699999999999 2024-04-19T01:04:34.774 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.774 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.774 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:34.774 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.060998+0000", 2024-04-19T01:04:34.775 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0019879009999999998 2024-04-19T01:04:34.775 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.775 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.775 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.775 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.775 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.775 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:280 3.9s0 3:98647342:::smithi15528762-43:head [append 0~753664] snapc 7=[7,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-19T01:04:34.775 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.869239+0000", 2024-04-19T01:04:34.775 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 19.814841162, 2024-04-19T01:04:34.775 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.062785076999999995, 2024-04-19T01:04:34.775 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:34.775 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:34.776 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:34.776 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:34.776 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:34.776 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:34.776 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 280 2024-04-19T01:04:34.776 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.776 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:34.776 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.776 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:34.776 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.869239+0000", 2024-04-19T01:04:34.776 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.776 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.776 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.776 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:34.777 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.869239+0000", 2024-04-19T01:04:34.777 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.777 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.777 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.777 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:34.777 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.869243+0000", 2024-04-19T01:04:34.777 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.444e-06 2024-04-19T01:04:34.778 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.778 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.778 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:34.778 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.870594+0000", 2024-04-19T01:04:34.778 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001351662 2024-04-19T01:04:34.778 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.778 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.778 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:34.778 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.870629+0000", 2024-04-19T01:04:34.778 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.4329000000000003e-05 2024-04-19T01:04:34.779 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.779 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.779 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:34.779 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.870647+0000", 2024-04-19T01:04:34.779 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7768999999999999e-05 2024-04-19T01:04:34.779 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.779 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.779 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.780 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.870806+0000", 2024-04-19T01:04:34.780 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015962300000000001 2024-04-19T01:04:34.780 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.780 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.780 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "op must wait for map", 2024-04-19T01:04:34.780 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.870819+0000", 2024-04-19T01:04:34.780 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.3266e-05 2024-04-19T01:04:34.780 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.780 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.781 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.781 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875741+0000", 2024-04-19T01:04:34.781 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0049211599999999999 2024-04-19T01:04:34.781 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.781 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.781 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:34.781 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.876018+0000", 2024-04-19T01:04:34.781 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000276944 2024-04-19T01:04:34.782 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.782 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.782 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:34.782 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.880141+0000", 2024-04-19T01:04:34.782 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0041231030000000004 2024-04-19T01:04:34.782 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.782 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.782 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:34.782 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.915254+0000", 2024-04-19T01:04:34.782 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.035113610000000003 2024-04-19T01:04:34.782 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.782 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.782 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:34.783 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.931834+0000", 2024-04-19T01:04:34.783 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.016580212 2024-04-19T01:04:34.783 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.783 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.783 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:34.783 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.932024+0000", 2024-04-19T01:04:34.783 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00018995500000000001 2024-04-19T01:04:34.783 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.783 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.783 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.783 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.784 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.784 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:281 3.9s0 3:98647342:::smithi15528762-43:head [append 0~376832] snapc 7=[7,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-19T01:04:34.784 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.872157+0000", 2024-04-19T01:04:34.784 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 19.811923538999999, 2024-04-19T01:04:34.784 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.061221235999999998, 2024-04-19T01:04:34.784 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:34.784 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:34.784 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:34.784 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:34.784 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:34.784 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:34.785 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 281 2024-04-19T01:04:34.785 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.785 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:34.785 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.785 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:34.785 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872157+0000", 2024-04-19T01:04:34.785 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.786 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.786 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.786 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:34.786 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872157+0000", 2024-04-19T01:04:34.786 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.786 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.786 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.786 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:34.786 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872159+0000", 2024-04-19T01:04:34.786 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5129999999999999e-06 2024-04-19T01:04:34.786 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.787 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.787 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:34.787 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872435+0000", 2024-04-19T01:04:34.787 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00027559500000000001 2024-04-19T01:04:34.787 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.787 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.787 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:34.787 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872470+0000", 2024-04-19T01:04:34.787 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.5170999999999999e-05 2024-04-19T01:04:34.787 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.788 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.788 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:34.788 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872488+0000", 2024-04-19T01:04:34.788 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.8136000000000001e-05 2024-04-19T01:04:34.788 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.788 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.789 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.789 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872639+0000", 2024-04-19T01:04:34.789 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015092 2024-04-19T01:04:34.789 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.789 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.789 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:04:34.789 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872658+0000", 2024-04-19T01:04:34.789 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.8882999999999999e-05 2024-04-19T01:04:34.789 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.789 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.789 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.789 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.903660+0000", 2024-04-19T01:04:34.789 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.031001846999999999 2024-04-19T01:04:34.789 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.790 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.790 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:34.790 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.904268+0000", 2024-04-19T01:04:34.790 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00060793999999999996 2024-04-19T01:04:34.790 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.790 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.790 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:34.790 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.906536+0000", 2024-04-19T01:04:34.790 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0022680180000000001 2024-04-19T01:04:34.790 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.790 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.790 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:34.790 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.918698+0000", 2024-04-19T01:04:34.791 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.012161852000000001 2024-04-19T01:04:34.791 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.791 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.791 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:34.791 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.933228+0000", 2024-04-19T01:04:34.791 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.014529781 2024-04-19T01:04:34.791 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.791 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.792 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:34.792 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.933378+0000", 2024-04-19T01:04:34.792 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015058000000000001 2024-04-19T01:04:34.792 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.792 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.792 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.792 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.792 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.792 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:282 3.9s0 3:98647342:::smithi15528762-43:head [setxattr _header (56) in=7b] snapc 7=[7,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-19T01:04:34.792 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.872536+0000", 2024-04-19T01:04:34.792 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 19.811544036000001, 2024-04-19T01:04:34.793 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.062122945999999998, 2024-04-19T01:04:34.793 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:34.793 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:34.793 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:34.793 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:34.793 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:34.793 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:34.793 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 282 2024-04-19T01:04:34.793 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.793 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:34.793 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.794 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:34.794 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872536+0000", 2024-04-19T01:04:34.794 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.794 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.794 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.794 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:34.794 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872536+0000", 2024-04-19T01:04:34.794 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.794 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.794 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.794 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:34.794 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872538+0000", 2024-04-19T01:04:34.795 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.423e-06 2024-04-19T01:04:34.795 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.795 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.795 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:34.795 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872548+0000", 2024-04-19T01:04:34.795 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.8430000000000005e-06 2024-04-19T01:04:34.795 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.795 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.795 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:34.795 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872569+0000", 2024-04-19T01:04:34.795 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1336e-05 2024-04-19T01:04:34.795 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.796 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.796 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:34.796 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872585+0000", 2024-04-19T01:04:34.796 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6288999999999999e-05 2024-04-19T01:04:34.796 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.796 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.796 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.796 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872783+0000", 2024-04-19T01:04:34.796 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000197488 2024-04-19T01:04:34.796 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.796 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.796 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:04:34.797 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872804+0000", 2024-04-19T01:04:34.797 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0839000000000002e-05 2024-04-19T01:04:34.797 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.797 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.797 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.797 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.915520+0000", 2024-04-19T01:04:34.797 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.042715887000000001 2024-04-19T01:04:34.800 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.800 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.800 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:34.800 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.915852+0000", 2024-04-19T01:04:34.800 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00033199899999999998 2024-04-19T01:04:34.800 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.800 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.800 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:34.801 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.916938+0000", 2024-04-19T01:04:34.801 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0010863940000000001 2024-04-19T01:04:34.801 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.801 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.801 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:34.801 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.918701+0000", 2024-04-19T01:04:34.801 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001763351 2024-04-19T01:04:34.801 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.801 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.801 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:34.801 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.934283+0000", 2024-04-19T01:04:34.802 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.015581526 2024-04-19T01:04:34.802 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.802 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.802 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:34.802 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.934659+0000", 2024-04-19T01:04:34.802 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00037657100000000002 2024-04-19T01:04:34.802 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.803 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.803 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.803 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.803 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.803 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:283 3.9s0 3:98647342:::smithi15528762-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-04-19T01:04:34.803 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.872609+0000", 2024-04-19T01:04:34.803 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 19.811471185999999, 2024-04-19T01:04:34.803 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.067525222999999995, 2024-04-19T01:04:34.803 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:34.803 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:34.803 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:04:34.803 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:34.803 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:34.803 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:34.804 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 283 2024-04-19T01:04:34.804 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.804 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:34.804 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.804 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:34.804 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872609+0000", 2024-04-19T01:04:34.804 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.804 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.804 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.804 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:34.804 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872609+0000", 2024-04-19T01:04:34.804 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.804 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.805 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.805 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:34.805 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872611+0000", 2024-04-19T01:04:34.805 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.367e-06 2024-04-19T01:04:34.805 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.805 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.805 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:34.805 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872617+0000", 2024-04-19T01:04:34.805 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.2400000000000004e-06 2024-04-19T01:04:34.805 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.805 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.805 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:34.805 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872636+0000", 2024-04-19T01:04:34.806 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.8941999999999999e-05 2024-04-19T01:04:34.806 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.806 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.806 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:34.806 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872652+0000", 2024-04-19T01:04:34.806 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5651999999999999e-05 2024-04-19T01:04:34.806 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.806 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.806 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.807 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872858+0000", 2024-04-19T01:04:34.807 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00020631100000000001 2024-04-19T01:04:34.807 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.807 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.807 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:04:34.807 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872867+0000", 2024-04-19T01:04:34.807 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.5400000000000001e-06 2024-04-19T01:04:34.807 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.807 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.807 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.807 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.917636+0000", 2024-04-19T01:04:34.807 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.044768372000000001 2024-04-19T01:04:34.808 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.808 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.808 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting for rw locks", 2024-04-19T01:04:34.808 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.917899+0000", 2024-04-19T01:04:34.808 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00026336799999999999 2024-04-19T01:04:34.808 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.808 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.808 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.808 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.935711+0000", 2024-04-19T01:04:34.809 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.017811862000000001 2024-04-19T01:04:34.809 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.809 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.809 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:34.809 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.935937+0000", 2024-04-19T01:04:34.809 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00022568800000000001 2024-04-19T01:04:34.809 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.809 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.809 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:34.809 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.940135+0000", 2024-04-19T01:04:34.809 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0041978809999999997 2024-04-19T01:04:34.810 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.810 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.810 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.810 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.810 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.810 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs1 48/31 ECSubWrite(tid=76, reqid=client.4604.0:345, at_version=48'22, trim_to=0'0, roll_forward_to=47'19))", 2024-04-19T01:04:34.810 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.538686+0000", 2024-04-19T01:04:34.810 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 16.145394582000002, 2024-04-19T01:04:34.810 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.040628868999999998, 2024-04-19T01:04:34.810 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:34.810 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:34.810 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:34.811 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:34.811 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.811 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:34.811 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538686+0000", 2024-04-19T01:04:34.811 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.811 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.811 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.811 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:34.811 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538686+0000", 2024-04-19T01:04:34.811 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.811 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.811 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.812 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:34.812 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538688+0000", 2024-04-19T01:04:34.812 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5799999999999999e-06 2024-04-19T01:04:34.812 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.812 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.812 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:34.812 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.539128+0000", 2024-04-19T01:04:34.812 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00044051499999999999 2024-04-19T01:04:34.812 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.812 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.812 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:34.813 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.539160+0000", 2024-04-19T01:04:34.813 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.2255999999999997e-05 2024-04-19T01:04:34.813 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.813 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.813 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:34.813 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.539184+0000", 2024-04-19T01:04:34.813 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3699000000000001e-05 2024-04-19T01:04:34.813 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.813 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.813 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.813 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.539555+0000", 2024-04-19T01:04:34.813 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00037067799999999999 2024-04-19T01:04:34.813 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.814 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.814 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:34.814 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.539639+0000", 2024-04-19T01:04:34.814 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.4625999999999999e-05 2024-04-19T01:04:34.814 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.814 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.814 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:34.814 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.579272+0000", 2024-04-19T01:04:34.814 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.039632435000000001 2024-04-19T01:04:34.814 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.814 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.814 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:34.815 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.579315+0000", 2024-04-19T01:04:34.815 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.3080000000000001e-05 2024-04-19T01:04:34.815 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.815 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.815 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.815 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.815 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.815 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:351 3.9s0 3:98647342:::smithi15528762-43:head [rollback 7] snapc 9=[8,7] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:04:34.815 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.542794+0000", 2024-04-19T01:04:34.815 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 16.141286296000001, 2024-04-19T01:04:34.815 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.031888056999999997, 2024-04-19T01:04:34.815 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:34.815 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:34.815 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:34.816 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:34.816 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:34.816 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:34.816 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 351 2024-04-19T01:04:34.816 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.816 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:34.816 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.816 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:34.816 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.542794+0000", 2024-04-19T01:04:34.816 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.816 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.816 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.816 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:34.816 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.542794+0000", 2024-04-19T01:04:34.817 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.817 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.817 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.817 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:34.817 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.542796+0000", 2024-04-19T01:04:34.817 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5200000000000001e-06 2024-04-19T01:04:34.817 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.817 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.817 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:34.817 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.542803+0000", 2024-04-19T01:04:34.817 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.776e-06 2024-04-19T01:04:34.817 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.818 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.818 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:34.818 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.542823+0000", 2024-04-19T01:04:34.818 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0285999999999999e-05 2024-04-19T01:04:34.818 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.819 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.819 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:34.819 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.542838+0000", 2024-04-19T01:04:34.819 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5308999999999999e-05 2024-04-19T01:04:34.819 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.819 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.819 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.819 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.549798+0000", 2024-04-19T01:04:34.819 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0069602739999999998 2024-04-19T01:04:34.819 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.819 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.820 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:34.820 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.550134+0000", 2024-04-19T01:04:34.820 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00033588900000000002 2024-04-19T01:04:34.820 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.820 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.820 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:34.820 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.560873+0000", 2024-04-19T01:04:34.820 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.010739149999999999 2024-04-19T01:04:34.820 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.820 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.820 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:34.820 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.574422+0000", 2024-04-19T01:04:34.820 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.013548350000000001 2024-04-19T01:04:34.821 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.821 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.821 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:34.821 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.574538+0000", 2024-04-19T01:04:34.821 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000116162 2024-04-19T01:04:34.821 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.821 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.822 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:34.822 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.574682+0000", 2024-04-19T01:04:34.822 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00014434100000000001 2024-04-19T01:04:34.822 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.822 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.822 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.822 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.822 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.822 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:352 3.9s0 3:98647342:::smithi15528762-43:head [append 0~0] snapc 9=[8,7] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:04:34.822 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.542861+0000", 2024-04-19T01:04:34.823 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 16.141219532000001, 2024-04-19T01:04:34.823 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.033209177999999999, 2024-04-19T01:04:34.823 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:34.823 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:34.823 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:34.823 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:34.823 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:34.823 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:34.823 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 352 2024-04-19T01:04:34.823 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.823 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:34.823 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.823 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:34.824 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.542861+0000", 2024-04-19T01:04:34.824 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.824 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.824 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.824 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:34.824 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.542861+0000", 2024-04-19T01:04:34.824 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.824 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.824 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.824 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:34.824 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.542862+0000", 2024-04-19T01:04:34.824 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.2130000000000001e-06 2024-04-19T01:04:34.824 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.825 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.825 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:34.825 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.542868+0000", 2024-04-19T01:04:34.825 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.4770000000000002e-06 2024-04-19T01:04:34.825 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.825 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.825 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:34.825 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.542886+0000", 2024-04-19T01:04:34.825 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.8119000000000001e-05 2024-04-19T01:04:34.825 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.825 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.825 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:34.825 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.542900+0000", 2024-04-19T01:04:34.826 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.4083000000000001e-05 2024-04-19T01:04:34.826 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.826 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.826 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.826 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.562653+0000", 2024-04-19T01:04:34.826 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.019752998000000001 2024-04-19T01:04:34.826 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.826 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.826 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:34.826 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.563024+0000", 2024-04-19T01:04:34.826 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00037130000000000003 2024-04-19T01:04:34.826 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.826 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.827 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:34.827 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.564055+0000", 2024-04-19T01:04:34.827 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0010311000000000001 2024-04-19T01:04:34.827 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.827 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.827 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:34.827 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.574688+0000", 2024-04-19T01:04:34.827 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.010632561 2024-04-19T01:04:34.827 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.827 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.827 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:34.827 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.574784+0000", 2024-04-19T01:04:34.827 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.6129000000000001e-05 2024-04-19T01:04:34.828 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.828 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.828 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:34.828 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.576070+0000", 2024-04-19T01:04:34.828 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0012861979999999999 2024-04-19T01:04:34.828 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.828 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.828 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.828 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.828 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.828 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:465 3.9s0 3:971ff57a:::smithi15528762-32:head [copy-from ver 9] snapc 11=[11,10,d] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-19T01:04:34.828 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:33.550965+0000", 2024-04-19T01:04:34.828 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 1.133115595, 2024-04-19T01:04:34.828 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.035302247000000002, 2024-04-19T01:04:34.829 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:34.829 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:34.829 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:34.829 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:34.829 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:34.829 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:34.829 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 465 2024-04-19T01:04:34.829 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.829 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:34.829 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.829 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:34.829 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.550965+0000", 2024-04-19T01:04:34.830 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.830 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.830 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.830 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:34.830 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.550965+0000", 2024-04-19T01:04:34.830 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.830 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.830 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.830 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:34.830 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.550968+0000", 2024-04-19T01:04:34.831 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.0599999999999999e-06 2024-04-19T01:04:34.831 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.831 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.831 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:34.831 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.550983+0000", 2024-04-19T01:04:34.831 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5396999999999998e-05 2024-04-19T01:04:34.831 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.831 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.831 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:34.831 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.551016+0000", 2024-04-19T01:04:34.831 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.2305000000000003e-05 2024-04-19T01:04:34.831 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.831 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.832 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:34.832 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.551032+0000", 2024-04-19T01:04:34.832 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6730000000000001e-05 2024-04-19T01:04:34.832 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.832 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.832 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.832 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.551143+0000", 2024-04-19T01:04:34.832 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000110433 2024-04-19T01:04:34.832 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.832 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.832 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:34.832 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.551331+0000", 2024-04-19T01:04:34.832 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000188462 2024-04-19T01:04:34.832 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.833 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.833 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:34.833 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.565853+0000", 2024-04-19T01:04:34.833 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.014521479 2024-04-19T01:04:34.833 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.833 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.833 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:34.833 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.573502+0000", 2024-04-19T01:04:34.833 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0076495579999999999 2024-04-19T01:04:34.833 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.833 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.833 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:34.833 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.586193+0000", 2024-04-19T01:04:34.834 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.012690450000000001 2024-04-19T01:04:34.834 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.834 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.834 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:34.834 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.586267+0000", 2024-04-19T01:04:34.834 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.4373000000000001e-05 2024-04-19T01:04:34.834 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.834 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.834 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.834 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.834 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.834 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:465 3.9s0 3:971ff57a:::smithi15528762-32:head [copy-from ver 9] snapc 11=[11,10,d] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e63)", 2024-04-19T01:04:34.835 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:33.552705+0000", 2024-04-19T01:04:34.835 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 1.131375813, 2024-04-19T01:04:34.835 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.033497738999999999, 2024-04-19T01:04:34.835 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:34.835 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:34.835 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "waiting for ondisk", 2024-04-19T01:04:34.835 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:34.835 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:34.835 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:34.835 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 465 2024-04-19T01:04:34.835 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.835 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:34.835 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.835 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:34.835 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.552705+0000", 2024-04-19T01:04:34.836 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.836 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.836 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.836 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:34.836 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.552705+0000", 2024-04-19T01:04:34.836 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:34.836 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.836 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.836 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:34.836 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.552706+0000", 2024-04-19T01:04:34.837 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7e-06 2024-04-19T01:04:34.837 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.837 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.837 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:34.837 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.552714+0000", 2024-04-19T01:04:34.837 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.4800000000000004e-06 2024-04-19T01:04:34.837 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.837 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.837 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:34.837 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.552737+0000", 2024-04-19T01:04:34.838 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3048999999999999e-05 2024-04-19T01:04:34.838 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.838 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.838 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:34.838 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.552761+0000", 2024-04-19T01:04:34.838 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4266000000000001e-05 2024-04-19T01:04:34.838 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.838 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.838 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.838 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.552862+0000", 2024-04-19T01:04:34.838 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00010106399999999999 2024-04-19T01:04:34.838 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.838 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.838 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting for blocked object", 2024-04-19T01:04:34.839 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.552952+0000", 2024-04-19T01:04:34.839 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.9524000000000005e-05 2024-04-19T01:04:34.839 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.839 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.839 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:34.839 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.572361+0000", 2024-04-19T01:04:34.839 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.019409182000000001 2024-04-19T01:04:34.839 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.839 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.839 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting for ondisk", 2024-04-19T01:04:34.839 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.572613+0000", 2024-04-19T01:04:34.839 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00025244999999999998 2024-04-19T01:04:34.839 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:34.839 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:34.840 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:34.840 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.586202+0000", 2024-04-19T01:04:34.840 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.013589024 2024-04-19T01:04:34.840 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.840 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.840 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.840 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:34.840 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:34.840 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:34.842 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-19T01:04:34.843 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:04:34.841+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:04:34.942 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:04:34.942+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:04:34.972 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:04:34.972 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-04-19T01:04:34.972 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-04-19T01:04:34.972 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:04:34.985 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-19T01:04:35.044 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:04:35.045+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:04:35.104 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:04:35.104 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-04-19T01:04:35.104 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-04-19T01:04:35.105 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-04-19T01:04:35.105 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:04:35.118 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-19T01:04:35.144 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:04:35.142+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:04:35.245 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:04:35.246 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-04-19T01:04:35.246 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-04-19T01:04:35.246 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-04-19T01:04:35.246 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.246 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.3s2 38/31 ECSubWrite(tid=13, reqid=client.4604.0:173, at_version=38'9, trim_to=0'0, roll_forward_to=34'7))", 2024-04-19T01:04:35.246 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:08.824914+0000", 2024-04-19T01:04:35.246 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 26.420369206, 2024-04-19T01:04:35.246 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015651700000000001, 2024-04-19T01:04:35.246 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:35.247 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:35.247 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:35.247 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:35.247 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.247 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:35.247 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:08.824914+0000", 2024-04-19T01:04:35.247 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.247 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.247 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.247 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:35.247 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:08.824914+0000", 2024-04-19T01:04:35.247 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.247 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.247 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.248 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:35.248 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:08.824915+0000", 2024-04-19T01:04:35.248 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.046e-06 2024-04-19T01:04:35.248 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.248 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.248 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:35.248 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:08.825330+0000", 2024-04-19T01:04:35.248 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00041453399999999999 2024-04-19T01:04:35.248 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.248 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.248 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:35.248 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:08.825359+0000", 2024-04-19T01:04:35.248 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9159e-05 2024-04-19T01:04:35.249 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.249 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.249 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:35.249 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:08.825377+0000", 2024-04-19T01:04:35.249 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8145999999999999e-05 2024-04-19T01:04:35.249 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.249 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.249 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:35.250 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:08.825501+0000", 2024-04-19T01:04:35.250 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012435899999999999 2024-04-19T01:04:35.250 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.250 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.250 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:35.250 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:08.825519+0000", 2024-04-19T01:04:35.250 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7909000000000001e-05 2024-04-19T01:04:35.250 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.250 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.250 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:35.250 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:08.840461+0000", 2024-04-19T01:04:35.250 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014941538000000001 2024-04-19T01:04:35.250 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.251 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.251 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:35.251 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:08.840566+0000", 2024-04-19T01:04:35.251 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000105009 2024-04-19T01:04:35.251 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.251 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.251 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.251 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.251 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.251 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 43/31 ECSubWrite(tid=43, reqid=client.4604.0:251, at_version=43'29, trim_to=0'0, roll_forward_to=43'27))", 2024-04-19T01:04:35.251 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.986035+0000", 2024-04-19T01:04:35.251 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.259248357000001, 2024-04-19T01:04:35.251 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024180535999999999, 2024-04-19T01:04:35.251 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:35.252 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:35.252 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:35.252 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:35.252 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.252 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:35.252 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986035+0000", 2024-04-19T01:04:35.252 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.252 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.252 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.252 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:35.252 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986035+0000", 2024-04-19T01:04:35.253 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.253 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.253 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.253 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:35.253 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986058+0000", 2024-04-19T01:04:35.253 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2903000000000001e-05 2024-04-19T01:04:35.253 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.253 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.253 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:35.253 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986841+0000", 2024-04-19T01:04:35.253 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00078285599999999996 2024-04-19T01:04:35.253 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.254 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.263 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:35.264 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986887+0000", 2024-04-19T01:04:35.264 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6202000000000003e-05 2024-04-19T01:04:35.264 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.264 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.264 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:35.264 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986913+0000", 2024-04-19T01:04:35.264 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6162000000000001e-05 2024-04-19T01:04:35.264 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.264 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.264 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:35.264 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987069+0000", 2024-04-19T01:04:35.264 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015625099999999999 2024-04-19T01:04:35.265 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.265 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.265 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:35.265 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987099+0000", 2024-04-19T01:04:35.265 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9578999999999999e-05 2024-04-19T01:04:35.265 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.265 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.265 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:35.265 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.010110+0000", 2024-04-19T01:04:35.265 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023011357999999999 2024-04-19T01:04:35.266 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.266 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.266 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:35.266 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.010215+0000", 2024-04-19T01:04:35.266 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000105225 2024-04-19T01:04:35.266 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.266 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.266 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.266 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.266 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.266 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds5 43/31 ECSubWrite(tid=38, reqid=client.4604.0:253, at_version=43'14, trim_to=0'0, roll_forward_to=43'12))", 2024-04-19T01:04:35.267 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.987245+0000", 2024-04-19T01:04:35.267 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.258038334999998, 2024-04-19T01:04:35.267 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022838027, 2024-04-19T01:04:35.267 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:35.267 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:35.267 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:35.267 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:35.267 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.267 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:35.267 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987245+0000", 2024-04-19T01:04:35.267 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.267 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.267 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.268 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:35.268 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987245+0000", 2024-04-19T01:04:35.268 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.268 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.269 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.269 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:35.269 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987247+0000", 2024-04-19T01:04:35.269 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2800000000000002e-06 2024-04-19T01:04:35.270 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.270 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.270 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:35.270 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988142+0000", 2024-04-19T01:04:35.270 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00089449199999999999 2024-04-19T01:04:35.270 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.270 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.270 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:35.271 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988180+0000", 2024-04-19T01:04:35.271 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8615000000000002e-05 2024-04-19T01:04:35.271 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.271 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.271 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:35.271 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988206+0000", 2024-04-19T01:04:35.271 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6023000000000001e-05 2024-04-19T01:04:35.271 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.271 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.271 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:35.271 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988349+0000", 2024-04-19T01:04:35.271 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014252199999999999 2024-04-19T01:04:35.271 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.271 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.272 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:35.272 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988365+0000", 2024-04-19T01:04:35.272 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5855e-05 2024-04-19T01:04:35.272 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.272 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.272 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:35.272 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.010029+0000", 2024-04-19T01:04:35.273 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.021664454999999999 2024-04-19T01:04:35.273 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.273 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.273 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:35.273 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.010083+0000", 2024-04-19T01:04:35.273 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3785000000000001e-05 2024-04-19T01:04:35.273 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.273 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.273 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.273 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.273 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.273 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.fs6 43/31 ECSubWrite(tid=47, reqid=client.4604.0:263, at_version=43'8, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:04:35.273 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.994092+0000", 2024-04-19T01:04:35.273 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.251191349999999, 2024-04-19T01:04:35.273 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.027264745999999999, 2024-04-19T01:04:35.274 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:35.274 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:35.274 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:35.274 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:35.274 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.274 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:35.274 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.994092+0000", 2024-04-19T01:04:35.274 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.274 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.274 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.275 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:35.275 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.994092+0000", 2024-04-19T01:04:35.275 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.275 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.275 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.276 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:35.276 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.994094+0000", 2024-04-19T01:04:35.276 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1399999999999998e-06 2024-04-19T01:04:35.276 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.276 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.276 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:35.276 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.994786+0000", 2024-04-19T01:04:35.276 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00069247999999999998 2024-04-19T01:04:35.276 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.276 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.277 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:35.277 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.994827+0000", 2024-04-19T01:04:35.277 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0105000000000003e-05 2024-04-19T01:04:35.277 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.277 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.277 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:35.277 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.994853+0000", 2024-04-19T01:04:35.277 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6259e-05 2024-04-19T01:04:35.277 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.277 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.277 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:35.277 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.995043+0000", 2024-04-19T01:04:35.277 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0001906 2024-04-19T01:04:35.278 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.278 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.278 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:35.278 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.995069+0000", 2024-04-19T01:04:35.278 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5633e-05 2024-04-19T01:04:35.278 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.278 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.278 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:35.278 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.021288+0000", 2024-04-19T01:04:35.278 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.026218865000000001 2024-04-19T01:04:35.278 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.279 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.279 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:35.279 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.021357+0000", 2024-04-19T01:04:35.279 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8664000000000005e-05 2024-04-19T01:04:35.279 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.279 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.279 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.279 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.279 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.279 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds5 43/31 ECSubWrite(tid=39, reqid=client.4604.0:254, at_version=43'15, trim_to=0'0, roll_forward_to=43'12))", 2024-04-19T01:04:35.279 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.997374+0000", 2024-04-19T01:04:35.279 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.247909499999999, 2024-04-19T01:04:35.279 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013084780000000001, 2024-04-19T01:04:35.279 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:35.279 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:35.280 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:35.280 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:35.280 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.280 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:35.280 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.997374+0000", 2024-04-19T01:04:35.280 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.280 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.280 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.280 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:35.280 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.997374+0000", 2024-04-19T01:04:35.280 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.280 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.280 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.280 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:35.281 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.997376+0000", 2024-04-19T01:04:35.281 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9e-06 2024-04-19T01:04:35.281 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.281 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.281 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:35.281 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.997615+0000", 2024-04-19T01:04:35.281 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00023961100000000001 2024-04-19T01:04:35.281 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.281 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.281 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:35.282 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.997664+0000", 2024-04-19T01:04:35.282 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8476000000000001e-05 2024-04-19T01:04:35.282 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.282 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.282 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:35.282 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.997689+0000", 2024-04-19T01:04:35.282 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5109e-05 2024-04-19T01:04:35.283 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.283 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.283 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:35.283 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.001350+0000", 2024-04-19T01:04:35.283 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0036615749999999998 2024-04-19T01:04:35.283 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.283 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.283 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:35.284 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.001375+0000", 2024-04-19T01:04:35.284 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5032999999999999e-05 2024-04-19T01:04:35.284 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.284 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.284 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:35.284 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.010366+0000", 2024-04-19T01:04:35.284 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0089907159999999993 2024-04-19T01:04:35.284 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.284 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.284 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:35.284 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.010458+0000", 2024-04-19T01:04:35.284 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2360000000000003e-05 2024-04-19T01:04:35.285 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.285 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.285 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.285 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.285 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.285 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.fs6 43/31 ECSubWrite(tid=48, reqid=client.4604.0:264, at_version=43'9, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:04:35.285 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.006377+0000", 2024-04-19T01:04:35.285 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.238906134, 2024-04-19T01:04:35.285 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015258295, 2024-04-19T01:04:35.285 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:35.285 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:35.285 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:35.285 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:35.285 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.286 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:35.286 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.006377+0000", 2024-04-19T01:04:35.286 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.286 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.286 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.286 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:35.286 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.006377+0000", 2024-04-19T01:04:35.286 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.286 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.286 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.286 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:35.286 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.006379+0000", 2024-04-19T01:04:35.286 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.114e-06 2024-04-19T01:04:35.286 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.287 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.287 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:35.287 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.006643+0000", 2024-04-19T01:04:35.287 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000263854 2024-04-19T01:04:35.287 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.287 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.287 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:35.287 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.006677+0000", 2024-04-19T01:04:35.287 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4388999999999999e-05 2024-04-19T01:04:35.287 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.287 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.287 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:35.287 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.006700+0000", 2024-04-19T01:04:35.287 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2849000000000001e-05 2024-04-19T01:04:35.288 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.288 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.288 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:35.288 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.011877+0000", 2024-04-19T01:04:35.288 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0051769679999999997 2024-04-19T01:04:35.288 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.288 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.288 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:35.288 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.011902+0000", 2024-04-19T01:04:35.288 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4926000000000002e-05 2024-04-19T01:04:35.288 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.288 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.288 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:35.288 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.021480+0000", 2024-04-19T01:04:35.289 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0095781359999999992 2024-04-19T01:04:35.289 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.289 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.289 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:35.289 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.021635+0000", 2024-04-19T01:04:35.289 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015505900000000001 2024-04-19T01:04:35.289 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.289 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.289 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.289 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.289 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.289 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.fs6 43/31 ECSubWrite(tid=49, reqid=client.4604.0:265, at_version=43'10, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:04:35.289 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.006754+0000", 2024-04-19T01:04:35.289 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.238529539000002, 2024-04-19T01:04:35.290 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015562184, 2024-04-19T01:04:35.290 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:35.290 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:35.290 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:35.290 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:35.290 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.290 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:35.290 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.006754+0000", 2024-04-19T01:04:35.290 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.290 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.290 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.290 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:35.290 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.006754+0000", 2024-04-19T01:04:35.291 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.291 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.291 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.291 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:35.291 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.006755+0000", 2024-04-19T01:04:35.291 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7000000000000003e-07 2024-04-19T01:04:35.291 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.291 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.291 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:35.291 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.006813+0000", 2024-04-19T01:04:35.291 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8227999999999999e-05 2024-04-19T01:04:35.291 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.291 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.292 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:35.292 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.006836+0000", 2024-04-19T01:04:35.292 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2949e-05 2024-04-19T01:04:35.292 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.292 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.292 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:35.292 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.006853+0000", 2024-04-19T01:04:35.292 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7133e-05 2024-04-19T01:04:35.292 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.292 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.292 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:35.292 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.020855+0000", 2024-04-19T01:04:35.292 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014002199999999999 2024-04-19T01:04:35.292 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.292 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.293 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:35.293 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.020875+0000", 2024-04-19T01:04:35.293 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0273999999999999e-05 2024-04-19T01:04:35.293 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.293 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.293 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:35.293 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.022241+0000", 2024-04-19T01:04:35.293 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001365556 2024-04-19T01:04:35.293 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.293 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.293 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:35.293 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.022316+0000", 2024-04-19T01:04:35.294 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4874000000000002e-05 2024-04-19T01:04:35.294 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.294 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.294 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.295 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.295 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.295 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.es4 47/31 ECSubWrite(tid=104, reqid=client.4604.0:312, at_version=47'18, trim_to=0'0, roll_forward_to=45'16))", 2024-04-19T01:04:35.295 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.582390+0000", 2024-04-19T01:04:35.296 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 17.662893569000001, 2024-04-19T01:04:35.296 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022795626999999999, 2024-04-19T01:04:35.296 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:35.296 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:35.296 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:35.296 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:35.296 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.297 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:35.297 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.582390+0000", 2024-04-19T01:04:35.297 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.297 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.297 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.297 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:35.297 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.582390+0000", 2024-04-19T01:04:35.297 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.298 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.298 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.298 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:35.298 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.582390+0000", 2024-04-19T01:04:35.298 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8299999999999996e-07 2024-04-19T01:04:35.298 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.298 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.299 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:35.299 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.582798+0000", 2024-04-19T01:04:35.299 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00040746499999999998 2024-04-19T01:04:35.299 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.299 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.299 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:35.299 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.582829+0000", 2024-04-19T01:04:35.299 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0988999999999998e-05 2024-04-19T01:04:35.300 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.300 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.300 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:35.300 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.582846+0000", 2024-04-19T01:04:35.301 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7252e-05 2024-04-19T01:04:35.302 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.302 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.302 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:35.302 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.582989+0000", 2024-04-19T01:04:35.302 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014342000000000001 2024-04-19T01:04:35.303 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.303 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.303 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:35.303 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.583006+0000", 2024-04-19T01:04:35.303 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6671999999999999e-05 2024-04-19T01:04:35.303 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.303 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.304 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:35.304 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.605104+0000", 2024-04-19T01:04:35.304 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022098222000000001 2024-04-19T01:04:35.304 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.304 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.304 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:35.304 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.605185+0000", 2024-04-19T01:04:35.304 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0823999999999997e-05 2024-04-19T01:04:35.304 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.305 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.306 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.306 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.306 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.306 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.es4 47/31 ECSubWrite(tid=105, reqid=client.4604.0:313, at_version=47'19, trim_to=0'0, roll_forward_to=45'16))", 2024-04-19T01:04:35.306 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.591244+0000", 2024-04-19T01:04:35.306 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 17.654038824000001, 2024-04-19T01:04:35.306 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013988917, 2024-04-19T01:04:35.307 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:35.307 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:35.307 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:35.307 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:35.307 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.307 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:35.307 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.591244+0000", 2024-04-19T01:04:35.307 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.307 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.307 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.307 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:35.307 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.591244+0000", 2024-04-19T01:04:35.307 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.308 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.308 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.308 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:35.308 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.591246+0000", 2024-04-19T01:04:35.308 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.73e-06 2024-04-19T01:04:35.308 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.308 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.308 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:35.308 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.591388+0000", 2024-04-19T01:04:35.308 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000141602 2024-04-19T01:04:35.308 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.308 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.308 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:35.308 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.591413+0000", 2024-04-19T01:04:35.308 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5133000000000001e-05 2024-04-19T01:04:35.309 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.309 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.309 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:35.309 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.591428+0000", 2024-04-19T01:04:35.309 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5142e-05 2024-04-19T01:04:35.309 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.309 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.309 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:35.309 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.592121+0000", 2024-04-19T01:04:35.309 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00069288499999999996 2024-04-19T01:04:35.309 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.309 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.309 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:35.310 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.592149+0000", 2024-04-19T01:04:35.310 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8382e-05 2024-04-19T01:04:35.310 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.310 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.310 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:35.310 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.605197+0000", 2024-04-19T01:04:35.310 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013047621000000001 2024-04-19T01:04:35.310 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.310 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.310 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:35.311 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.605233+0000", 2024-04-19T01:04:35.311 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6421999999999998e-05 2024-04-19T01:04:35.311 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.311 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.311 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.311 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.311 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.311 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.fs6 47/31 ECSubWrite(tid=61, reqid=client.4604.0:317, at_version=47'16, trim_to=0'0, roll_forward_to=45'14))", 2024-04-19T01:04:35.311 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.605368+0000", 2024-04-19T01:04:35.311 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 17.639915464000001, 2024-04-19T01:04:35.312 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018611628000000002, 2024-04-19T01:04:35.312 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:35.312 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:35.312 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:35.312 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:35.312 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.312 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:35.312 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.605368+0000", 2024-04-19T01:04:35.312 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.312 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.312 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.312 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:35.312 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.605368+0000", 2024-04-19T01:04:35.312 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.313 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.313 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.313 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:35.313 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.605368+0000", 2024-04-19T01:04:35.313 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3700000000000001e-07 2024-04-19T01:04:35.313 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.313 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.313 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:35.313 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.605723+0000", 2024-04-19T01:04:35.313 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035445499999999999 2024-04-19T01:04:35.313 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.313 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.313 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:35.313 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.605744+0000", 2024-04-19T01:04:35.314 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1229e-05 2024-04-19T01:04:35.314 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.314 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.314 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:35.314 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.605759+0000", 2024-04-19T01:04:35.315 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5e-05 2024-04-19T01:04:35.315 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.315 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.315 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:35.315 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.605888+0000", 2024-04-19T01:04:35.316 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012950200000000001 2024-04-19T01:04:35.316 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.316 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.316 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:35.316 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.605906+0000", 2024-04-19T01:04:35.316 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7462e-05 2024-04-19T01:04:35.316 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.316 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.316 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:35.316 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.623858+0000", 2024-04-19T01:04:35.316 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017952630000000001 2024-04-19T01:04:35.316 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.316 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.317 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:35.317 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.623979+0000", 2024-04-19T01:04:35.317 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000120913 2024-04-19T01:04:35.317 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.317 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.317 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.317 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.317 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.317 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.fs6 47/31 ECSubWrite(tid=62, reqid=client.4604.0:318, at_version=47'17, trim_to=0'0, roll_forward_to=45'14))", 2024-04-19T01:04:35.317 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.612635+0000", 2024-04-19T01:04:35.317 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 17.632647674000001, 2024-04-19T01:04:35.318 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014639107, 2024-04-19T01:04:35.318 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:35.318 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:35.318 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:35.318 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:35.318 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.318 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:35.318 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.612635+0000", 2024-04-19T01:04:35.318 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.318 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.318 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.318 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:35.319 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.612635+0000", 2024-04-19T01:04:35.319 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.319 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.319 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.319 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:35.319 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.612637+0000", 2024-04-19T01:04:35.319 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6199999999999999e-06 2024-04-19T01:04:35.319 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.319 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.319 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:35.319 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.613041+0000", 2024-04-19T01:04:35.319 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00040351699999999999 2024-04-19T01:04:35.319 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.319 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.320 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:35.320 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.613066+0000", 2024-04-19T01:04:35.320 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5675999999999999e-05 2024-04-19T01:04:35.320 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.320 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.320 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:35.320 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.613081+0000", 2024-04-19T01:04:35.320 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4929999999999999e-05 2024-04-19T01:04:35.320 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.320 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.320 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:35.320 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.613751+0000", 2024-04-19T01:04:35.320 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00066930199999999998 2024-04-19T01:04:35.320 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.321 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.321 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:35.321 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.613764+0000", 2024-04-19T01:04:35.321 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3454e-05 2024-04-19T01:04:35.321 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.321 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.322 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:35.322 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.627156+0000", 2024-04-19T01:04:35.322 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013391568 2024-04-19T01:04:35.322 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.322 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.322 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:35.322 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.627275+0000", 2024-04-19T01:04:35.322 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011904 2024-04-19T01:04:35.322 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.322 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.322 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.322 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.323 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.323 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds5 47/31 ECSubWrite(tid=61, reqid=client.4604.0:330, at_version=47'32, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:04:35.323 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.677579+0000", 2024-04-19T01:04:35.323 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 17.567704509999999, 2024-04-19T01:04:35.323 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.033702862, 2024-04-19T01:04:35.323 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:35.323 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:35.323 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:35.323 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:35.323 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.323 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:35.323 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.677579+0000", 2024-04-19T01:04:35.323 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.323 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.324 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.324 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:35.324 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.677579+0000", 2024-04-19T01:04:35.324 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.324 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.324 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.324 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:35.324 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.677581+0000", 2024-04-19T01:04:35.324 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2800000000000002e-06 2024-04-19T01:04:35.324 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.324 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.324 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:35.324 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678062+0000", 2024-04-19T01:04:35.325 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00048067900000000002 2024-04-19T01:04:35.325 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.325 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.325 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:35.325 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678096+0000", 2024-04-19T01:04:35.325 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4381999999999997e-05 2024-04-19T01:04:35.325 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.325 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.325 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:35.326 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678117+0000", 2024-04-19T01:04:35.326 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0679999999999999e-05 2024-04-19T01:04:35.326 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.326 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.326 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:35.326 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678249+0000", 2024-04-19T01:04:35.327 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013211899999999999 2024-04-19T01:04:35.327 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.327 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.327 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:35.327 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678270+0000", 2024-04-19T01:04:35.327 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1409000000000001e-05 2024-04-19T01:04:35.327 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.327 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.327 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:35.327 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711228+0000", 2024-04-19T01:04:35.327 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.032957575000000003 2024-04-19T01:04:35.327 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.327 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.327 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:35.328 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711281+0000", 2024-04-19T01:04:35.328 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3737999999999999e-05 2024-04-19T01:04:35.328 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.328 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.328 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.328 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.328 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.328 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds5 47/31 ECSubWrite(tid=62, reqid=client.4604.0:331, at_version=47'33, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:04:35.328 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.682545+0000", 2024-04-19T01:04:35.328 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 17.562738468999999, 2024-04-19T01:04:35.328 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.028771769999999999, 2024-04-19T01:04:35.328 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:35.328 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:35.328 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:35.328 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:35.329 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.329 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:35.329 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682545+0000", 2024-04-19T01:04:35.329 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.329 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.329 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.329 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:35.329 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682545+0000", 2024-04-19T01:04:35.329 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.329 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.329 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.329 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:35.329 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682546+0000", 2024-04-19T01:04:35.329 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.567e-06 2024-04-19T01:04:35.330 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.330 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.330 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:35.330 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682965+0000", 2024-04-19T01:04:35.330 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00041881399999999999 2024-04-19T01:04:35.330 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.330 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.330 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:35.330 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682999+0000", 2024-04-19T01:04:35.330 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3738999999999997e-05 2024-04-19T01:04:35.330 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.330 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.330 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:35.331 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.683021+0000", 2024-04-19T01:04:35.331 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2592e-05 2024-04-19T01:04:35.331 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.331 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.331 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:35.331 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.691707+0000", 2024-04-19T01:04:35.331 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0086853629999999998 2024-04-19T01:04:35.331 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.331 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.331 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:35.331 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.691732+0000", 2024-04-19T01:04:35.331 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5366000000000001e-05 2024-04-19T01:04:35.331 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.331 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.332 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:35.332 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711288+0000", 2024-04-19T01:04:35.332 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019555922999999999 2024-04-19T01:04:35.332 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.332 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.332 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:35.332 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711316+0000", 2024-04-19T01:04:35.332 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8405999999999999e-05 2024-04-19T01:04:35.332 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.332 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.332 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.332 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.332 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.332 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds5 47/31 ECSubWrite(tid=63, reqid=client.4604.0:332, at_version=47'34, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:04:35.333 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.688708+0000", 2024-04-19T01:04:35.333 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 17.556575572, 2024-04-19T01:04:35.333 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.02327926, 2024-04-19T01:04:35.333 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:35.333 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:35.333 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:35.333 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:35.333 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.333 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:35.333 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.688708+0000", 2024-04-19T01:04:35.333 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.333 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.333 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.333 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:35.334 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.688708+0000", 2024-04-19T01:04:35.334 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.334 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.334 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.334 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:35.334 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.688709+0000", 2024-04-19T01:04:35.334 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5e-06 2024-04-19T01:04:35.334 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.334 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.334 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:35.334 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.688989+0000", 2024-04-19T01:04:35.334 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00028006800000000002 2024-04-19T01:04:35.334 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.334 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.335 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:35.335 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.689021+0000", 2024-04-19T01:04:35.335 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2026000000000001e-05 2024-04-19T01:04:35.335 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.335 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.335 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:35.335 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.689047+0000", 2024-04-19T01:04:35.335 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5809999999999999e-05 2024-04-19T01:04:35.335 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.335 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.335 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:35.335 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.704765+0000", 2024-04-19T01:04:35.335 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015717913 2024-04-19T01:04:35.335 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.336 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.336 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:35.336 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.704793+0000", 2024-04-19T01:04:35.336 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8143000000000001e-05 2024-04-19T01:04:35.336 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.338 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.338 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:35.338 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711907+0000", 2024-04-19T01:04:35.338 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071141879999999996 2024-04-19T01:04:35.338 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.338 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.338 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:35.338 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711987+0000", 2024-04-19T01:04:35.339 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9611999999999996e-05 2024-04-19T01:04:35.339 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.339 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.339 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.339 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.339 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.339 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds5 47/31 ECSubWrite(tid=64, reqid=client.4604.0:333, at_version=47'35, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:04:35.339 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.692104+0000", 2024-04-19T01:04:35.339 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 17.553179312000001, 2024-04-19T01:04:35.339 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020504654000000001, 2024-04-19T01:04:35.339 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:35.339 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:35.339 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:35.339 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:35.340 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.340 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:35.340 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692104+0000", 2024-04-19T01:04:35.340 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.340 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.340 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.340 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:35.340 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692104+0000", 2024-04-19T01:04:35.340 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.340 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.340 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.340 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:35.340 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692105+0000", 2024-04-19T01:04:35.340 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.64e-06 2024-04-19T01:04:35.341 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.341 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.341 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:35.341 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692137+0000", 2024-04-19T01:04:35.341 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1631999999999998e-05 2024-04-19T01:04:35.341 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.341 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.341 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:35.341 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692165+0000", 2024-04-19T01:04:35.341 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7526e-05 2024-04-19T01:04:35.341 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.341 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.341 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:35.341 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692180+0000", 2024-04-19T01:04:35.341 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5146e-05 2024-04-19T01:04:35.342 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.342 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.342 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:35.342 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711392+0000", 2024-04-19T01:04:35.342 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019211967 2024-04-19T01:04:35.342 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.342 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.342 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:35.342 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711407+0000", 2024-04-19T01:04:35.342 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4866e-05 2024-04-19T01:04:35.342 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.342 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.342 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:35.342 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.712524+0000", 2024-04-19T01:04:35.342 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011173159999999999 2024-04-19T01:04:35.343 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.343 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.343 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:35.343 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.712608+0000", 2024-04-19T01:04:35.343 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4561000000000006e-05 2024-04-19T01:04:35.343 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.343 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.343 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.343 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.343 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.343 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.fs6 48/31 ECSubWrite(tid=76, reqid=client.4604.0:345, at_version=48'22, trim_to=0'0, roll_forward_to=47'19))", 2024-04-19T01:04:35.343 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:18.538520+0000", 2024-04-19T01:04:35.343 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 16.706762858000001, 2024-04-19T01:04:35.343 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019548551000000001, 2024-04-19T01:04:35.343 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:35.344 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:35.344 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:35.344 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:35.344 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.344 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:35.344 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.538520+0000", 2024-04-19T01:04:35.344 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.344 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.344 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.344 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:35.344 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.538520+0000", 2024-04-19T01:04:35.344 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.344 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.344 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.345 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:35.345 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.538521+0000", 2024-04-19T01:04:35.345 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.001e-06 2024-04-19T01:04:35.345 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.345 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.345 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:35.345 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.538786+0000", 2024-04-19T01:04:35.345 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026452700000000002 2024-04-19T01:04:35.345 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.345 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.345 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:35.345 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.538803+0000", 2024-04-19T01:04:35.345 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7428999999999999e-05 2024-04-19T01:04:35.345 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.345 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.346 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:35.346 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.538813+0000", 2024-04-19T01:04:35.346 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7040000000000006e-06 2024-04-19T01:04:35.346 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.346 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.346 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:35.346 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.538943+0000", 2024-04-19T01:04:35.346 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013024400000000001 2024-04-19T01:04:35.346 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.346 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.346 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:35.346 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.538970+0000", 2024-04-19T01:04:35.346 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7279000000000001e-05 2024-04-19T01:04:35.347 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.347 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.347 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:35.347 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.557980+0000", 2024-04-19T01:04:35.347 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019009169999999999 2024-04-19T01:04:35.347 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.347 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.347 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:35.347 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.558069+0000", 2024-04-19T01:04:35.347 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9197000000000007e-05 2024-04-19T01:04:35.347 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.347 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.347 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.347 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.347 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.348 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 59/31 ECSubWrite(tid=66, reqid=client.4604.0:423, at_version=59'51, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:04:35.348 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.541317+0000", 2024-04-19T01:04:35.348 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.7039659660000002, 2024-04-19T01:04:35.348 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.047133948000000002, 2024-04-19T01:04:35.348 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:35.348 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:35.348 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:35.349 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:35.349 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.349 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:35.349 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.541317+0000", 2024-04-19T01:04:35.349 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.349 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.349 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.349 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:35.349 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.541317+0000", 2024-04-19T01:04:35.349 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.349 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.349 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.349 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:35.350 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.541318+0000", 2024-04-19T01:04:35.350 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3399999999999999e-07 2024-04-19T01:04:35.350 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.350 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.350 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:35.350 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.543831+0000", 2024-04-19T01:04:35.350 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025130999999999999 2024-04-19T01:04:35.350 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.350 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.350 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:35.350 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.543879+0000", 2024-04-19T01:04:35.350 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8208999999999999e-05 2024-04-19T01:04:35.350 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.350 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.351 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:35.351 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.543903+0000", 2024-04-19T01:04:35.351 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.404e-05 2024-04-19T01:04:35.351 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.351 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.351 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:35.351 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.544070+0000", 2024-04-19T01:04:35.351 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000166372 2024-04-19T01:04:35.351 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.351 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.351 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:35.351 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.544097+0000", 2024-04-19T01:04:35.351 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7256000000000001e-05 2024-04-19T01:04:35.351 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.352 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.352 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:35.352 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.588382+0000", 2024-04-19T01:04:35.352 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.044284780000000003 2024-04-19T01:04:35.352 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.352 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.352 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:35.352 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.588451+0000", 2024-04-19T01:04:35.352 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9556999999999998e-05 2024-04-19T01:04:35.352 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.352 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.352 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.352 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.352 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.353 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 59/31 ECSubWrite(tid=67, reqid=client.4604.0:424, at_version=59'52, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:04:35.353 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.548963+0000", 2024-04-19T01:04:35.353 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.6963201999999997, 2024-04-19T01:04:35.353 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.039534443000000002, 2024-04-19T01:04:35.353 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:35.353 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:35.353 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:35.353 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:35.353 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.353 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:35.353 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.548963+0000", 2024-04-19T01:04:35.353 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.353 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.353 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.353 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:35.354 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.548963+0000", 2024-04-19T01:04:35.354 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.354 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.354 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.354 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:35.354 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.548965+0000", 2024-04-19T01:04:35.354 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1500000000000002e-06 2024-04-19T01:04:35.354 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.354 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.354 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:35.354 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549551+0000", 2024-04-19T01:04:35.354 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00058640299999999999 2024-04-19T01:04:35.354 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.354 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.355 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:35.355 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549596+0000", 2024-04-19T01:04:35.355 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4604999999999997e-05 2024-04-19T01:04:35.355 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.355 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.355 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:35.355 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549631+0000", 2024-04-19T01:04:35.355 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4725999999999999e-05 2024-04-19T01:04:35.355 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.355 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.355 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:35.355 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.567909+0000", 2024-04-19T01:04:35.355 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018277755999999999 2024-04-19T01:04:35.355 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.355 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.356 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:35.356 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.567944+0000", 2024-04-19T01:04:35.356 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5862999999999999e-05 2024-04-19T01:04:35.356 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.356 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.356 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:35.356 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.588471+0000", 2024-04-19T01:04:35.356 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020526902999999999 2024-04-19T01:04:35.356 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.356 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.356 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:35.356 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.588497+0000", 2024-04-19T01:04:35.356 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6037000000000002e-05 2024-04-19T01:04:35.356 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.356 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.357 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.357 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.357 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.357 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 59/31 ECSubWrite(tid=68, reqid=client.4604.0:425, at_version=59'53, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:04:35.357 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.556523+0000", 2024-04-19T01:04:35.357 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.6887597889999997, 2024-04-19T01:04:35.357 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.032109275999999999, 2024-04-19T01:04:35.357 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:35.357 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:35.357 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:35.357 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:35.357 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.357 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:35.357 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556523+0000", 2024-04-19T01:04:35.358 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.358 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.358 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.358 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:35.358 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556523+0000", 2024-04-19T01:04:35.358 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.358 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.358 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.358 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:35.358 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556526+0000", 2024-04-19T01:04:35.358 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4930000000000001e-06 2024-04-19T01:04:35.358 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.359 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.359 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:35.359 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556650+0000", 2024-04-19T01:04:35.359 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000124487 2024-04-19T01:04:35.359 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.359 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.359 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:35.359 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556701+0000", 2024-04-19T01:04:35.360 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1158e-05 2024-04-19T01:04:35.360 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.360 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.360 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:35.360 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556727+0000", 2024-04-19T01:04:35.360 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5843999999999999e-05 2024-04-19T01:04:35.360 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.360 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.360 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:35.360 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.585945+0000", 2024-04-19T01:04:35.360 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0292181 2024-04-19T01:04:35.360 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.360 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.360 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:35.361 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.585972+0000", 2024-04-19T01:04:35.361 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6356e-05 2024-04-19T01:04:35.361 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.361 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.361 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:35.361 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.588601+0000", 2024-04-19T01:04:35.361 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0026295839999999999 2024-04-19T01:04:35.361 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.361 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.361 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:35.361 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.588633+0000", 2024-04-19T01:04:35.361 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1254000000000003e-05 2024-04-19T01:04:35.361 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.361 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.362 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.362 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.362 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.362 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 59/31 ECSubWrite(tid=69, reqid=client.4604.0:426, at_version=59'54, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:04:35.362 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.558033+0000", 2024-04-19T01:04:35.362 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.6872498389999997, 2024-04-19T01:04:35.362 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.030934886000000002, 2024-04-19T01:04:35.362 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:35.362 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:35.362 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:35.362 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:35.362 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.362 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:35.363 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558033+0000", 2024-04-19T01:04:35.363 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.363 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.363 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.363 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:35.363 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558033+0000", 2024-04-19T01:04:35.363 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.363 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.363 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.363 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:35.363 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558035+0000", 2024-04-19T01:04:35.363 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7370000000000001e-06 2024-04-19T01:04:35.363 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.363 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.364 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:35.364 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558080+0000", 2024-04-19T01:04:35.364 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5089000000000001e-05 2024-04-19T01:04:35.364 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.364 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.364 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:35.364 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558126+0000", 2024-04-19T01:04:35.364 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5902000000000002e-05 2024-04-19T01:04:35.364 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.364 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.364 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:35.364 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558149+0000", 2024-04-19T01:04:35.364 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3136e-05 2024-04-19T01:04:35.364 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.365 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.365 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:35.365 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.588071+0000", 2024-04-19T01:04:35.365 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.029921515999999999 2024-04-19T01:04:35.365 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.365 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.365 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:35.365 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.588089+0000", 2024-04-19T01:04:35.365 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8788e-05 2024-04-19T01:04:35.365 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.365 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.365 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:35.365 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.588944+0000", 2024-04-19T01:04:35.365 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00085441300000000005 2024-04-19T01:04:35.365 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.366 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.366 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:35.366 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.588968+0000", 2024-04-19T01:04:35.366 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4304999999999999e-05 2024-04-19T01:04:35.366 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.366 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.366 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.366 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.366 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.366 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:04:35.367 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:04:35.244+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:04:35.368 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-19T01:04:35.370 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:04:35.370+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:04:35.469 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:04:35.468+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:04:35.538 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:04:35.538 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-04-19T01:04:35.538 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-04-19T01:04:35.539 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:04:35.552 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-19T01:04:35.557 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2024-04-19T01:04:35.565 INFO:tasks.rados.rados.0.smithi155.stdout:190: done (1 left) 2024-04-19T01:04:35.566 INFO:tasks.rados.rados.0.smithi155.stdout:191: done (0 left) 2024-04-19T01:04:35.566 INFO:tasks.rados.rados.0.smithi155.stdout:192: read oid 18 snap -1 2024-04-19T01:04:35.566 INFO:tasks.rados.rados.0.smithi155.stdout:192: expect (ObjNum 17 snap 0 seq_num 17) 2024-04-19T01:04:35.566 INFO:tasks.rados.rados.0.smithi155.stdout:193: delete oid 33 current snap is 17 2024-04-19T01:04:35.571 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:04:35.569+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:04:35.571 INFO:tasks.rados.rados.0.smithi155.stdout:192: expect (ObjNum 17 snap 0 seq_num 17) 2024-04-19T01:04:35.596 INFO:tasks.rados.rados.0.smithi155.stdout:192: done (1 left) 2024-04-19T01:04:35.596 INFO:tasks.rados.rados.0.smithi155.stdout:193: done (0 left) 2024-04-19T01:04:35.596 INFO:tasks.rados.rados.0.smithi155.stdout:194: delete oid 27 current snap is 17 2024-04-19T01:04:35.607 INFO:tasks.rados.rados.0.smithi155.stdout:194: done (0 left) 2024-04-19T01:04:35.607 INFO:tasks.rados.rados.0.smithi155.stdout:195: read oid 35 snap -1 2024-04-19T01:04:35.607 INFO:tasks.rados.rados.0.smithi155.stdout:195: expect (ObjNum 88 snap 15 seq_num 88) 2024-04-19T01:04:35.607 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 13 current snap is 17 2024-04-19T01:04:35.607 INFO:tasks.rados.rados.0.smithi155.stdout:196: seq_num 91 ranges {212992=770048,983040=475136,1458176=294912} 2024-04-19T01:04:35.623 INFO:tasks.rados.rados.0.smithi155.stdout:196: writing smithi15528762-13 from 212992 to 983040 tid 1 2024-04-19T01:04:35.631 INFO:tasks.rados.rados.0.smithi155.stdout:196: writing smithi15528762-13 from 983040 to 1458176 tid 2 2024-04-19T01:04:35.636 INFO:tasks.rados.rados.0.smithi155.stdout:196: writing smithi15528762-13 from 1458176 to 1753088 tid 3 2024-04-19T01:04:35.636 INFO:tasks.rados.rados.0.smithi155.stdout:197: read oid 28 snap -1 2024-04-19T01:04:35.637 INFO:tasks.rados.rados.0.smithi155.stdout:197: expect deleted 2024-04-19T01:04:35.637 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 18 current snap is 17 2024-04-19T01:04:35.637 INFO:tasks.rados.rados.0.smithi155.stdout:198: seq_num 92 ranges {294912=786432,1081344=491520,1572864=278528} 2024-04-19T01:04:35.654 INFO:tasks.rados.rados.0.smithi155.stdout:198: writing smithi15528762-18 from 294912 to 1081344 tid 1 2024-04-19T01:04:35.663 INFO:tasks.rados.rados.0.smithi155.stdout:198: writing smithi15528762-18 from 1081344 to 1572864 tid 2 2024-04-19T01:04:35.667 INFO:tasks.rados.rados.0.smithi155.stdout:198: writing smithi15528762-18 from 1572864 to 1851392 tid 3 2024-04-19T01:04:35.667 INFO:tasks.rados.rados.0.smithi155.stdout:199: snap_create 2024-04-19T01:04:35.667 INFO:tasks.rados.rados.0.smithi155.stdout:195: expect (ObjNum 88 snap 15 seq_num 88) 2024-04-19T01:04:35.671 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:04:35.669+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:04:35.728 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:04:35.728 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-04-19T01:04:35.728 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-04-19T01:04:35.728 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-04-19T01:04:35.729 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:04:35.742 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-19T01:04:35.772 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:04:35.770+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:04:35.834 INFO:tasks.rados.rados.0.smithi155.stdout:196: finishing write tid 1 to smithi15528762-13 2024-04-19T01:04:35.834 INFO:tasks.rados.rados.0.smithi155.stdout:196: finishing write tid 2 to smithi15528762-13 2024-04-19T01:04:35.834 INFO:tasks.rados.rados.0.smithi155.stdout:196: finishing write tid 3 to smithi15528762-13 2024-04-19T01:04:35.834 INFO:tasks.rados.rados.0.smithi155.stdout:196: finishing write tid 4 to smithi15528762-13 2024-04-19T01:04:35.834 INFO:tasks.rados.rados.0.smithi155.stdout:196: finishing write tid 5 to smithi15528762-13 2024-04-19T01:04:35.834 INFO:tasks.rados.rados.0.smithi155.stdout:196: oid 13 updating version 0 to 40 2024-04-19T01:04:35.834 INFO:tasks.rados.rados.0.smithi155.stdout:196: oid 13 updating version 40 to 41 2024-04-19T01:04:35.834 INFO:tasks.rados.rados.0.smithi155.stdout:196: oid 13 updating version 41 to 42 2024-04-19T01:04:35.834 INFO:tasks.rados.rados.0.smithi155.stdout:196: oid 13 version 42 is already newer than 39 2024-04-19T01:04:35.834 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 13 v 42 (ObjNum 91 snap 17 seq_num 91) dirty exists 2024-04-19T01:04:35.834 INFO:tasks.rados.rados.0.smithi155.stdout:196: left oid 13 (ObjNum 91 snap 17 seq_num 91) 2024-04-19T01:04:35.835 INFO:tasks.rados.rados.0.smithi155.stdout:198: finishing write tid 1 to smithi15528762-18 2024-04-19T01:04:35.835 INFO:tasks.rados.rados.0.smithi155.stdout:198: finishing write tid 2 to smithi15528762-18 2024-04-19T01:04:35.835 INFO:tasks.rados.rados.0.smithi155.stdout:198: finishing write tid 3 to smithi15528762-18 2024-04-19T01:04:35.835 INFO:tasks.rados.rados.0.smithi155.stdout:198: finishing write tid 4 to smithi15528762-18 2024-04-19T01:04:35.835 INFO:tasks.rados.rados.0.smithi155.stdout:198: finishing write tid 5 to smithi15528762-18 2024-04-19T01:04:35.835 INFO:tasks.rados.rados.0.smithi155.stdout:198: oid 18 updating version 0 to 56 2024-04-19T01:04:35.835 INFO:tasks.rados.rados.0.smithi155.stdout:198: oid 18 updating version 56 to 58 2024-04-19T01:04:35.835 INFO:tasks.rados.rados.0.smithi155.stdout:198: oid 18 updating version 58 to 59 2024-04-19T01:04:35.835 INFO:tasks.rados.rados.0.smithi155.stdout:198: oid 18 version 59 is already newer than 57 2024-04-19T01:04:35.835 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 18 v 59 (ObjNum 92 snap 17 seq_num 92) dirty exists 2024-04-19T01:04:35.835 INFO:tasks.rados.rados.0.smithi155.stdout:198: left oid 18 (ObjNum 92 snap 17 seq_num 92) 2024-04-19T01:04:35.871 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:04:35.871 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-04-19T01:04:35.872 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-04-19T01:04:35.872 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-04-19T01:04:35.872 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.872 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:240 3.1s0 3:82ff4f48:::smithi15528762-41:head [read 0~688128,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:35.872 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.861180+0000", 2024-04-19T01:04:35.872 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 22.009566230000001, 2024-04-19T01:04:35.873 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.026860730999999999, 2024-04-19T01:04:35.873 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:35.873 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:35.873 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-04-19T01:04:35.873 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:04:35.873 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:04:35.873 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:35.873 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 240 2024-04-19T01:04:35.874 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.874 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:35.874 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.874 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:35.874 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861180+0000", 2024-04-19T01:04:35.874 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.875 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.875 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.875 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:35.875 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861180+0000", 2024-04-19T01:04:35.875 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.875 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.875 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.875 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:35.875 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861182+0000", 2024-04-19T01:04:35.876 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.654e-06 2024-04-19T01:04:35.876 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.876 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.876 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:35.876 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861212+0000", 2024-04-19T01:04:35.876 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0511999999999999e-05 2024-04-19T01:04:35.876 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.876 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.876 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:35.876 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861234+0000", 2024-04-19T01:04:35.876 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2135999999999999e-05 2024-04-19T01:04:35.876 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.877 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.877 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:35.877 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861250+0000", 2024-04-19T01:04:35.877 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6141999999999999e-05 2024-04-19T01:04:35.877 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.877 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.877 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:35.877 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.865146+0000", 2024-04-19T01:04:35.877 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0038953350000000002 2024-04-19T01:04:35.878 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.878 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.878 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:04:35.878 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.865375+0000", 2024-04-19T01:04:35.878 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00022880900000000001 2024-04-19T01:04:35.878 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.878 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.879 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:35.879 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.888041+0000", 2024-04-19T01:04:35.879 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022666143 2024-04-19T01:04:35.879 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.879 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.879 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.879 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.879 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.879 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:241 3.1s0 3:82ff4f48:::smithi15528762-41:head [read 0~688128,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:35.880 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.861274+0000", 2024-04-19T01:04:35.880 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 22.009472382999999, 2024-04-19T01:04:35.880 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.027990521000000001, 2024-04-19T01:04:35.880 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:35.880 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:35.880 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-04-19T01:04:35.880 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:04:35.880 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:04:35.881 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:35.881 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 241 2024-04-19T01:04:35.881 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.881 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:35.881 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.881 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:35.881 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861274+0000", 2024-04-19T01:04:35.882 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.882 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.882 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.882 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:35.882 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861274+0000", 2024-04-19T01:04:35.882 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.883 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.883 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.883 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:35.883 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861275+0000", 2024-04-19T01:04:35.883 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3030000000000001e-06 2024-04-19T01:04:35.883 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.884 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.884 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:35.884 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861282+0000", 2024-04-19T01:04:35.884 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8569999999999997e-06 2024-04-19T01:04:35.884 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.884 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.884 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:35.885 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861300+0000", 2024-04-19T01:04:35.885 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7898999999999999e-05 2024-04-19T01:04:35.885 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.885 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.885 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:35.885 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861314+0000", 2024-04-19T01:04:35.885 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.364e-05 2024-04-19T01:04:35.885 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.886 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.886 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:35.886 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.868359+0000", 2024-04-19T01:04:35.886 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0070459019999999997 2024-04-19T01:04:35.886 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.886 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.886 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:04:35.887 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.868594+0000", 2024-04-19T01:04:35.887 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000234942 2024-04-19T01:04:35.887 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.887 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.887 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:35.887 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.889264+0000", 2024-04-19T01:04:35.887 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020669977999999999 2024-04-19T01:04:35.888 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.888 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.888 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.888 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.888 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.888 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s6 43/31 ECSubWrite(tid=67, reqid=osd.13.0:67, at_version=43'39, trim_to=0'0, roll_forward_to=41'37))", 2024-04-19T01:04:35.889 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.867247+0000", 2024-04-19T01:04:35.889 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 22.003499627, 2024-04-19T01:04:35.889 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019643094999999999, 2024-04-19T01:04:35.889 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:35.889 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:35.889 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:35.889 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:35.890 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.890 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:35.890 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.867247+0000", 2024-04-19T01:04:35.890 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.890 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.890 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.890 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:35.890 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.867247+0000", 2024-04-19T01:04:35.891 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.891 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.891 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.891 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:35.891 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.867247+0000", 2024-04-19T01:04:35.891 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7299999999999996e-07 2024-04-19T01:04:35.891 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.891 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.891 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:35.891 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.867324+0000", 2024-04-19T01:04:35.891 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6981e-05 2024-04-19T01:04:35.891 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.892 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.892 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:35.892 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.867357+0000", 2024-04-19T01:04:35.892 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3269000000000003e-05 2024-04-19T01:04:35.892 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.892 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.892 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:35.892 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.867380+0000", 2024-04-19T01:04:35.892 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2272999999999998e-05 2024-04-19T01:04:35.892 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.892 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.893 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:35.893 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.870949+0000", 2024-04-19T01:04:35.893 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0035693259999999998 2024-04-19T01:04:35.893 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.893 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.893 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:35.893 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.870973+0000", 2024-04-19T01:04:35.894 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3859e-05 2024-04-19T01:04:35.894 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.894 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.894 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:35.894 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.886787+0000", 2024-04-19T01:04:35.895 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0158142 2024-04-19T01:04:35.895 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.895 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.895 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:35.895 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.886890+0000", 2024-04-19T01:04:35.895 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000102614 2024-04-19T01:04:35.895 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.896 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.896 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.896 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.896 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.896 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s6 43/31 ECSubWrite(tid=69, reqid=osd.13.0:69, at_version=43'41, trim_to=0'0, roll_forward_to=41'37))", 2024-04-19T01:04:35.896 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.867419+0000", 2024-04-19T01:04:35.896 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 22.003327656, 2024-04-19T01:04:35.896 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019827727999999999, 2024-04-19T01:04:35.897 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:35.897 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:35.897 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:35.897 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:35.897 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.897 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:35.897 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.867419+0000", 2024-04-19T01:04:35.897 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.898 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.898 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.898 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:35.898 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.867419+0000", 2024-04-19T01:04:35.898 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.898 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.898 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.898 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:35.898 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.867420+0000", 2024-04-19T01:04:35.898 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.04e-06 2024-04-19T01:04:35.898 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.898 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.899 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:35.899 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.867470+0000", 2024-04-19T01:04:35.899 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9914999999999999e-05 2024-04-19T01:04:35.899 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.899 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.899 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:35.899 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.867498+0000", 2024-04-19T01:04:35.899 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8969e-05 2024-04-19T01:04:35.899 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.900 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.900 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:35.900 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.867517+0000", 2024-04-19T01:04:35.900 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8675999999999999e-05 2024-04-19T01:04:35.900 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.900 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.900 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:35.900 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.871964+0000", 2024-04-19T01:04:35.900 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0044469260000000004 2024-04-19T01:04:35.900 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.900 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.901 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:35.901 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.871987+0000", 2024-04-19T01:04:35.901 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2521999999999999e-05 2024-04-19T01:04:35.901 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.901 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.901 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:35.901 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.887168+0000", 2024-04-19T01:04:35.901 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015181019 2024-04-19T01:04:35.901 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.901 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.901 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:35.902 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.887246+0000", 2024-04-19T01:04:35.902 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8660999999999998e-05 2024-04-19T01:04:35.902 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.902 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.902 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.902 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.902 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.902 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s1 43/31 ECSubWrite(tid=43, reqid=client.4604.0:251, at_version=43'29, trim_to=0'0, roll_forward_to=43'27))", 2024-04-19T01:04:35.902 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.985924+0000", 2024-04-19T01:04:35.902 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.884821930000001, 2024-04-19T01:04:35.902 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.029591623000000001, 2024-04-19T01:04:35.902 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:35.902 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:35.903 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:35.903 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:35.903 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.903 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:35.903 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.985924+0000", 2024-04-19T01:04:35.903 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.903 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.903 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.903 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:35.903 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.985924+0000", 2024-04-19T01:04:35.903 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.903 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.904 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.904 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:35.904 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.985927+0000", 2024-04-19T01:04:35.904 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3460000000000001e-06 2024-04-19T01:04:35.904 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.904 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.904 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:35.904 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986452+0000", 2024-04-19T01:04:35.904 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00052555199999999999 2024-04-19T01:04:35.904 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.904 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.904 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:35.904 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986541+0000", 2024-04-19T01:04:35.905 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8352999999999999e-05 2024-04-19T01:04:35.905 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.905 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.905 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:35.905 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986597+0000", 2024-04-19T01:04:35.905 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6144999999999999e-05 2024-04-19T01:04:35.905 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.905 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.905 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:35.905 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986776+0000", 2024-04-19T01:04:35.905 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017914999999999999 2024-04-19T01:04:35.905 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.905 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.905 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:35.905 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986811+0000", 2024-04-19T01:04:35.906 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5233e-05 2024-04-19T01:04:35.906 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.906 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.906 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:35.906 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.015444+0000", 2024-04-19T01:04:35.906 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.028632676999999999 2024-04-19T01:04:35.906 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.907 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.907 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:35.907 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.015516+0000", 2024-04-19T01:04:35.907 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2167000000000002e-05 2024-04-19T01:04:35.907 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.907 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.907 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.908 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.908 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.908 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds6 43/31 ECSubWrite(tid=38, reqid=client.4604.0:253, at_version=43'14, trim_to=0'0, roll_forward_to=43'12))", 2024-04-19T01:04:35.908 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.987449+0000", 2024-04-19T01:04:35.908 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.883297049999999, 2024-04-19T01:04:35.908 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.027803888999999998, 2024-04-19T01:04:35.908 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:35.908 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:35.908 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:35.908 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:35.908 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.908 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:35.908 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987449+0000", 2024-04-19T01:04:35.908 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.909 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.909 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.909 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:35.909 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987449+0000", 2024-04-19T01:04:35.909 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.909 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.909 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.909 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:35.909 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987451+0000", 2024-04-19T01:04:35.909 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2929999999999999e-06 2024-04-19T01:04:35.909 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.909 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.909 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:35.909 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988097+0000", 2024-04-19T01:04:35.909 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00064595099999999997 2024-04-19T01:04:35.910 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.910 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.910 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:35.910 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988140+0000", 2024-04-19T01:04:35.910 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2809000000000003e-05 2024-04-19T01:04:35.910 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.910 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.910 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:35.910 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988168+0000", 2024-04-19T01:04:35.910 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7498999999999999e-05 2024-04-19T01:04:35.910 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.910 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.910 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:35.910 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988319+0000", 2024-04-19T01:04:35.911 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000151621 2024-04-19T01:04:35.911 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.911 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.911 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:35.911 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988345+0000", 2024-04-19T01:04:35.911 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5846e-05 2024-04-19T01:04:35.911 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.911 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.911 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:35.911 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.015183+0000", 2024-04-19T01:04:35.911 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.026837928 2024-04-19T01:04:35.911 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.911 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.912 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:35.912 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.015253+0000", 2024-04-19T01:04:35.912 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9942000000000002e-05 2024-04-19T01:04:35.912 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.912 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.912 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.912 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.912 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.913 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:289 3.1s0 3:8ea800b0:::smithi15528762-8:head [stat,delete] snapc 7=[7,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-19T01:04:35.913 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.872650+0000", 2024-04-19T01:04:35.913 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.998096162, 2024-04-19T01:04:35.914 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.043399042999999998, 2024-04-19T01:04:35.914 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:35.914 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:35.914 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:35.915 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:04:35.915 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:04:35.915 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:35.915 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 289 2024-04-19T01:04:35.915 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.915 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:35.915 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.915 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:35.915 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.872650+0000", 2024-04-19T01:04:35.915 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.915 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.915 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.915 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:35.915 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.872650+0000", 2024-04-19T01:04:35.916 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.916 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.916 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.916 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:35.916 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.872653+0000", 2024-04-19T01:04:35.916 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.23e-06 2024-04-19T01:04:35.916 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.916 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.916 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:35.916 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.872669+0000", 2024-04-19T01:04:35.916 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5293000000000001e-05 2024-04-19T01:04:35.916 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.916 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.916 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:35.917 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.872701+0000", 2024-04-19T01:04:35.917 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2391999999999998e-05 2024-04-19T01:04:35.917 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.917 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.917 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:35.917 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.872721+0000", 2024-04-19T01:04:35.917 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0400000000000001e-05 2024-04-19T01:04:35.917 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.917 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.917 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:35.917 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.872889+0000", 2024-04-19T01:04:35.917 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016723000000000001 2024-04-19T01:04:35.917 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.917 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.917 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:04:35.918 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.873172+0000", 2024-04-19T01:04:35.918 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000283338 2024-04-19T01:04:35.918 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.918 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.918 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:35.918 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.875406+0000", 2024-04-19T01:04:35.918 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022335839999999998 2024-04-19T01:04:35.918 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.918 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.918 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:35.918 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.914580+0000", 2024-04-19T01:04:35.918 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.039174253999999999 2024-04-19T01:04:35.918 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.919 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.919 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-04-19T01:04:35.919 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.914900+0000", 2024-04-19T01:04:35.919 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000320663 2024-04-19T01:04:35.919 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.919 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.919 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:35.919 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.916049+0000", 2024-04-19T01:04:35.920 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001148659 2024-04-19T01:04:35.920 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.920 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.920 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.920 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.920 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.920 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s1 44/31 ECSubWrite(tid=49, reqid=osd.8.0:49, at_version=44'33, trim_to=0'0, roll_forward_to=43'29))", 2024-04-19T01:04:35.920 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.876436+0000", 2024-04-19T01:04:35.920 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.994310478999999, 2024-04-19T01:04:35.920 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024659179, 2024-04-19T01:04:35.920 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:35.920 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:35.920 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:35.920 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:35.921 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.921 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:35.921 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876436+0000", 2024-04-19T01:04:35.921 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.921 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.921 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.921 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:35.921 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876436+0000", 2024-04-19T01:04:35.921 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.921 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.921 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.922 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:35.922 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876437+0000", 2024-04-19T01:04:35.922 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2860000000000001e-06 2024-04-19T01:04:35.922 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.922 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.922 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:35.922 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876485+0000", 2024-04-19T01:04:35.922 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8294999999999998e-05 2024-04-19T01:04:35.923 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.923 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.923 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:35.923 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876512+0000", 2024-04-19T01:04:35.923 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6363000000000002e-05 2024-04-19T01:04:35.923 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.923 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.923 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:35.923 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876531+0000", 2024-04-19T01:04:35.923 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8902999999999998e-05 2024-04-19T01:04:35.924 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.924 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.924 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:35.924 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.877917+0000", 2024-04-19T01:04:35.924 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0013865870000000001 2024-04-19T01:04:35.924 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.924 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.924 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:35.925 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.877944+0000", 2024-04-19T01:04:35.925 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6553000000000001e-05 2024-04-19T01:04:35.925 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.925 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.925 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:35.925 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.901009+0000", 2024-04-19T01:04:35.925 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023064772000000001 2024-04-19T01:04:35.925 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.925 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.925 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:35.925 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.901095+0000", 2024-04-19T01:04:35.925 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6420000000000003e-05 2024-04-19T01:04:35.925 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.925 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.926 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.926 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.926 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.926 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s6 44/31 ECSubWrite(tid=82, reqid=client.4604.0:280, at_version=44'50, trim_to=0'0, roll_forward_to=43'48))", 2024-04-19T01:04:35.926 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.882093+0000", 2024-04-19T01:04:35.926 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.988652867999999, 2024-04-19T01:04:35.926 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.042938993000000002, 2024-04-19T01:04:35.926 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:35.926 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:35.926 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:35.926 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:35.926 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.926 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:35.927 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882093+0000", 2024-04-19T01:04:35.927 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.927 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.927 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.927 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:35.927 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882093+0000", 2024-04-19T01:04:35.927 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.927 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.927 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.927 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:35.927 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882094+0000", 2024-04-19T01:04:35.927 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0130000000000001e-06 2024-04-19T01:04:35.927 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.927 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.927 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:35.928 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882590+0000", 2024-04-19T01:04:35.928 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00049558999999999996 2024-04-19T01:04:35.928 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.928 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.928 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:35.928 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882631+0000", 2024-04-19T01:04:35.928 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1399000000000001e-05 2024-04-19T01:04:35.928 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.928 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.928 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:35.928 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882654+0000", 2024-04-19T01:04:35.928 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2535999999999999e-05 2024-04-19T01:04:35.928 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.928 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.929 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:35.929 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882871+0000", 2024-04-19T01:04:35.929 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021730999999999999 2024-04-19T01:04:35.929 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.929 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.929 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:35.929 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882896+0000", 2024-04-19T01:04:35.929 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5111999999999999e-05 2024-04-19T01:04:35.929 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.929 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.929 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:35.929 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.924961+0000", 2024-04-19T01:04:35.929 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.042065022000000001 2024-04-19T01:04:35.929 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.930 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.930 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:35.930 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.925032+0000", 2024-04-19T01:04:35.930 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1011000000000002e-05 2024-04-19T01:04:35.930 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.930 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.930 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.930 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.930 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.930 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s6 44/31 ECSubWrite(tid=84, reqid=client.4604.0:281, at_version=44'51, trim_to=0'0, roll_forward_to=43'48))", 2024-04-19T01:04:35.930 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.907467+0000", 2024-04-19T01:04:35.930 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.963279338, 2024-04-19T01:04:35.930 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018626453000000001, 2024-04-19T01:04:35.930 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:35.931 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:35.931 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:35.931 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:35.931 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.931 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:35.931 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.907467+0000", 2024-04-19T01:04:35.931 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.931 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.931 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.931 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:35.931 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.907467+0000", 2024-04-19T01:04:35.931 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.931 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.931 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.931 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:35.932 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.907468+0000", 2024-04-19T01:04:35.932 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3230000000000001e-06 2024-04-19T01:04:35.932 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.932 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.932 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:35.932 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.907725+0000", 2024-04-19T01:04:35.932 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00025646499999999999 2024-04-19T01:04:35.932 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.933 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.933 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:35.933 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.907754+0000", 2024-04-19T01:04:35.933 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9799e-05 2024-04-19T01:04:35.933 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.933 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.933 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:35.933 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.907776+0000", 2024-04-19T01:04:35.933 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1849999999999999e-05 2024-04-19T01:04:35.933 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.933 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.933 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:35.933 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.914523+0000", 2024-04-19T01:04:35.933 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0067470459999999996 2024-04-19T01:04:35.933 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.934 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.934 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:35.934 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.914550+0000", 2024-04-19T01:04:35.934 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6259e-05 2024-04-19T01:04:35.934 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.934 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.934 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:35.934 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.926057+0000", 2024-04-19T01:04:35.934 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011507224999999999 2024-04-19T01:04:35.934 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.934 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.934 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:35.935 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.926093+0000", 2024-04-19T01:04:35.935 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6486000000000003e-05 2024-04-19T01:04:35.935 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.935 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.935 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.935 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.935 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.935 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.0s6 44/31 ECSubWrite(tid=25, reqid=client.4604.0:284, at_version=44'22, trim_to=0'0, roll_forward_to=44'20))", 2024-04-19T01:04:35.935 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.921980+0000", 2024-04-19T01:04:35.935 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.948766594999999, 2024-04-19T01:04:35.935 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.036689795999999997, 2024-04-19T01:04:35.936 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:35.936 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:35.936 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:35.936 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:35.936 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.936 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:35.936 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.921980+0000", 2024-04-19T01:04:35.936 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.936 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.936 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.936 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:35.936 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.921980+0000", 2024-04-19T01:04:35.936 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.936 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.936 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.937 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:35.937 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.921981+0000", 2024-04-19T01:04:35.937 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.378e-06 2024-04-19T01:04:35.937 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.937 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.937 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:35.937 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.923355+0000", 2024-04-19T01:04:35.937 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0013742820000000001 2024-04-19T01:04:35.937 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.937 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.937 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:35.937 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.923387+0000", 2024-04-19T01:04:35.937 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1702999999999998e-05 2024-04-19T01:04:35.937 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.937 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.938 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:35.938 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.923405+0000", 2024-04-19T01:04:35.938 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8071999999999999e-05 2024-04-19T01:04:35.938 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.938 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.938 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:35.938 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.923699+0000", 2024-04-19T01:04:35.938 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029384700000000002 2024-04-19T01:04:35.938 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.938 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.938 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:35.938 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.923727+0000", 2024-04-19T01:04:35.938 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8085999999999998e-05 2024-04-19T01:04:35.938 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.938 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.939 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:35.939 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.958618+0000", 2024-04-19T01:04:35.939 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.034891349000000002 2024-04-19T01:04:35.939 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.939 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.939 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:35.939 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.958669+0000", 2024-04-19T01:04:35.939 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1078999999999997e-05 2024-04-19T01:04:35.939 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.939 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.939 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.939 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.939 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.939 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds6 47/31 ECSubWrite(tid=61, reqid=client.4604.0:330, at_version=47'32, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:04:35.939 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.677697+0000", 2024-04-19T01:04:35.940 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 18.193049650999999, 2024-04-19T01:04:35.940 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.034267463999999997, 2024-04-19T01:04:35.940 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:35.940 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:35.940 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:35.940 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:35.940 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.940 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:35.940 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.677697+0000", 2024-04-19T01:04:35.940 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.941 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.941 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.941 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:35.941 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.677697+0000", 2024-04-19T01:04:35.941 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.941 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.941 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.941 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:35.941 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.677699+0000", 2024-04-19T01:04:35.941 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3929999999999998e-06 2024-04-19T01:04:35.941 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.941 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.941 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:35.941 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678182+0000", 2024-04-19T01:04:35.941 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00048326199999999998 2024-04-19T01:04:35.942 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.942 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.942 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:35.942 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678227+0000", 2024-04-19T01:04:35.942 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4301999999999997e-05 2024-04-19T01:04:35.942 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.942 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.946 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:35.946 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678254+0000", 2024-04-19T01:04:35.946 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7396e-05 2024-04-19T01:04:35.946 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.946 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.946 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:35.946 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678409+0000", 2024-04-19T01:04:35.946 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000154759 2024-04-19T01:04:35.946 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.946 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.947 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:35.947 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678438+0000", 2024-04-19T01:04:35.947 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8926000000000001e-05 2024-04-19T01:04:35.947 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.947 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.947 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:35.947 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711880+0000", 2024-04-19T01:04:35.947 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.033442209000000001 2024-04-19T01:04:35.947 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.947 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.947 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:35.947 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711964+0000", 2024-04-19T01:04:35.947 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4216999999999997e-05 2024-04-19T01:04:35.947 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.948 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.948 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.948 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.948 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.948 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds6 47/31 ECSubWrite(tid=62, reqid=client.4604.0:331, at_version=47'33, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:04:35.948 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.682679+0000", 2024-04-19T01:04:35.948 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 18.188067261, 2024-04-19T01:04:35.948 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.029354201999999999, 2024-04-19T01:04:35.948 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:35.948 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:35.948 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:35.948 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:35.949 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.949 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:35.949 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682679+0000", 2024-04-19T01:04:35.949 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.949 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.949 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.949 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:35.949 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682679+0000", 2024-04-19T01:04:35.949 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.949 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.949 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.949 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:35.950 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682681+0000", 2024-04-19T01:04:35.950 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7799999999999999e-06 2024-04-19T01:04:35.950 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.950 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.950 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:35.950 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682945+0000", 2024-04-19T01:04:35.950 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000263875 2024-04-19T01:04:35.950 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.950 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.950 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:35.950 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682975+0000", 2024-04-19T01:04:35.950 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0216e-05 2024-04-19T01:04:35.950 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.950 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.951 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:35.951 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682990+0000", 2024-04-19T01:04:35.951 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5333e-05 2024-04-19T01:04:35.951 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.951 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.951 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:35.951 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692463+0000", 2024-04-19T01:04:35.951 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0094724660000000006 2024-04-19T01:04:35.951 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.951 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.951 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:35.951 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692479+0000", 2024-04-19T01:04:35.951 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5892e-05 2024-04-19T01:04:35.951 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.952 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.952 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:35.952 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711976+0000", 2024-04-19T01:04:35.952 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019497012000000001 2024-04-19T01:04:35.952 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.952 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.952 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:35.952 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.712033+0000", 2024-04-19T01:04:35.952 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7627999999999998e-05 2024-04-19T01:04:35.952 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.952 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.952 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.952 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.952 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.952 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds6 47/31 ECSubWrite(tid=63, reqid=client.4604.0:332, at_version=47'34, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:04:35.953 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.688944+0000", 2024-04-19T01:04:35.953 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 18.181802399999999, 2024-04-19T01:04:35.953 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023924708999999999, 2024-04-19T01:04:35.953 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:35.953 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:35.953 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:35.953 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:35.953 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.953 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:35.954 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.688944+0000", 2024-04-19T01:04:35.954 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.954 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.954 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.954 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:35.954 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.688944+0000", 2024-04-19T01:04:35.954 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.954 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.955 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.955 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:35.955 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.688945+0000", 2024-04-19T01:04:35.955 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5540000000000001e-06 2024-04-19T01:04:35.955 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.955 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.955 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:35.955 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.689042+0000", 2024-04-19T01:04:35.955 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6903000000000004e-05 2024-04-19T01:04:35.955 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.955 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.955 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:35.955 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.689071+0000", 2024-04-19T01:04:35.956 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8249e-05 2024-04-19T01:04:35.956 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.956 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.956 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:35.956 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.689092+0000", 2024-04-19T01:04:35.956 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1685999999999999e-05 2024-04-19T01:04:35.956 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.956 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.957 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:35.957 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.705134+0000", 2024-04-19T01:04:35.957 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016041454 2024-04-19T01:04:35.957 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.957 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.957 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:35.957 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.705155+0000", 2024-04-19T01:04:35.957 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.177e-05 2024-04-19T01:04:35.957 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.957 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.957 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:35.957 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.712801+0000", 2024-04-19T01:04:35.957 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0076452489999999998 2024-04-19T01:04:35.958 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.958 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.958 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:35.958 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.712869+0000", 2024-04-19T01:04:35.958 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7843999999999996e-05 2024-04-19T01:04:35.958 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.958 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.958 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.958 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.958 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.958 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds6 47/31 ECSubWrite(tid=64, reqid=client.4604.0:333, at_version=47'35, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:04:35.958 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.692117+0000", 2024-04-19T01:04:35.958 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 18.178629567000002, 2024-04-19T01:04:35.959 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020827234, 2024-04-19T01:04:35.959 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:35.959 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:35.959 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:35.959 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:35.959 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.959 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:35.959 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692117+0000", 2024-04-19T01:04:35.959 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.959 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.959 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.959 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:35.959 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692117+0000", 2024-04-19T01:04:35.959 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.959 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.960 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.960 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:35.960 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692118+0000", 2024-04-19T01:04:35.960 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7489999999999999e-06 2024-04-19T01:04:35.960 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.960 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.960 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:35.960 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692163+0000", 2024-04-19T01:04:35.960 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4342999999999999e-05 2024-04-19T01:04:35.960 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.960 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.960 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:35.960 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692204+0000", 2024-04-19T01:04:35.960 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1588999999999998e-05 2024-04-19T01:04:35.961 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.961 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.961 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:35.961 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692227+0000", 2024-04-19T01:04:35.961 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3059000000000001e-05 2024-04-19T01:04:35.961 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.961 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.961 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:35.961 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711560+0000", 2024-04-19T01:04:35.961 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019332623 2024-04-19T01:04:35.961 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.961 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.961 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:35.961 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711575+0000", 2024-04-19T01:04:35.962 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5218999999999999e-05 2024-04-19T01:04:35.962 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.962 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.962 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:35.962 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.712879+0000", 2024-04-19T01:04:35.962 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001303397 2024-04-19T01:04:35.962 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.962 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.962 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:35.962 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.712944+0000", 2024-04-19T01:04:35.962 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5254999999999998e-05 2024-04-19T01:04:35.962 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.962 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.962 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.963 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.963 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.963 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s1 59/31 ECSubWrite(tid=66, reqid=client.4604.0:423, at_version=59'51, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:04:35.963 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.541655+0000", 2024-04-19T01:04:35.963 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.329091215, 2024-04-19T01:04:35.963 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.038825746000000001, 2024-04-19T01:04:35.963 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:35.963 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:35.963 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:35.963 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:35.963 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.963 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:35.963 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.541655+0000", 2024-04-19T01:04:35.963 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.963 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.963 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.964 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:35.964 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.541655+0000", 2024-04-19T01:04:35.964 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.964 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.964 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.964 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:35.964 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.541656+0000", 2024-04-19T01:04:35.964 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1699999999999998e-07 2024-04-19T01:04:35.964 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.964 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.964 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:35.964 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.544278+0000", 2024-04-19T01:04:35.964 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002622728 2024-04-19T01:04:35.964 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.965 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.965 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:35.965 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.544314+0000", 2024-04-19T01:04:35.965 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5751999999999999e-05 2024-04-19T01:04:35.965 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.965 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.965 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:35.965 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.544338+0000", 2024-04-19T01:04:35.965 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4003e-05 2024-04-19T01:04:35.965 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.965 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.965 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:35.965 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.544577+0000", 2024-04-19T01:04:35.965 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00023931500000000001 2024-04-19T01:04:35.965 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.966 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.966 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:35.966 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.544603+0000", 2024-04-19T01:04:35.966 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5854999999999999e-05 2024-04-19T01:04:35.966 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.966 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.966 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:35.966 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.580392+0000", 2024-04-19T01:04:35.966 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.035788477999999999 2024-04-19T01:04:35.966 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.967 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.967 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:35.967 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.580481+0000", 2024-04-19T01:04:35.967 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8998000000000004e-05 2024-04-19T01:04:35.967 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.967 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.967 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.967 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.967 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.967 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s1 59/31 ECSubWrite(tid=67, reqid=client.4604.0:424, at_version=59'52, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:04:35.968 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.549050+0000", 2024-04-19T01:04:35.968 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.3216958429999996, 2024-04-19T01:04:35.968 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.031505582999999997, 2024-04-19T01:04:35.968 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:35.968 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:35.968 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:35.968 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:35.968 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.968 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:35.968 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549050+0000", 2024-04-19T01:04:35.968 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.968 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.968 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.968 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:35.968 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549050+0000", 2024-04-19T01:04:35.969 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.969 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.969 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.969 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:35.969 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549052+0000", 2024-04-19T01:04:35.969 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0669999999999999e-06 2024-04-19T01:04:35.969 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.969 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.969 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:35.969 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.550013+0000", 2024-04-19T01:04:35.969 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00096065699999999996 2024-04-19T01:04:35.969 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.969 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.969 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:35.970 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.550048+0000", 2024-04-19T01:04:35.970 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4535000000000001e-05 2024-04-19T01:04:35.970 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.970 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.970 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:35.970 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.550071+0000", 2024-04-19T01:04:35.970 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3057e-05 2024-04-19T01:04:35.970 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.970 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.970 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:35.970 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.561847+0000", 2024-04-19T01:04:35.970 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.01177603 2024-04-19T01:04:35.970 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.970 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.971 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:35.971 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.561873+0000", 2024-04-19T01:04:35.971 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6679999999999999e-05 2024-04-19T01:04:35.971 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.971 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.971 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:35.971 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.580493+0000", 2024-04-19T01:04:35.971 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018619125 2024-04-19T01:04:35.971 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.971 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.971 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:35.971 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.580556+0000", 2024-04-19T01:04:35.971 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3431999999999998e-05 2024-04-19T01:04:35.971 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.972 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.972 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.972 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.972 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.972 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s1 59/31 ECSubWrite(tid=68, reqid=client.4604.0:425, at_version=59'53, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:04:35.972 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.556579+0000", 2024-04-19T01:04:35.972 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.3141675189999997, 2024-04-19T01:04:35.972 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024080832999999999, 2024-04-19T01:04:35.972 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:35.972 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:35.972 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:35.972 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:35.972 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.972 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:35.973 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556579+0000", 2024-04-19T01:04:35.973 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.973 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.973 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.973 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:35.973 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556579+0000", 2024-04-19T01:04:35.973 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.973 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.973 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.973 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:35.973 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556581+0000", 2024-04-19T01:04:35.973 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0700000000000001e-06 2024-04-19T01:04:35.973 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.973 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.974 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:35.974 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556696+0000", 2024-04-19T01:04:35.974 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011476099999999999 2024-04-19T01:04:35.974 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.974 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.974 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:35.974 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556723+0000", 2024-04-19T01:04:35.974 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.764e-05 2024-04-19T01:04:35.974 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.974 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.974 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:35.974 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556744+0000", 2024-04-19T01:04:35.974 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1229e-05 2024-04-19T01:04:35.974 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.975 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.975 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:35.975 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.573868+0000", 2024-04-19T01:04:35.975 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017123104 2024-04-19T01:04:35.975 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.975 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.975 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:35.975 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.573896+0000", 2024-04-19T01:04:35.975 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8526000000000001e-05 2024-04-19T01:04:35.975 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.975 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.975 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:35.975 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.580562+0000", 2024-04-19T01:04:35.975 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0066657180000000002 2024-04-19T01:04:35.975 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.976 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.976 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:35.976 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.580660+0000", 2024-04-19T01:04:35.976 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7785e-05 2024-04-19T01:04:35.976 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.976 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.976 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.976 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.976 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.976 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s1 59/31 ECSubWrite(tid=69, reqid=client.4604.0:426, at_version=59'54, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:04:35.976 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.558218+0000", 2024-04-19T01:04:35.976 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.3125282909999996, 2024-04-19T01:04:35.976 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024150672000000001, 2024-04-19T01:04:35.976 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:35.976 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:35.977 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:35.977 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:35.977 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.977 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:35.977 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558218+0000", 2024-04-19T01:04:35.977 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.977 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.977 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.977 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:35.978 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558218+0000", 2024-04-19T01:04:35.978 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.978 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.978 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.978 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:35.978 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558219+0000", 2024-04-19T01:04:35.978 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7700000000000004e-07 2024-04-19T01:04:35.978 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.978 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.978 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:35.978 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558256+0000", 2024-04-19T01:04:35.978 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6981999999999999e-05 2024-04-19T01:04:35.978 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.978 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.979 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:35.979 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558275+0000", 2024-04-19T01:04:35.979 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9097e-05 2024-04-19T01:04:35.979 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.979 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.979 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:35.979 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558323+0000", 2024-04-19T01:04:35.979 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7961999999999999e-05 2024-04-19T01:04:35.979 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.979 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.979 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:35.979 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.579875+0000", 2024-04-19T01:04:35.979 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.021552616 2024-04-19T01:04:35.979 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.980 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.980 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:35.980 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.579901+0000", 2024-04-19T01:04:35.980 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6049000000000001e-05 2024-04-19T01:04:35.980 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.980 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.980 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:35.980 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.582293+0000", 2024-04-19T01:04:35.980 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0023912080000000001 2024-04-19T01:04:35.980 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.980 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.980 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:35.980 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.582369+0000", 2024-04-19T01:04:35.980 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5981000000000003e-05 2024-04-19T01:04:35.980 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.981 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.981 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.981 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.981 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.981 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s6 63/31 ECSubWrite(tid=152, reqid=client.4604.0:465, at_version=63'100, trim_to=0'0, roll_forward_to=62'98))", 2024-04-19T01:04:35.981 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:33.566850+0000", 2024-04-19T01:04:35.981 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 2.3038964310000001, 2024-04-19T01:04:35.981 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019378244999999999, 2024-04-19T01:04:35.981 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:35.981 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:35.981 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:35.981 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:35.981 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.981 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:35.981 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:33.566850+0000", 2024-04-19T01:04:35.982 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.982 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.982 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.982 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:35.982 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:33.566850+0000", 2024-04-19T01:04:35.982 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:35.982 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.982 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.982 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:35.982 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:33.566850+0000", 2024-04-19T01:04:35.982 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7300000000000001e-07 2024-04-19T01:04:35.982 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.982 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.983 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:35.983 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:33.567763+0000", 2024-04-19T01:04:35.983 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00091228399999999997 2024-04-19T01:04:35.983 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.983 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.983 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:35.983 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:33.567802+0000", 2024-04-19T01:04:35.983 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9622000000000001e-05 2024-04-19T01:04:35.983 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.983 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.984 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:35.984 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:33.567827+0000", 2024-04-19T01:04:35.984 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4797e-05 2024-04-19T01:04:35.984 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.984 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.984 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:35.984 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:33.568026+0000", 2024-04-19T01:04:35.984 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019901900000000001 2024-04-19T01:04:35.984 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.984 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.984 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:35.984 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:33.568083+0000", 2024-04-19T01:04:35.984 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6570999999999998e-05 2024-04-19T01:04:35.984 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.984 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.985 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:35.985 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:33.586143+0000", 2024-04-19T01:04:35.985 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018060321000000001 2024-04-19T01:04:35.985 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:35.985 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:35.985 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:35.985 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:33.586228+0000", 2024-04-19T01:04:35.985 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5158e-05 2024-04-19T01:04:35.985 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.985 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.985 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.985 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:35.985 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:35.985 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:04:35.986 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-19T01:04:35.988 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:35.986+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:35.988 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-19T01:04:36.088 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:04:36.089+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:04:36.116 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:04:36.116 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-04-19T01:04:36.116 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-04-19T01:04:36.116 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:04:36.130 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-19T01:04:36.148 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-04-19T01:04:36.148 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T01:04:36.168 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-04-19T01:04:36.168 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T01:04:36.187 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-04-19T01:04:36.187 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T01:04:36.188 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:04:36.188+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:04:36.208 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-04-19T01:04:36.209 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T01:04:36.228 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-04-19T01:04:36.228 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T01:04:36.251 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-04-19T01:04:36.251 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T01:04:36.255 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:04:36.255 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-04-19T01:04:36.255 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-04-19T01:04:36.255 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-04-19T01:04:36.255 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:04:36.269 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-19T01:04:36.273 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-04-19T01:04:36.273 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T01:04:36.289 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:04:36.290+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:04:36.300 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-04-19T01:04:36.301 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T01:04:36.317 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2024-04-19T01:04:36.317 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T01:04:36.342 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2024-04-19T01:04:36.342 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T01:04:36.365 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2024-04-19T01:04:36.365 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T01:04:36.388 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2024-04-19T01:04:36.389 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T01:04:36.390 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:36.386+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:36.410 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2024-04-19T01:04:36.411 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-19T01:04:36.434 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2024-04-19T01:04:36.435 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-19T01:04:36.435 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:36.435 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:36.449 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-04-19T01:04:36.457 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2024-04-19T01:04:36.458 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-19T01:04:36.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:04:36.478+0000 7f3701a47640 0 -- 172.21.15.33:0/3271648932 >> [v2:172.21.15.155:6807/741132602,v1:172.21.15.155:6810/741132602] conn(0x7f36fc2245c0 msgr2=0x7f36fc2269b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-19T01:04:36.483 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2024-04-19T01:04:36.491 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:04:36.490+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:04:36.507 INFO:tasks.rados.rados.0.smithi155.stdout:195: done (4 left) 2024-04-19T01:04:36.507 INFO:tasks.rados.rados.0.smithi155.stdout:196: done (3 left) 2024-04-19T01:04:36.507 INFO:tasks.rados.rados.0.smithi155.stdout:197: done (2 left) 2024-04-19T01:04:36.507 INFO:tasks.rados.rados.0.smithi155.stdout:198: done (1 left) 2024-04-19T01:04:36.507 INFO:tasks.rados.rados.0.smithi155.stdout:199: done (0 left) 2024-04-19T01:04:36.508 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 46 current snap is 18 2024-04-19T01:04:36.508 INFO:tasks.rados.rados.0.smithi155.stdout:200: seq_num 93 ranges {770048=802816,1572864=638976,2211840=131072} 2024-04-19T01:04:36.524 INFO:tasks.rados.rados.0.smithi155.stdout:200: writing smithi15528762-46 from 770048 to 1572864 tid 1 2024-04-19T01:04:36.532 INFO:tasks.rados.rados.0.smithi155.stdout:200: writing smithi15528762-46 from 1572864 to 2211840 tid 2 2024-04-19T01:04:36.534 INFO:tasks.rados.rados.0.smithi155.stdout:200: writing smithi15528762-46 from 2211840 to 2342912 tid 3 2024-04-19T01:04:36.534 INFO:tasks.rados.rados.0.smithi155.stdout:201: read oid 30 snap -1 2024-04-19T01:04:36.534 INFO:tasks.rados.rados.0.smithi155.stdout:201: expect (ObjNum 87 snap 15 seq_num 87) 2024-04-19T01:04:36.534 INFO:tasks.rados.rados.0.smithi155.stdout:202: read oid 14 snap 17 2024-04-19T01:04:36.534 INFO:tasks.rados.rados.0.smithi155.stdout:202: expect (ObjNum 65 snap 6 seq_num 65) 2024-04-19T01:04:36.534 INFO:tasks.rados.rados.0.smithi155.stdout:203: snap_remove snap 14 2024-04-19T01:04:36.578 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:36.578 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:36.593 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-04-19T01:04:36.593 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:04:36.590+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:04:36.620 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-04-19T01:04:36.620 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-19T01:04:36.694 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:04:36.694+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:04:36.716 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:36.716 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:36.730 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-04-19T01:04:36.796 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:04:36.794+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:04:36.847 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:36.847 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:36.861 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-19T01:04:36.912 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:04:36.909+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:04:36.991 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:36.991 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:36.997 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:36.998+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:37.005 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-19T01:04:37.097 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:04:37.097+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:04:37.169 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:37.169 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:37.183 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-19T01:04:37.198 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:37.198+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:37.300 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:04:37.301+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:04:37.344 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:37.344 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:37.358 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-19T01:04:37.400 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:04:37.398+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:04:37.502 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:04:37.501+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:04:37.505 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.0 to 1 (10000) 2024-04-19T01:04:37.512 INFO:tasks.rados.rados.0.smithi155.stdout:203: done (3 left) 2024-04-19T01:04:37.512 INFO:tasks.rados.rados.0.smithi155.stdout:204: snap_create 2024-04-19T01:04:37.512 INFO:tasks.rados.rados.0.smithi155.stdout:201: expect (ObjNum 87 snap 15 seq_num 87) 2024-04-19T01:04:37.529 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:37.529 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:37.530 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-19T01:04:37.540 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-19T01:04:37.569 INFO:tasks.rados.rados.0.smithi155.stdout:202: expect (ObjNum 65 snap 6 seq_num 65) 2024-04-19T01:04:37.590 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-19T01:04:37.606 INFO:tasks.rados.rados.0.smithi155.stdout:200: finishing write tid 1 to smithi15528762-46 2024-04-19T01:04:37.606 INFO:tasks.rados.rados.0.smithi155.stdout:200: finishing write tid 2 to smithi15528762-46 2024-04-19T01:04:37.607 INFO:tasks.rados.rados.0.smithi155.stdout:200: finishing write tid 3 to smithi15528762-46 2024-04-19T01:04:37.607 INFO:tasks.rados.rados.0.smithi155.stdout:200: finishing write tid 4 to smithi15528762-46 2024-04-19T01:04:37.607 INFO:tasks.rados.rados.0.smithi155.stdout:200: finishing write tid 5 to smithi15528762-46 2024-04-19T01:04:37.607 INFO:tasks.rados.rados.0.smithi155.stdout:200: oid 46 updating version 0 to 43 2024-04-19T01:04:37.607 INFO:tasks.rados.rados.0.smithi155.stdout:200: oid 46 updating version 43 to 45 2024-04-19T01:04:37.607 INFO:tasks.rados.rados.0.smithi155.stdout:200: oid 46 version 45 is already newer than 44 2024-04-19T01:04:37.607 INFO:tasks.rados.rados.0.smithi155.stdout:200: oid 46 updating version 45 to 46 2024-04-19T01:04:37.607 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 46 v 46 (ObjNum 93 snap 18 seq_num 93) dirty exists 2024-04-19T01:04:37.607 INFO:tasks.rados.rados.0.smithi155.stdout:200: left oid 46 (ObjNum 93 snap 18 seq_num 93) 2024-04-19T01:04:37.607 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:04:37.605+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:04:37.700 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:37.700 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:37.703 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:04:37.702+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:04:37.708 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-19T01:04:37.804 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:04:37.804+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:04:37.876 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:37.876 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:37.890 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-19T01:04:37.905 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:04:37.902+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:04:38.009 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:04:38.005+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:04:38.011 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:38.011 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:38.026 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-19T01:04:38.108 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:04:38.106+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:04:38.191 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:38.191 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:38.205 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-19T01:04:38.209 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:38.206+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:38.310 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:38.312+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:38.329 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:38.329 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:38.347 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-04-19T01:04:38.412 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:38.412+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:38.471 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:38.471 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:38.485 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-04-19T01:04:38.503 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-19T01:04:38.512 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:38.512+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:38.512 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.1 to 1 (10000) 2024-04-19T01:04:38.513 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2024-04-19T01:04:38.519 INFO:tasks.rados.rados.0.smithi155.stdout:200: done (3 left) 2024-04-19T01:04:38.519 INFO:tasks.rados.rados.0.smithi155.stdout:201: done (2 left) 2024-04-19T01:04:38.519 INFO:tasks.rados.rados.0.smithi155.stdout:202: done (1 left) 2024-04-19T01:04:38.519 INFO:tasks.rados.rados.0.smithi155.stdout:204: done (0 left) 2024-04-19T01:04:38.519 INFO:tasks.rados.rados.0.smithi155.stdout:205: rmattr oid 12 current snap is 19 2024-04-19T01:04:38.519 INFO:tasks.rados.rados.0.smithi155.stdout:205: done (0 left) 2024-04-19T01:04:38.519 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 13 current snap is 19 2024-04-19T01:04:38.520 INFO:tasks.rados.rados.0.smithi155.stdout:206: seq_num 94 ranges {1753088=425984,2179072=786432,2965504=376832} 2024-04-19T01:04:38.534 INFO:tasks.rados.rados.0.smithi155.stdout:206: writing smithi15528762-13 from 1753088 to 2179072 tid 1 2024-04-19T01:04:38.542 INFO:tasks.rados.rados.0.smithi155.stdout:206: writing smithi15528762-13 from 2179072 to 2965504 tid 2 2024-04-19T01:04:38.543 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-19T01:04:38.543 INFO:tasks.rados.rados.0.smithi155.stdout:206: writing smithi15528762-13 from 2965504 to 3342336 tid 3 2024-04-19T01:04:38.544 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 7 current snap is 19 2024-04-19T01:04:38.544 INFO:tasks.rados.rados.0.smithi155.stdout:207: seq_num 95 ranges {1474560=442368,1916928=540672,2457600=622592} 2024-04-19T01:04:38.556 INFO:tasks.rados.rados.0.smithi155.stdout:207: writing smithi15528762-7 from 1474560 to 1916928 tid 1 2024-04-19T01:04:38.560 INFO:tasks.rados.rados.0.smithi155.stdout:207: writing smithi15528762-7 from 1916928 to 2457600 tid 2 2024-04-19T01:04:38.565 INFO:tasks.rados.rados.0.smithi155.stdout:207: writing smithi15528762-7 from 2457600 to 3080192 tid 3 2024-04-19T01:04:38.565 INFO:tasks.rados.rados.0.smithi155.stdout:208: snap_create 2024-04-19T01:04:38.566 INFO:tasks.rados.rados.0.smithi155.stdout:206: finishing write tid 1 to smithi15528762-13 2024-04-19T01:04:38.585 INFO:tasks.rados.rados.0.smithi155.stdout:206: finishing write tid 2 to smithi15528762-13 2024-04-19T01:04:38.585 INFO:tasks.rados.rados.0.smithi155.stdout:206: finishing write tid 3 to smithi15528762-13 2024-04-19T01:04:38.585 INFO:tasks.rados.rados.0.smithi155.stdout:206: finishing write tid 4 to smithi15528762-13 2024-04-19T01:04:38.585 INFO:tasks.rados.rados.0.smithi155.stdout:206: finishing write tid 5 to smithi15528762-13 2024-04-19T01:04:38.585 INFO:tasks.rados.rados.0.smithi155.stdout:206: oid 13 updating version 0 to 50 2024-04-19T01:04:38.585 INFO:tasks.rados.rados.0.smithi155.stdout:206: oid 13 version 50 is already newer than 48 2024-04-19T01:04:38.585 INFO:tasks.rados.rados.0.smithi155.stdout:206: oid 13 version 50 is already newer than 49 2024-04-19T01:04:38.585 INFO:tasks.rados.rados.0.smithi155.stdout:206: oid 13 version 50 is already newer than 47 2024-04-19T01:04:38.586 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 13 v 50 (ObjNum 94 snap 19 seq_num 94) dirty exists 2024-04-19T01:04:38.586 INFO:tasks.rados.rados.0.smithi155.stdout:206: left oid 13 (ObjNum 94 snap 19 seq_num 94) 2024-04-19T01:04:38.588 INFO:tasks.rados.rados.0.smithi155.stdout:207: finishing write tid 1 to smithi15528762-7 2024-04-19T01:04:38.589 INFO:tasks.rados.rados.0.smithi155.stdout:207: finishing write tid 2 to smithi15528762-7 2024-04-19T01:04:38.608 INFO:tasks.rados.rados.0.smithi155.stdout:207: finishing write tid 3 to smithi15528762-7 2024-04-19T01:04:38.608 INFO:tasks.rados.rados.0.smithi155.stdout:207: finishing write tid 4 to smithi15528762-7 2024-04-19T01:04:38.614 INFO:tasks.rados.rados.0.smithi155.stdout:207: finishing write tid 5 to smithi15528762-7 2024-04-19T01:04:38.614 INFO:tasks.rados.rados.0.smithi155.stdout:207: oid 7 updating version 0 to 36 2024-04-19T01:04:38.614 INFO:tasks.rados.rados.0.smithi155.stdout:207: oid 7 version 36 is already newer than 35 2024-04-19T01:04:38.614 INFO:tasks.rados.rados.0.smithi155.stdout:207: oid 7 updating version 36 to 37 2024-04-19T01:04:38.614 INFO:tasks.rados.rados.0.smithi155.stdout:207: oid 7 updating version 37 to 38 2024-04-19T01:04:38.614 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 7 v 38 (ObjNum 95 snap 19 seq_num 95) dirty exists 2024-04-19T01:04:38.614 INFO:tasks.rados.rados.0.smithi155.stdout:207: left oid 7 (ObjNum 95 snap 19 seq_num 95) 2024-04-19T01:04:38.627 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:38.627 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:38.642 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-04-19T01:04:38.643 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:04:38.642+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:04:38.664 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-04-19T01:04:38.664 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T01:04:38.685 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-04-19T01:04:38.685 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T01:04:38.707 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-04-19T01:04:38.707 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T01:04:38.714 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:04:38.712+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:04:38.725 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-04-19T01:04:38.726 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T01:04:38.742 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-04-19T01:04:38.742 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T01:04:38.764 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-04-19T01:04:38.764 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T01:04:38.787 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-04-19T01:04:38.787 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T01:04:38.805 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-04-19T01:04:38.806 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T01:04:38.813 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:38.813 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:38.814 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:38.816+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:38.822 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2024-04-19T01:04:38.822 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T01:04:38.827 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-19T01:04:38.842 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2024-04-19T01:04:38.843 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T01:04:38.863 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2024-04-19T01:04:38.863 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T01:04:38.884 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2024-04-19T01:04:38.885 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T01:04:38.902 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2024-04-19T01:04:38.903 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-19T01:04:38.915 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:38.914+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:38.922 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2024-04-19T01:04:38.922 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-19T01:04:38.943 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2024-04-19T01:04:38.943 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-19T01:04:38.943 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:38.943 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:04:38.944 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2024-04-19T01:04:38.944 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:38.956 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-19T01:04:39.046 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:04:39.046+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:04:39.117 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:39.117 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:04:39.118 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2024-04-19T01:04:39.118 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2024-04-19T01:04:39.118 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:39.118 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:39.114+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:39.135 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-19T01:04:39.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:04:39.146+0000 7fc4a6d9d640 0 -- 172.21.15.33:0/4122627043 >> [v2:172.21.15.155:6807/741132602,v1:172.21.15.155:6810/741132602] conn(0x7fc4a8226a00 msgr2=0x7fc4a8228df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-19T01:04:39.169 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2024-04-19T01:04:39.169 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-19T01:04:39.227 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:04:39.218+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:04:39.275 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:39.276 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2024-04-19T01:04:39.276 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2024-04-19T01:04:39.276 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-04-19T01:04:39.276 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.276 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:172 3.cs0 3:356739ec:::smithi15528762-4:head [create,setxattr aSrcO`NnS[qadSjpQT[PSmfh\\PfYagpctmbQ_kXi (1753) in=40b,setxattr hratdsb^VNONN^^acUkPeXNZohcdVkmXXhdTeofs (2805) in=40b,setxattr abOo``aaqamjUn`hc`Qde]XU\\lqYVrmVdcimNs_a (35) in=40b,setxattr ZheopalZpQPnebTbena`gaWXtmaSTdQkji`aZYOi (34) in=40b,setxattr katXYockjtYtieO]`jgpSrZZ\\mq`gOYplPOVal`Y (842) in=40b,setxattr [^]lbsa\\UX[na]akS\\hYcdqU^]aWe_[U[OlaQXUY (1894) in=40b,setxattr We][_Xheab\\Ylg`q^hsi`XTX_tQS`Ng\\]aqZTnV[ (1473) in=40b,setxattr TTRSo^W`XoqVWpSjRgc[adm`ROSSVYUNlsilh\\kS (911) in=40b,setxattr bqbUQROpngcT`ndhrZQP`iraoUgs[ZZatacnZoc_ (2876) in=40b,setxattr qPPSibdg_e\\Y_V`kTQO^\\`_jr^jt\\aiadkepqYUj (3576) in=40b,setxattr q`N\\bPOqejbpq`QZVQfWgjScjfahlgXWodnRReWS (1017) in=40b,setxattr `pdT]PSaeeZ^jhWYc]iO`^tRXWcjedgQdke^adpf (7928) in=40b,setxattr matSmfe^casVRfWbNqkt_mZNWbhhh`\\Waf\\djeca (7157) in=40b,setxattr ]SWX\\hrhracj[TeSRYP[bZ[[lSSRghmfarTYpp]U (735) in=40b,setxattr WrnQrVa (13964) in=7b,setxattr PcP_]XU]rhapTs\\X`ojNmm_Qsjkkjecf`VgrdTrR (1542) in=40b,setxattr o^RO^oeZlji]\\\\UhVsnb]t_WXomaRr^higjSUShQ (1016) in=40b,setxattr Z_ofVVcfZdUV[UaRnTV_Ni^_rRNl`\\kt`]aaaoed (701) in=40b,setxattr X[m^[`bVlbgi``^mWZ]cl[S_nn[Ok\\ldS\\_h]]qk (1999) in=40b,setxattr fpmQVVnVPSmZmt`OeaT`adhoNjhpOb[`\\aPWQ^Rd (700) in=40b] snapc 5=[5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-19T01:04:39.276 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:08.804031+0000", 2024-04-19T01:04:39.276 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 30.469695545, 2024-04-19T01:04:39.276 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014732528999999999, 2024-04-19T01:04:39.276 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:39.276 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:39.276 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:39.276 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:39.276 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:39.277 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:39.277 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 172 2024-04-19T01:04:39.277 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.277 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:39.277 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.277 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:39.277 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.804031+0000", 2024-04-19T01:04:39.277 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.277 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.277 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.277 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:39.277 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.804031+0000", 2024-04-19T01:04:39.277 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.277 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.278 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.278 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:39.278 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.804034+0000", 2024-04-19T01:04:39.278 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5389999999999999e-06 2024-04-19T01:04:39.278 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.278 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.278 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:39.278 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.804179+0000", 2024-04-19T01:04:39.278 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014485600000000001 2024-04-19T01:04:39.278 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.278 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.278 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:39.278 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.804208+0000", 2024-04-19T01:04:39.278 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8615999999999998e-05 2024-04-19T01:04:39.279 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.279 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.279 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:39.279 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.804223+0000", 2024-04-19T01:04:39.279 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5244e-05 2024-04-19T01:04:39.279 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.282 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.282 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.284 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.804362+0000", 2024-04-19T01:04:39.284 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013886000000000001 2024-04-19T01:04:39.284 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.284 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.284 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op must wait for map", 2024-04-19T01:04:39.284 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.804377+0000", 2024-04-19T01:04:39.284 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4832e-05 2024-04-19T01:04:39.284 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.284 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.284 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.284 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.809596+0000", 2024-04-19T01:04:39.284 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0052189250000000001 2024-04-19T01:04:39.285 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.285 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.285 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:39.285 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.809925+0000", 2024-04-19T01:04:39.285 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00032908900000000002 2024-04-19T01:04:39.286 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.286 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.286 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:39.286 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.813383+0000", 2024-04-19T01:04:39.286 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0034579929999999999 2024-04-19T01:04:39.286 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.286 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.286 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:39.286 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.815667+0000", 2024-04-19T01:04:39.286 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.002284632 2024-04-19T01:04:39.286 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.286 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.286 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:39.286 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.818639+0000", 2024-04-19T01:04:39.287 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.002972003 2024-04-19T01:04:39.287 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.287 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.287 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:39.287 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.818763+0000", 2024-04-19T01:04:39.287 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012394000000000001 2024-04-19T01:04:39.288 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.288 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:39.288 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.288 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.288 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.288 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:232 3.cs0 3:3233f066:::smithi15528762-50:2 [read 0~835584,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:39.288 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.838358+0000", 2024-04-19T01:04:39.288 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 25.435368835999999, 2024-04-19T01:04:39.288 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011277769, 2024-04-19T01:04:39.288 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:39.288 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:39.288 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:39.288 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:39.288 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:39.288 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:39.289 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 232 2024-04-19T01:04:39.289 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.289 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:39.289 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.289 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:39.289 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838358+0000", 2024-04-19T01:04:39.289 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.289 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.289 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.289 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:39.289 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838358+0000", 2024-04-19T01:04:39.289 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.289 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.289 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.290 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:39.290 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838360+0000", 2024-04-19T01:04:39.290 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8770000000000002e-06 2024-04-19T01:04:39.290 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.290 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.290 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:39.290 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838388+0000", 2024-04-19T01:04:39.290 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7282000000000001e-05 2024-04-19T01:04:39.290 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.290 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.290 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:39.290 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838427+0000", 2024-04-19T01:04:39.290 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9242999999999997e-05 2024-04-19T01:04:39.290 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.290 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.291 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:39.291 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838444+0000", 2024-04-19T01:04:39.291 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7127000000000001e-05 2024-04-19T01:04:39.291 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.291 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.291 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.291 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838641+0000", 2024-04-19T01:04:39.291 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00019704499999999999 2024-04-19T01:04:39.291 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.291 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.291 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op must wait for map", 2024-04-19T01:04:39.291 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838653+0000", 2024-04-19T01:04:39.291 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2187999999999999e-05 2024-04-19T01:04:39.291 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.292 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.292 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.292 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.842741+0000", 2024-04-19T01:04:39.292 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0040879879999999999 2024-04-19T01:04:39.295 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.295 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.295 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:39.295 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.842898+0000", 2024-04-19T01:04:39.295 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015647299999999999 2024-04-19T01:04:39.295 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.295 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.295 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:39.296 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.849635+0000", 2024-04-19T01:04:39.296 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0067375459999999996 2024-04-19T01:04:39.296 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.296 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:39.296 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.296 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.296 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.296 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:233 3.cs0 3:3233f066:::smithi15528762-50:2 [read 0~835584,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:39.296 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.838487+0000", 2024-04-19T01:04:39.296 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 25.435239822, 2024-04-19T01:04:39.296 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013873264, 2024-04-19T01:04:39.296 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:39.296 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:39.297 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:39.297 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:39.297 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:39.297 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:39.297 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 233 2024-04-19T01:04:39.297 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.297 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:39.297 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.297 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:39.297 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838487+0000", 2024-04-19T01:04:39.297 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.297 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.300 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.300 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:39.300 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838487+0000", 2024-04-19T01:04:39.300 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.301 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.301 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.301 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:39.301 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838488+0000", 2024-04-19T01:04:39.301 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0380000000000001e-06 2024-04-19T01:04:39.301 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.301 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.301 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:39.301 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838495+0000", 2024-04-19T01:04:39.301 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.8299999999999996e-06 2024-04-19T01:04:39.301 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.301 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.301 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:39.301 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838509+0000", 2024-04-19T01:04:39.302 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3954000000000001e-05 2024-04-19T01:04:39.302 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.302 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.302 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:39.302 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838519+0000", 2024-04-19T01:04:39.302 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.9729999999999992e-06 2024-04-19T01:04:39.302 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.302 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.302 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.303 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838755+0000", 2024-04-19T01:04:39.303 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00023589999999999999 2024-04-19T01:04:39.303 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.303 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.303 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:04:39.303 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838764+0000", 2024-04-19T01:04:39.303 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.0429999999999996e-06 2024-04-19T01:04:39.303 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.304 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.304 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.304 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.844563+0000", 2024-04-19T01:04:39.304 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0057983749999999997 2024-04-19T01:04:39.304 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.304 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.304 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:39.305 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.844897+0000", 2024-04-19T01:04:39.305 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00033450199999999998 2024-04-19T01:04:39.305 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.305 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.305 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:39.305 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.852360+0000", 2024-04-19T01:04:39.305 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.007462649 2024-04-19T01:04:39.305 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.305 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:39.305 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.305 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.305 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.305 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:234 3.cs0 3:3233f066:::smithi15528762-50:2 [read 0~835584,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:39.305 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.838536+0000", 2024-04-19T01:04:39.305 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 25.435190336000002, 2024-04-19T01:04:39.306 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.016682111999999999, 2024-04-19T01:04:39.306 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:39.306 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:39.306 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:39.306 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:39.306 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:39.306 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:39.306 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 234 2024-04-19T01:04:39.306 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.306 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:39.306 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.306 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:39.307 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838536+0000", 2024-04-19T01:04:39.307 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.307 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.307 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.307 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:39.307 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838536+0000", 2024-04-19T01:04:39.307 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.307 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.307 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.307 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:39.307 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838537+0000", 2024-04-19T01:04:39.308 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1200000000000002e-07 2024-04-19T01:04:39.308 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.308 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.308 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:39.308 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838542+0000", 2024-04-19T01:04:39.308 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3789999999999997e-06 2024-04-19T01:04:39.308 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.308 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.308 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:39.308 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838553+0000", 2024-04-19T01:04:39.308 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0986e-05 2024-04-19T01:04:39.308 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.308 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.308 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:39.308 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838564+0000", 2024-04-19T01:04:39.309 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0895e-05 2024-04-19T01:04:39.309 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.309 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.309 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.309 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.840056+0000", 2024-04-19T01:04:39.309 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014917229999999999 2024-04-19T01:04:39.309 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.309 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.309 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:04:39.309 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.840066+0000", 2024-04-19T01:04:39.309 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.9329999999999996e-06 2024-04-19T01:04:39.310 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.310 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.310 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.310 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.846974+0000", 2024-04-19T01:04:39.310 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0069077770000000004 2024-04-19T01:04:39.310 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.310 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.310 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:39.310 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.847132+0000", 2024-04-19T01:04:39.310 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015876400000000001 2024-04-19T01:04:39.310 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.310 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.310 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:39.310 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.855218+0000", 2024-04-19T01:04:39.311 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0080858430000000005 2024-04-19T01:04:39.311 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.311 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:39.311 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.311 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.311 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.311 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:263 3.fs0 3:f8a60714:::smithi15528762-14:head [append 0~720896] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:39.311 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.990535+0000", 2024-04-19T01:04:39.311 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 25.283191049999999, 2024-04-19T01:04:39.311 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.043978092000000003, 2024-04-19T01:04:39.311 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:39.311 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:39.311 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:39.311 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:39.311 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:39.312 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:39.312 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 263 2024-04-19T01:04:39.312 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.312 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:39.312 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.312 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:39.312 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.990535+0000", 2024-04-19T01:04:39.312 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.312 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.312 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.312 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:39.312 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.990535+0000", 2024-04-19T01:04:39.312 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.312 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.313 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.313 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:39.313 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.990537+0000", 2024-04-19T01:04:39.313 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4160000000000001e-06 2024-04-19T01:04:39.313 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.313 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.313 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:39.313 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991425+0000", 2024-04-19T01:04:39.313 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00088775099999999999 2024-04-19T01:04:39.314 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.314 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.314 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:39.314 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991440+0000", 2024-04-19T01:04:39.314 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5305999999999999e-05 2024-04-19T01:04:39.314 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.314 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.314 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:39.314 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991447+0000", 2024-04-19T01:04:39.314 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.4760000000000001e-06 2024-04-19T01:04:39.314 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.314 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.314 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.314 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991516+0000", 2024-04-19T01:04:39.315 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.9041999999999994e-05 2024-04-19T01:04:39.315 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.315 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.315 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:39.315 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991620+0000", 2024-04-19T01:04:39.315 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000103282 2024-04-19T01:04:39.315 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.315 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.315 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:39.315 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993018+0000", 2024-04-19T01:04:39.315 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001397976 2024-04-19T01:04:39.315 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.315 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.316 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:39.316 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.004440+0000", 2024-04-19T01:04:39.316 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011422261 2024-04-19T01:04:39.316 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.316 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.316 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:39.316 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.034454+0000", 2024-04-19T01:04:39.316 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.030013782999999999 2024-04-19T01:04:39.316 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.316 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.316 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:39.316 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.034513+0000", 2024-04-19T01:04:39.317 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.9799000000000001e-05 2024-04-19T01:04:39.317 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.317 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:39.317 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.317 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.317 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.317 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:264 3.fs0 3:f8a60714:::smithi15528762-14:head [append 0~376832] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:39.317 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.992666+0000", 2024-04-19T01:04:39.317 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 25.281060277999998, 2024-04-19T01:04:39.317 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.041981560000000001, 2024-04-19T01:04:39.317 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:39.317 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:39.317 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:39.317 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:39.318 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:39.318 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:39.318 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 264 2024-04-19T01:04:39.318 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.318 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:39.318 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.318 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:39.318 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.992666+0000", 2024-04-19T01:04:39.318 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.318 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.318 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.319 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:39.319 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.992666+0000", 2024-04-19T01:04:39.319 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.319 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.319 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.319 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:39.319 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.992668+0000", 2024-04-19T01:04:39.319 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.483e-06 2024-04-19T01:04:39.319 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.319 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.320 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:39.320 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993021+0000", 2024-04-19T01:04:39.320 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000352958 2024-04-19T01:04:39.320 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.320 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.320 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:39.320 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993031+0000", 2024-04-19T01:04:39.320 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0723000000000001e-05 2024-04-19T01:04:39.320 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.320 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.320 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:39.321 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993040+0000", 2024-04-19T01:04:39.321 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.507e-06 2024-04-19T01:04:39.321 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.321 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.321 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.321 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.000363+0000", 2024-04-19T01:04:39.321 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0073236819999999998 2024-04-19T01:04:39.321 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.321 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.321 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:39.321 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.000482+0000", 2024-04-19T01:04:39.321 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011817299999999999 2024-04-19T01:04:39.321 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.321 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.322 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:39.322 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.001332+0000", 2024-04-19T01:04:39.322 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000849886 2024-04-19T01:04:39.322 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.322 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.322 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:39.322 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.006292+0000", 2024-04-19T01:04:39.322 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0049605090000000001 2024-04-19T01:04:39.322 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.322 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.322 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:39.322 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.034602+0000", 2024-04-19T01:04:39.322 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.028310076999999999 2024-04-19T01:04:39.322 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.323 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.323 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:39.323 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.034648+0000", 2024-04-19T01:04:39.323 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5562000000000002e-05 2024-04-19T01:04:39.323 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.323 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:39.323 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.323 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.323 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.323 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:265 3.fs0 3:f8a60714:::smithi15528762-14:head [setxattr _header (56) in=7b] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:39.323 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.993067+0000", 2024-04-19T01:04:39.328 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 25.28065904, 2024-04-19T01:04:39.328 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.041773988999999997, 2024-04-19T01:04:39.328 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:39.328 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:39.328 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:39.328 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:39.328 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:39.329 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:39.329 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 265 2024-04-19T01:04:39.329 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.329 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:39.329 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.329 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:39.329 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993067+0000", 2024-04-19T01:04:39.329 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.329 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.329 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.329 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:39.329 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993067+0000", 2024-04-19T01:04:39.329 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.329 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.330 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.330 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:39.330 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993068+0000", 2024-04-19T01:04:39.330 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2699999999999999e-07 2024-04-19T01:04:39.330 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.330 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.330 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:39.330 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993072+0000", 2024-04-19T01:04:39.330 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0539999999999996e-06 2024-04-19T01:04:39.331 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.331 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.331 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:39.331 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993080+0000", 2024-04-19T01:04:39.331 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6140000000000003e-06 2024-04-19T01:04:39.331 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.331 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.331 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:39.331 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993119+0000", 2024-04-19T01:04:39.331 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8754000000000002e-05 2024-04-19T01:04:39.331 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.331 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.331 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.332 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.004460+0000", 2024-04-19T01:04:39.332 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011341188 2024-04-19T01:04:39.332 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.332 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.332 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:39.332 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.004518+0000", 2024-04-19T01:04:39.332 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8306e-05 2024-04-19T01:04:39.332 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.332 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.332 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:39.332 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.004774+0000", 2024-04-19T01:04:39.332 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00025565400000000002 2024-04-19T01:04:39.332 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.332 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.333 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:39.333 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.006318+0000", 2024-04-19T01:04:39.333 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0015444409999999999 2024-04-19T01:04:39.333 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.333 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.333 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:39.333 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.034787+0000", 2024-04-19T01:04:39.333 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.028468566000000001 2024-04-19T01:04:39.333 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.333 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.333 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:39.333 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.034841+0000", 2024-04-19T01:04:39.333 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4685000000000002e-05 2024-04-19T01:04:39.333 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.333 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:39.334 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.334 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.334 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.334 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:266 3.fs0 3:f8a60714:::smithi15528762-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:39.334 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.993144+0000", 2024-04-19T01:04:39.334 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 25.280581987000001, 2024-04-19T01:04:39.334 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.042235719999999997, 2024-04-19T01:04:39.334 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:39.334 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:39.334 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:39.334 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:39.334 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:39.334 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:39.335 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 266 2024-04-19T01:04:39.335 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.335 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:39.335 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.335 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:39.335 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993144+0000", 2024-04-19T01:04:39.335 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.335 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.335 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.335 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:39.335 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993144+0000", 2024-04-19T01:04:39.335 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.335 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.336 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.336 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:39.336 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993145+0000", 2024-04-19T01:04:39.336 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5099999999999999e-07 2024-04-19T01:04:39.336 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.336 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.336 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:39.336 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993149+0000", 2024-04-19T01:04:39.336 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4539999999999999e-06 2024-04-19T01:04:39.336 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.336 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.336 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:39.336 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993157+0000", 2024-04-19T01:04:39.336 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.6349999999999995e-06 2024-04-19T01:04:39.337 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.337 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.337 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:39.337 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993166+0000", 2024-04-19T01:04:39.337 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2709999999999998e-06 2024-04-19T01:04:39.337 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.337 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.337 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.337 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.004924+0000", 2024-04-19T01:04:39.337 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011757834 2024-04-19T01:04:39.337 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.337 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.337 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:04:39.337 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.004971+0000", 2024-04-19T01:04:39.338 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6947000000000003e-05 2024-04-19T01:04:39.338 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.338 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.338 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.338 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.035366+0000", 2024-04-19T01:04:39.338 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.03039501 2024-04-19T01:04:39.338 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.338 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.338 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:39.338 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.035380+0000", 2024-04-19T01:04:39.338 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3918e-05 2024-04-19T01:04:39.338 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.338 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:39.338 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.339 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.339 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.339 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:317 3.fs0 3:f23900a8:::smithi15528762-6:head [append 0~475136] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:04:39.339 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.601883+0000", 2024-04-19T01:04:39.339 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 21.671843009, 2024-04-19T01:04:39.339 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.029134103000000001, 2024-04-19T01:04:39.339 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:39.339 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:39.339 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:39.339 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:39.339 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:39.339 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:39.339 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 317 2024-04-19T01:04:39.339 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.340 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:39.340 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.340 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:39.340 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.601883+0000", 2024-04-19T01:04:39.340 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.340 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.340 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.340 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:39.340 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.601883+0000", 2024-04-19T01:04:39.340 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.340 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.340 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.340 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:39.340 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.601886+0000", 2024-04-19T01:04:39.341 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.2709999999999999e-06 2024-04-19T01:04:39.341 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.341 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.341 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:39.341 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.602696+0000", 2024-04-19T01:04:39.341 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00080995700000000004 2024-04-19T01:04:39.341 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.341 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.341 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:39.341 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.602714+0000", 2024-04-19T01:04:39.341 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8168999999999998e-05 2024-04-19T01:04:39.341 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.341 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.341 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:39.342 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.602721+0000", 2024-04-19T01:04:39.342 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.554e-06 2024-04-19T01:04:39.342 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.342 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.342 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.342 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.602799+0000", 2024-04-19T01:04:39.342 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7804000000000002e-05 2024-04-19T01:04:39.342 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.342 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.342 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:39.342 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.602920+0000", 2024-04-19T01:04:39.342 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012105299999999999 2024-04-19T01:04:39.342 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.342 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.343 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:39.343 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.604215+0000", 2024-04-19T01:04:39.343 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0012944129999999999 2024-04-19T01:04:39.343 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.343 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.343 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:39.343 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.609630+0000", 2024-04-19T01:04:39.343 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0054153789999999997 2024-04-19T01:04:39.343 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.343 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.343 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:39.343 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.630924+0000", 2024-04-19T01:04:39.343 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021294306999999998 2024-04-19T01:04:39.343 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.343 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.344 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:39.344 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.631018+0000", 2024-04-19T01:04:39.344 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.3195999999999993e-05 2024-04-19T01:04:39.344 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.344 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:39.344 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.344 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.344 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.344 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:318 3.fs0 3:f23900a8:::smithi15528762-6:head [append 0~622592] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:04:39.344 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.609539+0000", 2024-04-19T01:04:39.344 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 21.664187415000001, 2024-04-19T01:04:39.344 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021603679000000001, 2024-04-19T01:04:39.344 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:39.344 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:39.345 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:39.345 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:39.345 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:39.345 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:39.345 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 318 2024-04-19T01:04:39.345 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.345 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:39.345 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.345 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:39.345 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.609539+0000", 2024-04-19T01:04:39.345 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.345 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.345 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.345 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:39.345 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.609539+0000", 2024-04-19T01:04:39.346 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.346 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.346 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.346 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:39.347 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.609541+0000", 2024-04-19T01:04:39.347 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.855e-06 2024-04-19T01:04:39.347 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.347 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.347 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:39.347 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610203+0000", 2024-04-19T01:04:39.347 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00066190200000000002 2024-04-19T01:04:39.347 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.347 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.347 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:39.347 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610215+0000", 2024-04-19T01:04:39.347 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2143e-05 2024-04-19T01:04:39.347 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.347 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.348 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:39.348 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610221+0000", 2024-04-19T01:04:39.348 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4590000000000003e-06 2024-04-19T01:04:39.348 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.348 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.348 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.348 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610283+0000", 2024-04-19T01:04:39.348 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1322999999999995e-05 2024-04-19T01:04:39.348 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.348 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.348 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:39.348 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610394+0000", 2024-04-19T01:04:39.348 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000111044 2024-04-19T01:04:39.348 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.348 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.349 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:39.349 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611327+0000", 2024-04-19T01:04:39.349 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00093369799999999995 2024-04-19T01:04:39.349 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.349 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.350 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:39.350 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.623095+0000", 2024-04-19T01:04:39.350 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011767454 2024-04-19T01:04:39.350 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.350 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.350 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:39.350 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.631101+0000", 2024-04-19T01:04:39.350 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0080059889999999998 2024-04-19T01:04:39.350 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.350 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.350 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:39.351 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.631143+0000", 2024-04-19T01:04:39.351 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1811999999999999e-05 2024-04-19T01:04:39.351 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.351 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:39.351 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.351 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.351 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.351 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 47/31 ECSubWriteReply(tid=61, last_complete=47'16, committed=1, applied=1))", 2024-04-19T01:04:39.351 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.610333+0000", 2024-04-19T01:04:39.351 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 21.663393182, 2024-04-19T01:04:39.351 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0095713929999999992, 2024-04-19T01:04:39.352 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:39.352 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:39.352 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:39.353 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:39.353 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.353 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:39.353 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610333+0000", 2024-04-19T01:04:39.353 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.353 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.353 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.353 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:39.353 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610333+0000", 2024-04-19T01:04:39.354 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.354 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.354 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.354 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:39.354 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610335+0000", 2024-04-19T01:04:39.354 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.688e-06 2024-04-19T01:04:39.354 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.354 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.354 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:39.354 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610345+0000", 2024-04-19T01:04:39.354 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0505e-05 2024-04-19T01:04:39.354 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.354 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.354 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:39.355 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610372+0000", 2024-04-19T01:04:39.355 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7053e-05 2024-04-19T01:04:39.355 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.355 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.355 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:39.355 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610391+0000", 2024-04-19T01:04:39.355 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8694000000000001e-05 2024-04-19T01:04:39.356 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.357 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.358 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.358 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.619882+0000", 2024-04-19T01:04:39.358 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0094910199999999993 2024-04-19T01:04:39.358 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.358 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.358 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:39.358 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.619905+0000", 2024-04-19T01:04:39.358 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.2433e-05 2024-04-19T01:04:39.358 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.358 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:39.358 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.358 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.358 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.359 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 47/31 ECSubWriteReply(tid=61, last_complete=47'16, committed=1, applied=1))", 2024-04-19T01:04:39.359 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.610648+0000", 2024-04-19T01:04:39.359 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 21.663078590000001, 2024-04-19T01:04:39.359 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0093177699999999995, 2024-04-19T01:04:39.359 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:39.359 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:39.359 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:39.359 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:39.359 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.359 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:39.359 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610648+0000", 2024-04-19T01:04:39.359 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.359 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.359 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.360 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:39.360 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610648+0000", 2024-04-19T01:04:39.360 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.360 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.360 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.360 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:39.360 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610649+0000", 2024-04-19T01:04:39.360 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1100000000000005e-07 2024-04-19T01:04:39.360 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.360 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.360 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:39.360 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610657+0000", 2024-04-19T01:04:39.360 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.3839999999999997e-06 2024-04-19T01:04:39.360 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.360 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.361 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:39.361 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610668+0000", 2024-04-19T01:04:39.361 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1229e-05 2024-04-19T01:04:39.361 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.361 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.361 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:39.361 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610676+0000", 2024-04-19T01:04:39.361 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7700000000000001e-06 2024-04-19T01:04:39.361 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.361 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.361 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.361 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.619945+0000", 2024-04-19T01:04:39.361 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0092685019999999996 2024-04-19T01:04:39.361 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.362 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.362 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:39.362 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.619966+0000", 2024-04-19T01:04:39.362 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1073999999999998e-05 2024-04-19T01:04:39.362 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.362 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:39.362 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.362 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.362 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.363 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:319 3.fs0 3:f23900a8:::smithi15528762-6:head [append 0~147456] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:04:39.363 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.611402+0000", 2024-04-19T01:04:39.363 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 21.662324452, 2024-04-19T01:04:39.363 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.020301099, 2024-04-19T01:04:39.363 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:39.364 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:39.364 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:39.364 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:39.365 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:39.365 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:39.365 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 319 2024-04-19T01:04:39.365 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.365 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:39.365 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.365 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:39.365 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611402+0000", 2024-04-19T01:04:39.365 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.365 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.365 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.366 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:39.366 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611402+0000", 2024-04-19T01:04:39.366 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.366 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.366 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.366 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:39.366 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611404+0000", 2024-04-19T01:04:39.366 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.52e-06 2024-04-19T01:04:39.366 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.366 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.367 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:39.367 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611632+0000", 2024-04-19T01:04:39.367 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000227526 2024-04-19T01:04:39.367 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.367 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.367 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:39.367 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611660+0000", 2024-04-19T01:04:39.367 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7841999999999999e-05 2024-04-19T01:04:39.367 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.367 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.367 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:39.367 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611678+0000", 2024-04-19T01:04:39.367 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8479e-05 2024-04-19T01:04:39.367 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.368 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.368 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.368 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.620224+0000", 2024-04-19T01:04:39.368 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0085460999999999992 2024-04-19T01:04:39.368 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.368 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.368 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:39.368 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.620334+0000", 2024-04-19T01:04:39.368 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010964 2024-04-19T01:04:39.368 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.368 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.368 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:39.368 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.620982+0000", 2024-04-19T01:04:39.368 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00064788699999999999 2024-04-19T01:04:39.368 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.369 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.369 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:39.369 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.623230+0000", 2024-04-19T01:04:39.369 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0022481950000000001 2024-04-19T01:04:39.369 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.369 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.369 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:39.369 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.631664+0000", 2024-04-19T01:04:39.369 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0084339810000000001 2024-04-19T01:04:39.369 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.369 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.369 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:39.369 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.631703+0000", 2024-04-19T01:04:39.369 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8928999999999999e-05 2024-04-19T01:04:39.370 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.370 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:39.370 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.370 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.370 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.370 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:320 3.fs0 3:f23900a8:::smithi15528762-6:head [setxattr _header (55) in=7b] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:04:39.370 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.611709+0000", 2024-04-19T01:04:39.370 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 21.662016957999999, 2024-04-19T01:04:39.370 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.020427773, 2024-04-19T01:04:39.370 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:39.370 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:39.370 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:39.370 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:39.371 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:39.371 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:39.371 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 320 2024-04-19T01:04:39.371 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.371 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:39.371 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.371 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:39.371 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611709+0000", 2024-04-19T01:04:39.371 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.371 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.371 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.371 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:39.371 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611709+0000", 2024-04-19T01:04:39.371 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.371 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.372 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.372 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:39.372 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611711+0000", 2024-04-19T01:04:39.372 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.451e-06 2024-04-19T01:04:39.372 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.372 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.372 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:39.372 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611718+0000", 2024-04-19T01:04:39.372 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5800000000000003e-06 2024-04-19T01:04:39.372 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.372 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.372 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:39.372 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611736+0000", 2024-04-19T01:04:39.372 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7339000000000001e-05 2024-04-19T01:04:39.373 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.373 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.373 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:39.373 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611749+0000", 2024-04-19T01:04:39.373 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3557000000000001e-05 2024-04-19T01:04:39.373 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.373 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.373 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.373 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.622598+0000", 2024-04-19T01:04:39.373 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010848315000000001 2024-04-19T01:04:39.373 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.373 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.373 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:39.373 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.622654+0000", 2024-04-19T01:04:39.374 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6311999999999999e-05 2024-04-19T01:04:39.374 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.374 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.374 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:39.375 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.622895+0000", 2024-04-19T01:04:39.375 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00024134300000000001 2024-04-19T01:04:39.375 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.375 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.375 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:39.375 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.623782+0000", 2024-04-19T01:04:39.375 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00088635099999999996 2024-04-19T01:04:39.375 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.375 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.375 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:39.375 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.632093+0000", 2024-04-19T01:04:39.375 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.008311466 2024-04-19T01:04:39.376 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.376 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.376 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:39.376 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.632137+0000", 2024-04-19T01:04:39.376 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4058999999999999e-05 2024-04-19T01:04:39.376 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.376 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:39.376 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.377 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.377 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.377 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:321 3.fs0 3:f23900a8:::smithi15528762-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:04:39.377 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.611769+0000", 2024-04-19T01:04:39.377 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 21.661957421, 2024-04-19T01:04:39.377 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021684730999999999, 2024-04-19T01:04:39.377 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:39.377 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:39.377 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:39.377 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:39.377 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:39.377 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:39.377 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 321 2024-04-19T01:04:39.377 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.378 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:39.378 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.378 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:39.378 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611769+0000", 2024-04-19T01:04:39.378 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.378 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.378 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.378 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:39.378 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611769+0000", 2024-04-19T01:04:39.378 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.378 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.378 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.378 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:39.378 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611770+0000", 2024-04-19T01:04:39.379 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.04e-06 2024-04-19T01:04:39.379 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.379 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.379 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:39.379 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611775+0000", 2024-04-19T01:04:39.379 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3310000000000003e-06 2024-04-19T01:04:39.379 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.379 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.379 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:39.379 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611793+0000", 2024-04-19T01:04:39.379 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7408e-05 2024-04-19T01:04:39.379 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.379 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.379 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:39.379 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611808+0000", 2024-04-19T01:04:39.380 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5259000000000001e-05 2024-04-19T01:04:39.380 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.380 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.380 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.380 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.623039+0000", 2024-04-19T01:04:39.380 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011230756 2024-04-19T01:04:39.380 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.380 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.380 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:04:39.380 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.623089+0000", 2024-04-19T01:04:39.380 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0078999999999999e-05 2024-04-19T01:04:39.380 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.380 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.380 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.380 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.632385+0000", 2024-04-19T01:04:39.381 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0092956649999999998 2024-04-19T01:04:39.381 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.381 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.381 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:39.381 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.632471+0000", 2024-04-19T01:04:39.381 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.6693999999999994e-05 2024-04-19T01:04:39.381 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.381 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.381 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:39.381 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.633454+0000", 2024-04-19T01:04:39.381 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00098249899999999996 2024-04-19T01:04:39.381 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.381 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:39.381 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.382 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.382 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.382 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:345 3.fs0 3:f23900a8:::smithi15528762-6:head [append 0~557056] snapc 9=[8,7] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:04:39.382 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.534632+0000", 2024-04-19T01:04:39.382 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 20.739094597000001, 2024-04-19T01:04:39.382 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.044819185999999997, 2024-04-19T01:04:39.382 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:39.382 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:39.382 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:39.382 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:39.382 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:39.382 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:39.382 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 345 2024-04-19T01:04:39.382 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.382 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:39.383 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.383 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:39.383 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.534632+0000", 2024-04-19T01:04:39.383 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.383 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.383 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.383 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:39.383 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.534632+0000", 2024-04-19T01:04:39.383 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.383 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.383 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.383 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:39.383 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.534634+0000", 2024-04-19T01:04:39.383 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4099999999999998e-06 2024-04-19T01:04:39.383 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.384 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.384 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:39.384 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.535690+0000", 2024-04-19T01:04:39.384 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010557260000000001 2024-04-19T01:04:39.384 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.384 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.384 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:39.384 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.535710+0000", 2024-04-19T01:04:39.384 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0339e-05 2024-04-19T01:04:39.384 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.384 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.384 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:39.385 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.535720+0000", 2024-04-19T01:04:39.385 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7599999999999997e-06 2024-04-19T01:04:39.385 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.385 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.385 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.386 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.535817+0000", 2024-04-19T01:04:39.386 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7218000000000003e-05 2024-04-19T01:04:39.386 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.386 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.386 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:39.386 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.536037+0000", 2024-04-19T01:04:39.386 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00021956699999999999 2024-04-19T01:04:39.386 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.386 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.387 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:39.387 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.537439+0000", 2024-04-19T01:04:39.387 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014022240000000001 2024-04-19T01:04:39.387 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.387 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.387 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:39.387 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.557535+0000", 2024-04-19T01:04:39.387 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.020095874999999999 2024-04-19T01:04:39.387 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.387 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.387 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:39.387 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.579388+0000", 2024-04-19T01:04:39.388 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021852772999999999 2024-04-19T01:04:39.388 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.388 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.388 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:39.388 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.579451+0000", 2024-04-19T01:04:39.388 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3293999999999994e-05 2024-04-19T01:04:39.388 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.388 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:39.388 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.388 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.388 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.388 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:346 3.fs0 3:f23900a8:::smithi15528762-6:head [append 0~671744] snapc 9=[8,7] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:04:39.388 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.540969+0000", 2024-04-19T01:04:39.388 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 20.732757069000002, 2024-04-19T01:04:39.389 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.039266644000000003, 2024-04-19T01:04:39.389 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:39.389 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:39.389 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:39.389 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:39.389 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:39.389 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:39.389 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 346 2024-04-19T01:04:39.389 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.389 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:39.389 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.389 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:39.389 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.540969+0000", 2024-04-19T01:04:39.389 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.390 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.390 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.390 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:39.390 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.540969+0000", 2024-04-19T01:04:39.390 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.390 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.390 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.390 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:39.390 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.540971+0000", 2024-04-19T01:04:39.390 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9039999999999999e-06 2024-04-19T01:04:39.390 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.391 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.391 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:39.391 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542173+0000", 2024-04-19T01:04:39.391 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00120192 2024-04-19T01:04:39.391 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.391 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.391 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:39.391 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542196+0000", 2024-04-19T01:04:39.391 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3159e-05 2024-04-19T01:04:39.391 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.391 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.391 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:39.392 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542210+0000", 2024-04-19T01:04:39.392 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3899e-05 2024-04-19T01:04:39.392 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.392 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.392 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.392 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.546462+0000", 2024-04-19T01:04:39.392 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0042514629999999996 2024-04-19T01:04:39.392 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.392 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.392 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:39.392 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.546596+0000", 2024-04-19T01:04:39.392 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013463900000000001 2024-04-19T01:04:39.393 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.393 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.393 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:39.393 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.547898+0000", 2024-04-19T01:04:39.393 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0013016239999999999 2024-04-19T01:04:39.393 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.393 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.393 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:39.393 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.558500+0000", 2024-04-19T01:04:39.393 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010601928 2024-04-19T01:04:39.393 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.393 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.393 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:39.393 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.580189+0000", 2024-04-19T01:04:39.394 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021688796 2024-04-19T01:04:39.394 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.394 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.394 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:39.394 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.580236+0000", 2024-04-19T01:04:39.394 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7311999999999997e-05 2024-04-19T01:04:39.394 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.394 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:39.394 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.394 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.394 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.394 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:347 3.fs0 3:f23900a8:::smithi15528762-6:head [append 0~98304] snapc 9=[8,7] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:04:39.394 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.542237+0000", 2024-04-19T01:04:39.394 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 20.731489512, 2024-04-19T01:04:39.395 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.038983274999999998, 2024-04-19T01:04:39.395 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:39.395 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:39.395 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:39.395 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:39.395 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:39.396 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:39.396 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 347 2024-04-19T01:04:39.396 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.396 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:39.396 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.396 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:39.396 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542237+0000", 2024-04-19T01:04:39.396 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.396 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.396 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.396 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:39.396 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542237+0000", 2024-04-19T01:04:39.396 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.396 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.397 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.397 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:39.397 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542238+0000", 2024-04-19T01:04:39.397 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3659999999999999e-06 2024-04-19T01:04:39.397 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.397 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.397 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:39.398 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542318+0000", 2024-04-19T01:04:39.398 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.0158000000000001e-05 2024-04-19T01:04:39.398 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.398 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.398 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:39.398 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542336+0000", 2024-04-19T01:04:39.398 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7836999999999999e-05 2024-04-19T01:04:39.398 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.398 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.398 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:39.398 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542348+0000", 2024-04-19T01:04:39.398 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1646e-05 2024-04-19T01:04:39.398 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.398 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.399 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.399 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.553869+0000", 2024-04-19T01:04:39.399 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011521164 2024-04-19T01:04:39.399 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.399 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.399 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:39.399 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.554020+0000", 2024-04-19T01:04:39.399 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015064200000000001 2024-04-19T01:04:39.399 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.399 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.399 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:39.400 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.554913+0000", 2024-04-19T01:04:39.400 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00089314099999999997 2024-04-19T01:04:39.400 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.400 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.400 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:39.400 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.558556+0000", 2024-04-19T01:04:39.400 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003642803 2024-04-19T01:04:39.400 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.400 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.400 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:39.400 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.581179+0000", 2024-04-19T01:04:39.400 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.022623306999999999 2024-04-19T01:04:39.400 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.400 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.401 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:39.401 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.581220+0000", 2024-04-19T01:04:39.401 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1211000000000002e-05 2024-04-19T01:04:39.401 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.401 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:39.401 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.401 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.401 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.401 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:348 3.fs0 3:f23900a8:::smithi15528762-6:head [setxattr _header (55) in=7b] snapc 9=[8,7] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:04:39.401 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.542369+0000", 2024-04-19T01:04:39.401 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 20.731357652, 2024-04-19T01:04:39.401 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.039646576000000003, 2024-04-19T01:04:39.401 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:39.401 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:39.402 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:39.402 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:39.402 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:39.402 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:39.402 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 348 2024-04-19T01:04:39.402 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.402 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:39.402 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.402 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:39.402 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542369+0000", 2024-04-19T01:04:39.402 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.402 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.402 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.402 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:39.402 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542369+0000", 2024-04-19T01:04:39.403 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.403 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.403 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.403 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:39.403 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542370+0000", 2024-04-19T01:04:39.403 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.384e-06 2024-04-19T01:04:39.404 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.404 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.404 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:39.404 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542376+0000", 2024-04-19T01:04:39.404 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.806e-06 2024-04-19T01:04:39.404 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.404 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.404 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:39.404 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542400+0000", 2024-04-19T01:04:39.404 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4218000000000001e-05 2024-04-19T01:04:39.404 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.405 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.405 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:39.405 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542411+0000", 2024-04-19T01:04:39.405 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0801999999999999e-05 2024-04-19T01:04:39.405 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.405 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.405 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.405 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.557611+0000", 2024-04-19T01:04:39.405 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015199724 2024-04-19T01:04:39.405 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.405 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.406 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:39.406 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.557731+0000", 2024-04-19T01:04:39.406 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012045900000000001 2024-04-19T01:04:39.406 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.406 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.406 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:39.406 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.558229+0000", 2024-04-19T01:04:39.406 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00049771099999999997 2024-04-19T01:04:39.406 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.406 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.407 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:39.407 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.559080+0000", 2024-04-19T01:04:39.407 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00085117400000000001 2024-04-19T01:04:39.407 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.407 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.407 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:39.407 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.581912+0000", 2024-04-19T01:04:39.407 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.022832086000000001 2024-04-19T01:04:39.408 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.408 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.408 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:39.408 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.582015+0000", 2024-04-19T01:04:39.408 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000103212 2024-04-19T01:04:39.408 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.408 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:39.408 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.408 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.408 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.408 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:349 3.fs0 3:f23900a8:::smithi15528762-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:04:39.408 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.542427+0000", 2024-04-19T01:04:39.408 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 20.731299481000001, 2024-04-19T01:04:39.408 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.041128310000000001, 2024-04-19T01:04:39.409 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:39.409 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:39.409 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:39.409 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:39.409 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:39.409 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:39.409 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 349 2024-04-19T01:04:39.409 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.409 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:39.409 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.409 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:39.410 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542427+0000", 2024-04-19T01:04:39.410 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.410 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.410 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.410 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:39.410 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542427+0000", 2024-04-19T01:04:39.410 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.410 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.410 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.410 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:39.410 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542428+0000", 2024-04-19T01:04:39.410 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.4e-07 2024-04-19T01:04:39.410 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.410 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.411 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:39.411 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542432+0000", 2024-04-19T01:04:39.411 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5499999999999996e-06 2024-04-19T01:04:39.411 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.411 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.411 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:39.411 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542446+0000", 2024-04-19T01:04:39.411 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3365e-05 2024-04-19T01:04:39.411 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.411 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.411 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:39.411 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542456+0000", 2024-04-19T01:04:39.411 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.029e-05 2024-04-19T01:04:39.411 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.412 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.412 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.412 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.558587+0000", 2024-04-19T01:04:39.412 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.016131106999999999 2024-04-19T01:04:39.412 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.412 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.412 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:04:39.412 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.558676+0000", 2024-04-19T01:04:39.412 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.8823e-05 2024-04-19T01:04:39.412 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.412 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.412 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.412 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.582441+0000", 2024-04-19T01:04:39.412 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.023764864 2024-04-19T01:04:39.412 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.413 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.413 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:39.413 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.582493+0000", 2024-04-19T01:04:39.413 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2130999999999999e-05 2024-04-19T01:04:39.413 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.413 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.413 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:39.413 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.583555+0000", 2024-04-19T01:04:39.413 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010623399999999999 2024-04-19T01:04:39.413 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.413 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:39.413 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.413 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.413 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:39.413 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:39.418 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-19T01:04:39.419 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:39.414+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:39.515 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.2 to 1 (10000) 2024-04-19T01:04:39.519 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:04:39.518+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:04:39.522 INFO:tasks.rados.rados.0.smithi155.stdout:206: done (2 left) 2024-04-19T01:04:39.522 INFO:tasks.rados.rados.0.smithi155.stdout:207: done (1 left) 2024-04-19T01:04:39.522 INFO:tasks.rados.rados.0.smithi155.stdout:208: done (0 left) 2024-04-19T01:04:39.522 INFO:tasks.rados.rados.0.smithi155.stdout:209: rmattr oid 26 current snap is 20 2024-04-19T01:04:39.522 INFO:tasks.rados.rados.0.smithi155.stdout:209: done (0 left) 2024-04-19T01:04:39.522 INFO:tasks.rados.rados.0.smithi155.stdout:210: rmattr oid 47 current snap is 20 2024-04-19T01:04:39.522 INFO:tasks.rados.rados.0.smithi155.stdout:210: done (0 left) 2024-04-19T01:04:39.523 INFO:tasks.rados.rados.0.smithi155.stdout:211: rmattr oid 22 current snap is 20 2024-04-19T01:04:39.523 INFO:tasks.rados.rados.0.smithi155.stdout:211: done (0 left) 2024-04-19T01:04:39.523 INFO:tasks.rados.rados.0.smithi155.stdout:212: snap_remove snap 15 2024-04-19T01:04:39.541 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-19T01:04:39.554 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:39.555 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:04:39.555 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2024-04-19T01:04:39.555 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:39.569 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-19T01:04:39.620 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:39.620+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:39.721 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:04:39.720+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:04:39.735 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:39.735 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:04:39.735 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2024-04-19T01:04:39.735 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2024-04-19T01:04:39.735 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:39.746 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-19T01:04:39.823 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:39.824+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:39.864 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:39.865 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2024-04-19T01:04:39.865 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2024-04-19T01:04:39.865 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-04-19T01:04:39.865 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.865 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.9s4 43/31 ECSubWrite(tid=70, reqid=client.4604.0:235, at_version=43'43, trim_to=0'0, roll_forward_to=41'37))", 2024-04-19T01:04:39.865 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.869289+0000", 2024-04-19T01:04:39.865 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 25.995042194, 2024-04-19T01:04:39.865 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0068565980000000002, 2024-04-19T01:04:39.865 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:39.865 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:39.865 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:39.865 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:39.866 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.866 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:39.866 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.869289+0000", 2024-04-19T01:04:39.866 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.866 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.866 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.866 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:39.866 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.869289+0000", 2024-04-19T01:04:39.866 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.866 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.866 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.866 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:39.867 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.869290+0000", 2024-04-19T01:04:39.867 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2419999999999999e-06 2024-04-19T01:04:39.867 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.867 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.867 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:39.867 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.869823+0000", 2024-04-19T01:04:39.867 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00053351500000000003 2024-04-19T01:04:39.867 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.867 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.867 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:39.867 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.869851+0000", 2024-04-19T01:04:39.867 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7634999999999999e-05 2024-04-19T01:04:39.867 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.868 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.868 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:39.868 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.869862+0000", 2024-04-19T01:04:39.868 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0677e-05 2024-04-19T01:04:39.868 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.868 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.868 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.868 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.870063+0000", 2024-04-19T01:04:39.868 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00020101900000000001 2024-04-19T01:04:39.868 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.868 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.868 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:39.868 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.870075+0000", 2024-04-19T01:04:39.868 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1855000000000001e-05 2024-04-19T01:04:39.869 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.869 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.869 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:39.869 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.876107+0000", 2024-04-19T01:04:39.869 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0060328990000000004 2024-04-19T01:04:39.869 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.870 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.871 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:39.871 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.876145+0000", 2024-04-19T01:04:39.871 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7756000000000002e-05 2024-04-19T01:04:39.871 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.871 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:39.871 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.871 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.871 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.871 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds1 43/31 ECSubWrite(tid=38, reqid=client.4604.0:253, at_version=43'14, trim_to=0'0, roll_forward_to=43'12))", 2024-04-19T01:04:39.871 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.986285+0000", 2024-04-19T01:04:39.871 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 25.87804534, 2024-04-19T01:04:39.871 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0090548699999999996, 2024-04-19T01:04:39.871 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:39.871 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:39.871 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:39.872 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:39.872 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.872 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:39.878 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.986285+0000", 2024-04-19T01:04:39.878 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.878 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.878 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.878 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:39.878 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.986285+0000", 2024-04-19T01:04:39.878 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.879 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.879 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.879 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:39.879 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.986287+0000", 2024-04-19T01:04:39.879 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.421e-06 2024-04-19T01:04:39.879 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.883 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.883 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:39.883 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.986389+0000", 2024-04-19T01:04:39.883 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010215100000000001 2024-04-19T01:04:39.883 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.883 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.883 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:39.883 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.986414+0000", 2024-04-19T01:04:39.883 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4447000000000002e-05 2024-04-19T01:04:39.884 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.884 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.884 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:39.884 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.986435+0000", 2024-04-19T01:04:39.884 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1069000000000001e-05 2024-04-19T01:04:39.884 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.884 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.884 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.884 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.986566+0000", 2024-04-19T01:04:39.884 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013169899999999999 2024-04-19T01:04:39.884 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.884 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.884 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:39.884 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.986604+0000", 2024-04-19T01:04:39.884 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7857e-05 2024-04-19T01:04:39.885 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.885 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.885 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:39.885 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.995257+0000", 2024-04-19T01:04:39.885 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0086525209999999998 2024-04-19T01:04:39.885 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.885 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.885 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:39.885 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.995340+0000", 2024-04-19T01:04:39.885 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.3704999999999999e-05 2024-04-19T01:04:39.885 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.885 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:39.885 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.885 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.886 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.886 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds1 43/31 ECSubWrite(tid=39, reqid=client.4604.0:254, at_version=43'15, trim_to=0'0, roll_forward_to=43'12))", 2024-04-19T01:04:39.886 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.996134+0000", 2024-04-19T01:04:39.886 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 25.868196888, 2024-04-19T01:04:39.886 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0070946079999999996, 2024-04-19T01:04:39.886 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:39.886 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:39.886 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:39.886 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:39.886 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.886 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:39.886 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.996134+0000", 2024-04-19T01:04:39.886 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.886 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.887 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.887 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:39.887 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.996134+0000", 2024-04-19T01:04:39.887 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.887 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.887 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.887 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:39.887 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.996135+0000", 2024-04-19T01:04:39.887 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2839999999999999e-06 2024-04-19T01:04:39.887 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.887 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.887 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:39.887 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.996186+0000", 2024-04-19T01:04:39.887 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0572999999999998e-05 2024-04-19T01:04:39.887 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.888 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.888 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:39.888 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.996212+0000", 2024-04-19T01:04:39.888 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6658000000000001e-05 2024-04-19T01:04:39.888 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.888 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.888 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:39.888 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.996220+0000", 2024-04-19T01:04:39.888 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5939999999999996e-06 2024-04-19T01:04:39.888 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.888 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.888 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.888 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.996318+0000", 2024-04-19T01:04:39.888 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7595000000000003e-05 2024-04-19T01:04:39.888 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.889 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.889 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:39.889 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.996326+0000", 2024-04-19T01:04:39.889 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.8650000000000001e-06 2024-04-19T01:04:39.889 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.889 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.889 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:39.889 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.003178+0000", 2024-04-19T01:04:39.889 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0068528449999999998 2024-04-19T01:04:39.889 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.889 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.899 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:39.900 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.003229+0000", 2024-04-19T01:04:39.900 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0194000000000001e-05 2024-04-19T01:04:39.900 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.900 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:39.900 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.900 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.900 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.900 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.9s4 44/31 ECSubWrite(tid=82, reqid=client.4604.0:280, at_version=44'50, trim_to=0'0, roll_forward_to=43'48))", 2024-04-19T01:04:39.901 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:14.880621+0000", 2024-04-19T01:04:39.901 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 24.983710132999999, 2024-04-19T01:04:39.901 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.009454835, 2024-04-19T01:04:39.901 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:39.901 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:39.901 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:39.901 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:39.901 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.901 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:39.901 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.880621+0000", 2024-04-19T01:04:39.901 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.901 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.902 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.902 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:39.902 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.880621+0000", 2024-04-19T01:04:39.902 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.902 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.902 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.902 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:39.903 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.880622+0000", 2024-04-19T01:04:39.903 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.6200000000000006e-07 2024-04-19T01:04:39.903 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.903 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.903 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:39.903 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.881098+0000", 2024-04-19T01:04:39.903 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00047665400000000002 2024-04-19T01:04:39.903 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.903 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.903 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:39.903 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.881115+0000", 2024-04-19T01:04:39.903 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6612999999999998e-05 2024-04-19T01:04:39.903 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.903 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.904 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:39.904 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.881124+0000", 2024-04-19T01:04:39.904 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.3959999999999995e-06 2024-04-19T01:04:39.904 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.904 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.904 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.904 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.881176+0000", 2024-04-19T01:04:39.904 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.1965000000000001e-05 2024-04-19T01:04:39.904 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.905 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.905 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:39.905 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.881182+0000", 2024-04-19T01:04:39.905 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.806e-06 2024-04-19T01:04:39.905 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.905 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.905 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:39.905 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.890038+0000", 2024-04-19T01:04:39.905 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0088557170000000008 2024-04-19T01:04:39.905 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.905 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.905 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:39.905 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.890076+0000", 2024-04-19T01:04:39.905 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7722000000000002e-05 2024-04-19T01:04:39.905 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.906 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:39.906 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.906 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.906 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.906 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.0s2 44/31 ECSubWrite(tid=25, reqid=client.4604.0:284, at_version=44'22, trim_to=0'0, roll_forward_to=44'20))", 2024-04-19T01:04:39.906 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:14.920303+0000", 2024-04-19T01:04:39.906 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 24.944027859999998, 2024-04-19T01:04:39.906 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011517777999999999, 2024-04-19T01:04:39.906 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:39.906 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:39.906 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:39.906 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:39.906 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.906 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:39.907 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.920303+0000", 2024-04-19T01:04:39.907 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.907 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.907 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.907 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:39.907 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.920303+0000", 2024-04-19T01:04:39.907 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.907 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.907 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.907 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:39.907 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.920304+0000", 2024-04-19T01:04:39.907 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2900000000000003e-07 2024-04-19T01:04:39.907 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.907 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.908 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:39.908 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.921382+0000", 2024-04-19T01:04:39.908 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010785510000000001 2024-04-19T01:04:39.908 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.908 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.908 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:39.908 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.921394+0000", 2024-04-19T01:04:39.908 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2183e-05 2024-04-19T01:04:39.908 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.908 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.908 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:39.908 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.921403+0000", 2024-04-19T01:04:39.908 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9330000000000007e-06 2024-04-19T01:04:39.908 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.908 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.909 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.909 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.921468+0000", 2024-04-19T01:04:39.909 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4191999999999998e-05 2024-04-19T01:04:39.909 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.909 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.909 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:39.909 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.921479+0000", 2024-04-19T01:04:39.909 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1763000000000001e-05 2024-04-19T01:04:39.909 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.909 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.909 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:39.909 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.931782+0000", 2024-04-19T01:04:39.909 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010302891999999999 2024-04-19T01:04:39.909 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.909 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.910 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:39.910 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.931821+0000", 2024-04-19T01:04:39.910 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8535000000000003e-05 2024-04-19T01:04:39.910 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.910 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:39.910 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.910 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.910 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.910 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds1 47/31 ECSubWrite(tid=61, reqid=client.4604.0:330, at_version=47'32, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:04:39.910 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.676507+0000", 2024-04-19T01:04:39.910 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 22.18782375, 2024-04-19T01:04:39.910 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010458914, 2024-04-19T01:04:39.910 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:39.910 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:39.911 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:39.911 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:39.911 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.911 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:39.911 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.676507+0000", 2024-04-19T01:04:39.911 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.911 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.911 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.911 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:39.911 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.676507+0000", 2024-04-19T01:04:39.911 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.911 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.911 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.911 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:39.911 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.676508+0000", 2024-04-19T01:04:39.912 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2399999999999996e-07 2024-04-19T01:04:39.912 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.912 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.912 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:39.912 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.676672+0000", 2024-04-19T01:04:39.912 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000164258 2024-04-19T01:04:39.912 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.912 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.912 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:39.912 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.676688+0000", 2024-04-19T01:04:39.912 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5443000000000001e-05 2024-04-19T01:04:39.912 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.912 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.912 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:39.912 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.676696+0000", 2024-04-19T01:04:39.913 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1550000000000007e-06 2024-04-19T01:04:39.921 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.921 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.921 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.921 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.676799+0000", 2024-04-19T01:04:39.921 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000102786 2024-04-19T01:04:39.921 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.921 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.921 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:39.921 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.676814+0000", 2024-04-19T01:04:39.921 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4951e-05 2024-04-19T01:04:39.921 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.922 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.922 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:39.922 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.686911+0000", 2024-04-19T01:04:39.922 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010097415 2024-04-19T01:04:39.922 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.922 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.922 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:39.922 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.686966+0000", 2024-04-19T01:04:39.923 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4982000000000003e-05 2024-04-19T01:04:39.923 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.923 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:39.924 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.924 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.924 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.924 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds1 48/31 ECSubWrite(tid=67, reqid=osd.12.0:67, at_version=48'37, trim_to=0'0, roll_forward_to=47'35))", 2024-04-19T01:04:39.924 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.511815+0000", 2024-04-19T01:04:39.924 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 21.352515377, 2024-04-19T01:04:39.924 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0069408539999999998, 2024-04-19T01:04:39.925 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:39.925 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:39.925 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:39.925 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:39.925 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.925 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:39.925 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.511815+0000", 2024-04-19T01:04:39.925 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.925 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.925 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.925 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:39.925 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.511815+0000", 2024-04-19T01:04:39.925 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.926 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.926 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.926 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:39.926 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.511816+0000", 2024-04-19T01:04:39.926 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6000000000000004e-07 2024-04-19T01:04:39.926 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.926 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.926 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:39.926 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.511877+0000", 2024-04-19T01:04:39.926 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0924000000000001e-05 2024-04-19T01:04:39.926 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.926 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.926 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:39.926 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.511908+0000", 2024-04-19T01:04:39.926 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0635000000000001e-05 2024-04-19T01:04:39.927 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.927 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.927 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:39.927 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.511931+0000", 2024-04-19T01:04:39.927 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3462e-05 2024-04-19T01:04:39.927 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.927 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.928 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.928 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.512031+0000", 2024-04-19T01:04:39.928 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000100202 2024-04-19T01:04:39.928 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.928 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.928 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:39.928 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.512049+0000", 2024-04-19T01:04:39.928 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7623000000000001e-05 2024-04-19T01:04:39.928 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.928 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.928 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:39.928 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.518712+0000", 2024-04-19T01:04:39.928 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0066631950000000002 2024-04-19T01:04:39.928 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.929 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.929 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:39.929 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.518756+0000", 2024-04-19T01:04:39.929 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4252999999999998e-05 2024-04-19T01:04:39.929 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.929 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:39.929 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.929 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.929 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.929 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds1 48/31 ECSubWrite(tid=68, reqid=osd.12.0:68, at_version=48'39, trim_to=0'0, roll_forward_to=47'35))", 2024-04-19T01:04:39.929 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.513367+0000", 2024-04-19T01:04:39.929 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 21.350963589999999, 2024-04-19T01:04:39.929 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0073005730000000003, 2024-04-19T01:04:39.929 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:39.930 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:39.930 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:39.930 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:39.930 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.930 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:39.930 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.513367+0000", 2024-04-19T01:04:39.930 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.930 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.930 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.930 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:39.930 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.513367+0000", 2024-04-19T01:04:39.930 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.930 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.930 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.930 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:39.931 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.513369+0000", 2024-04-19T01:04:39.931 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6670000000000001e-06 2024-04-19T01:04:39.931 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.931 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.931 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:39.931 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.513417+0000", 2024-04-19T01:04:39.931 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8534e-05 2024-04-19T01:04:39.931 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.931 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.931 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:39.931 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.513461+0000", 2024-04-19T01:04:39.931 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3151000000000001e-05 2024-04-19T01:04:39.931 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.931 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.932 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:39.932 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.513484+0000", 2024-04-19T01:04:39.932 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3088999999999999e-05 2024-04-19T01:04:39.932 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.932 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.932 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.932 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.514630+0000", 2024-04-19T01:04:39.932 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001145868 2024-04-19T01:04:39.932 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.932 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.932 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:39.932 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.514638+0000", 2024-04-19T01:04:39.932 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.8769999999999999e-06 2024-04-19T01:04:39.932 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.932 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.933 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:39.933 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.520622+0000", 2024-04-19T01:04:39.933 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0059831240000000003 2024-04-19T01:04:39.933 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.933 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.933 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:39.933 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.520668+0000", 2024-04-19T01:04:39.933 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6263000000000001e-05 2024-04-19T01:04:39.933 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.933 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:39.933 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.933 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.933 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.933 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.3s5 48/31 ECSubWrite(tid=46, reqid=client.4604.0:343, at_version=48'36, trim_to=0'0, roll_forward_to=48'34))", 2024-04-19T01:04:39.933 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.541349+0000", 2024-04-19T01:04:39.934 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 21.322981338999998, 2024-04-19T01:04:39.934 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010550992, 2024-04-19T01:04:39.934 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:39.934 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:39.934 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:39.934 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:39.934 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.934 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:39.934 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.541349+0000", 2024-04-19T01:04:39.934 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.934 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.934 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.934 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:39.934 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.541349+0000", 2024-04-19T01:04:39.935 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.935 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.935 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.935 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:39.935 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.541351+0000", 2024-04-19T01:04:39.935 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.248e-06 2024-04-19T01:04:39.935 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.935 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.935 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:39.935 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.541675+0000", 2024-04-19T01:04:39.935 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00032395500000000001 2024-04-19T01:04:39.935 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.935 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.935 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:39.936 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.541707+0000", 2024-04-19T01:04:39.936 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2042000000000003e-05 2024-04-19T01:04:39.936 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.936 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.936 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:39.936 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.541720+0000", 2024-04-19T01:04:39.936 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3101999999999999e-05 2024-04-19T01:04:39.936 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.936 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.936 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.936 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.541873+0000", 2024-04-19T01:04:39.936 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015282199999999999 2024-04-19T01:04:39.936 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.936 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.936 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:39.936 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.541899+0000", 2024-04-19T01:04:39.937 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6381999999999999e-05 2024-04-19T01:04:39.937 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.937 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.937 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:39.937 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.551828+0000", 2024-04-19T01:04:39.937 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0099288589999999999 2024-04-19T01:04:39.937 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.937 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.937 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:39.937 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.551900+0000", 2024-04-19T01:04:39.937 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2582000000000004e-05 2024-04-19T01:04:39.937 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.937 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:39.937 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.937 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.938 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.938 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.as1 48/31 ECSubWrite(tid=71, reqid=client.4604.0:354, at_version=48'10, trim_to=0'0, roll_forward_to=34'8))", 2024-04-19T01:04:39.938 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.556705+0000", 2024-04-19T01:04:39.938 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 21.307625564999999, 2024-04-19T01:04:39.939 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013008369000000001, 2024-04-19T01:04:39.939 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:39.939 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:39.939 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:39.939 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:39.939 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.939 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:39.939 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.556705+0000", 2024-04-19T01:04:39.939 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.939 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.939 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.939 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:39.939 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.556705+0000", 2024-04-19T01:04:39.939 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.939 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.940 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.940 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:39.940 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.556706+0000", 2024-04-19T01:04:39.940 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2800000000000005e-07 2024-04-19T01:04:39.940 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.940 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.940 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:39.940 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.557214+0000", 2024-04-19T01:04:39.940 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00050754200000000004 2024-04-19T01:04:39.940 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.941 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.941 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:39.941 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.557239+0000", 2024-04-19T01:04:39.941 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.567e-05 2024-04-19T01:04:39.941 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.941 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.941 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:39.941 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.557268+0000", 2024-04-19T01:04:39.941 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8496e-05 2024-04-19T01:04:39.941 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.941 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.941 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.941 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.557438+0000", 2024-04-19T01:04:39.941 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000169963 2024-04-19T01:04:39.942 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.942 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.942 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:39.942 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.557491+0000", 2024-04-19T01:04:39.942 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2689999999999999e-05 2024-04-19T01:04:39.942 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.942 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.942 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:39.942 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.569674+0000", 2024-04-19T01:04:39.942 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012183727 2024-04-19T01:04:39.942 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.942 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.942 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:39.942 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.569714+0000", 2024-04-19T01:04:39.942 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9353000000000001e-05 2024-04-19T01:04:39.943 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.943 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:39.943 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.943 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.943 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.943 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.3s5 49/31 ECSubWrite(tid=48, reqid=osd.15.0:48, at_version=49'38, trim_to=0'0, roll_forward_to=48'36))", 2024-04-19T01:04:39.943 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:19.533518+0000", 2024-04-19T01:04:39.943 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 20.33081258, 2024-04-19T01:04:39.943 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011056513, 2024-04-19T01:04:39.943 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:39.943 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:39.943 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:39.943 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:39.943 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.944 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:39.944 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.533518+0000", 2024-04-19T01:04:39.944 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.944 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.944 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.944 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:39.944 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.533518+0000", 2024-04-19T01:04:39.944 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.944 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.944 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.944 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:39.944 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.533519+0000", 2024-04-19T01:04:39.944 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4000000000000002e-07 2024-04-19T01:04:39.944 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.944 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.945 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:39.945 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.533543+0000", 2024-04-19T01:04:39.945 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4494999999999998e-05 2024-04-19T01:04:39.945 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.945 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.945 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:39.945 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.533556+0000", 2024-04-19T01:04:39.945 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3145e-05 2024-04-19T01:04:39.945 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.945 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.945 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:39.945 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.533565+0000", 2024-04-19T01:04:39.945 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.4330000000000004e-06 2024-04-19T01:04:39.945 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.945 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.946 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.946 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.533697+0000", 2024-04-19T01:04:39.946 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013253399999999999 2024-04-19T01:04:39.946 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.946 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.946 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:39.946 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.533714+0000", 2024-04-19T01:04:39.946 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7163000000000002e-05 2024-04-19T01:04:39.946 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.946 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.946 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:39.946 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.544518+0000", 2024-04-19T01:04:39.946 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010803063999999999 2024-04-19T01:04:39.946 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.946 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.947 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:39.947 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.544575+0000", 2024-04-19T01:04:39.947 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.7238999999999999e-05 2024-04-19T01:04:39.947 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.947 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:39.947 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.947 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.947 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.947 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.3s5 49/31 ECSubWrite(tid=49, reqid=osd.15.0:49, at_version=49'40, trim_to=0'0, roll_forward_to=48'36))", 2024-04-19T01:04:39.947 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:19.535950+0000", 2024-04-19T01:04:39.947 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 20.328380678999999, 2024-04-19T01:04:39.947 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0096398579999999994, 2024-04-19T01:04:39.947 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:39.948 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:39.948 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:39.948 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:39.948 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.948 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:39.948 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.535950+0000", 2024-04-19T01:04:39.948 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.948 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.949 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.949 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:39.949 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.535950+0000", 2024-04-19T01:04:39.949 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.949 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.949 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.949 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:39.949 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.535952+0000", 2024-04-19T01:04:39.956 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4249999999999999e-06 2024-04-19T01:04:39.956 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.956 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.956 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:39.956 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.535984+0000", 2024-04-19T01:04:39.956 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2076000000000003e-05 2024-04-19T01:04:39.956 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.957 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.957 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:39.957 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.536002+0000", 2024-04-19T01:04:39.957 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8252000000000001e-05 2024-04-19T01:04:39.958 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.958 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.959 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:39.959 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.536015+0000", 2024-04-19T01:04:39.959 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3358e-05 2024-04-19T01:04:39.959 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.959 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.959 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.959 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.536116+0000", 2024-04-19T01:04:39.959 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000101105 2024-04-19T01:04:39.959 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.960 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.960 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:39.960 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.536132+0000", 2024-04-19T01:04:39.960 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5495999999999999e-05 2024-04-19T01:04:39.960 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.960 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.960 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:39.960 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.545543+0000", 2024-04-19T01:04:39.961 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0094109139999999994 2024-04-19T01:04:39.961 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.961 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.961 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:39.961 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.545590+0000", 2024-04-19T01:04:39.962 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7231999999999998e-05 2024-04-19T01:04:39.962 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.962 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:39.962 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.962 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.962 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.962 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.es3 49/31 ECSubWrite(tid=125, reqid=osd.13.0:125, at_version=49'28, trim_to=0'0, roll_forward_to=48'26))", 2024-04-19T01:04:39.962 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:19.538199+0000", 2024-04-19T01:04:39.962 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 20.326131384, 2024-04-19T01:04:39.962 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0081672089999999999, 2024-04-19T01:04:39.963 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:39.963 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:39.963 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:39.963 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:39.963 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.963 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:39.963 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538199+0000", 2024-04-19T01:04:39.964 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.965 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.965 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.965 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:39.965 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538199+0000", 2024-04-19T01:04:39.965 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.965 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.965 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.965 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:39.965 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538200+0000", 2024-04-19T01:04:39.965 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8799999999999998e-07 2024-04-19T01:04:39.965 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.966 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.966 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:39.966 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538236+0000", 2024-04-19T01:04:39.967 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.6575999999999997e-05 2024-04-19T01:04:39.971 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.971 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.972 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:39.972 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538257+0000", 2024-04-19T01:04:39.972 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0874999999999999e-05 2024-04-19T01:04:39.972 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.973 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.973 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:39.973 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538275+0000", 2024-04-19T01:04:39.973 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7745999999999999e-05 2024-04-19T01:04:39.973 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.973 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.973 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.973 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538356+0000", 2024-04-19T01:04:39.973 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1361999999999998e-05 2024-04-19T01:04:39.973 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.973 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.973 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:39.973 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538366+0000", 2024-04-19T01:04:39.973 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.9809999999999998e-06 2024-04-19T01:04:39.974 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.974 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.974 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:39.974 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.546323+0000", 2024-04-19T01:04:39.974 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0079562420000000005 2024-04-19T01:04:39.974 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.974 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.974 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:39.974 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.546367+0000", 2024-04-19T01:04:39.974 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4039000000000003e-05 2024-04-19T01:04:39.974 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.974 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:39.974 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.975 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.975 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.975 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.9s4 49/31 ECSubWrite(tid=124, reqid=osd.13.0:124, at_version=49'75, trim_to=0'0, roll_forward_to=48'73))", 2024-04-19T01:04:39.975 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:19.538317+0000", 2024-04-19T01:04:39.975 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 20.326014118, 2024-04-19T01:04:39.975 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0082727589999999993, 2024-04-19T01:04:39.975 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:39.975 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:39.975 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:39.975 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:39.975 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.975 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:39.975 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538317+0000", 2024-04-19T01:04:39.976 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.976 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.976 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.976 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:39.976 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538317+0000", 2024-04-19T01:04:39.976 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.976 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.976 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.976 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:39.976 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538317+0000", 2024-04-19T01:04:39.976 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1200000000000003e-07 2024-04-19T01:04:39.976 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.976 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.977 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:39.978 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538344+0000", 2024-04-19T01:04:39.978 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6293000000000001e-05 2024-04-19T01:04:39.978 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.979 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.979 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:39.979 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538360+0000", 2024-04-19T01:04:39.979 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6761000000000001e-05 2024-04-19T01:04:39.979 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.979 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.980 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:39.980 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538374+0000", 2024-04-19T01:04:39.980 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4017e-05 2024-04-19T01:04:39.980 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.980 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.980 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.981 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538500+0000", 2024-04-19T01:04:39.981 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000125587 2024-04-19T01:04:39.981 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.981 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.981 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:39.981 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538510+0000", 2024-04-19T01:04:39.981 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.046e-05 2024-04-19T01:04:39.981 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.981 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.981 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:39.981 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.546545+0000", 2024-04-19T01:04:39.981 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0080348729999999997 2024-04-19T01:04:39.981 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.982 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.982 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:39.982 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.546589+0000", 2024-04-19T01:04:39.982 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4156000000000002e-05 2024-04-19T01:04:39.982 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.982 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:39.982 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.982 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.982 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.982 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.0s2 49/31 ECSubWrite(tid=39, reqid=osd.1.0:39, at_version=49'35, trim_to=0'0, roll_forward_to=48'33))", 2024-04-19T01:04:39.983 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:19.539477+0000", 2024-04-19T01:04:39.983 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 20.324853632, 2024-04-19T01:04:39.983 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0068473759999999996, 2024-04-19T01:04:39.983 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:39.983 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:39.983 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:39.983 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:39.983 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.983 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:39.983 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.539477+0000", 2024-04-19T01:04:39.983 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.983 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.983 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.983 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:39.984 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.539477+0000", 2024-04-19T01:04:39.984 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.984 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.984 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.984 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:39.984 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.539478+0000", 2024-04-19T01:04:39.984 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9099999999999999e-07 2024-04-19T01:04:39.984 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.984 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.984 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:39.984 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.539511+0000", 2024-04-19T01:04:39.984 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.3732000000000002e-05 2024-04-19T01:04:39.984 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.985 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.985 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:39.985 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.539530+0000", 2024-04-19T01:04:39.985 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8765000000000001e-05 2024-04-19T01:04:39.985 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.985 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.985 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:39.985 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.539566+0000", 2024-04-19T01:04:39.985 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5482999999999999e-05 2024-04-19T01:04:39.985 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.985 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.985 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:39.985 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.539713+0000", 2024-04-19T01:04:39.986 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000147193 2024-04-19T01:04:39.986 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.986 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.986 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:39.986 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.539729+0000", 2024-04-19T01:04:39.986 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.666e-05 2024-04-19T01:04:39.986 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.986 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.986 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:39.986 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.546279+0000", 2024-04-19T01:04:39.986 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0065497539999999996 2024-04-19T01:04:39.987 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.987 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.987 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:39.987 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.546325+0000", 2024-04-19T01:04:39.987 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5398000000000001e-05 2024-04-19T01:04:39.987 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.987 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:39.987 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:39.987 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.987 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.988 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.9s4 49/31 ECSubWrite(tid=126, reqid=osd.13.0:126, at_version=49'77, trim_to=0'0, roll_forward_to=48'73))", 2024-04-19T01:04:39.988 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:19.539763+0000", 2024-04-19T01:04:39.988 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 20.32456827, 2024-04-19T01:04:39.988 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0076408589999999998, 2024-04-19T01:04:39.988 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:39.988 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:39.988 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:39.988 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:39.988 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:39.988 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:39.988 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.539763+0000", 2024-04-19T01:04:39.988 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:39.989 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:39.989 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.002 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:40.002 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.539763+0000", 2024-04-19T01:04:40.002 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.002 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.002 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.002 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:40.002 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.539764+0000", 2024-04-19T01:04:40.002 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.3900000000000003e-07 2024-04-19T01:04:40.002 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.002 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.002 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:40.003 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.539791+0000", 2024-04-19T01:04:40.003 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7328999999999999e-05 2024-04-19T01:04:40.003 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.003 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.003 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:40.003 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.539808+0000", 2024-04-19T01:04:40.003 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6960999999999999e-05 2024-04-19T01:04:40.003 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.003 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.003 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:40.003 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.539817+0000", 2024-04-19T01:04:40.003 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.1789999999999997e-06 2024-04-19T01:04:40.003 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.003 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.004 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:40.004 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.544922+0000", 2024-04-19T01:04:40.004 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0051049770000000001 2024-04-19T01:04:40.004 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.004 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.004 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:40.004 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.544936+0000", 2024-04-19T01:04:40.004 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3712e-05 2024-04-19T01:04:40.004 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.004 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.004 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:40.004 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.547361+0000", 2024-04-19T01:04:40.004 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0024257039999999999 2024-04-19T01:04:40.005 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.005 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.005 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:40.005 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.547403+0000", 2024-04-19T01:04:40.005 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2058000000000003e-05 2024-04-19T01:04:40.005 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.005 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:40.005 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.005 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.005 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.005 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.9s4 63/31 ECSubWrite(tid=152, reqid=client.4604.0:465, at_version=63'100, trim_to=0'0, roll_forward_to=62'98))", 2024-04-19T01:04:40.005 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:33.565180+0000", 2024-04-19T01:04:40.005 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 6.2991510819999998, 2024-04-19T01:04:40.005 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018638968999999998, 2024-04-19T01:04:40.006 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:40.006 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:40.006 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:40.006 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:40.006 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.006 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:40.006 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:33.565180+0000", 2024-04-19T01:04:40.006 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.006 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.006 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.007 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:40.007 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:33.565180+0000", 2024-04-19T01:04:40.007 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.007 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.007 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.007 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:40.007 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:33.565181+0000", 2024-04-19T01:04:40.007 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7700000000000004e-07 2024-04-19T01:04:40.007 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.007 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.007 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:40.007 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:33.565938+0000", 2024-04-19T01:04:40.007 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00075768199999999997 2024-04-19T01:04:40.007 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.008 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.008 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:40.008 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:33.565973+0000", 2024-04-19T01:04:40.008 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5021999999999998e-05 2024-04-19T01:04:40.008 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.008 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.008 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:40.008 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:33.566001+0000", 2024-04-19T01:04:40.008 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8133e-05 2024-04-19T01:04:40.008 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.008 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.008 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:40.008 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:33.566147+0000", 2024-04-19T01:04:40.008 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000145539 2024-04-19T01:04:40.009 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.009 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.009 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:40.009 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:33.566165+0000", 2024-04-19T01:04:40.009 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7788999999999999e-05 2024-04-19T01:04:40.009 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.009 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.009 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:40.009 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:33.583741+0000", 2024-04-19T01:04:40.009 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017576789999999998 2024-04-19T01:04:40.009 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.009 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.009 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:40.009 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:33.583819+0000", 2024-04-19T01:04:40.009 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7237000000000006e-05 2024-04-19T01:04:40.010 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.010 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:40.010 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.010 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.010 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.010 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.as1 64/31 ECSubWrite(tid=106, reqid=osd.12.0:103, at_version=0'0, trim_to=0'0, roll_forward_to=64'24))", 2024-04-19T01:04:40.010 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:34.823315+0000", 2024-04-19T01:04:40.010 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.0410159840000004, 2024-04-19T01:04:40.010 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.006772503, 2024-04-19T01:04:40.010 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:40.010 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:40.010 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:40.010 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:40.010 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.011 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:40.011 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:34.823315+0000", 2024-04-19T01:04:40.011 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.011 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.011 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.011 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:40.011 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:34.823315+0000", 2024-04-19T01:04:40.011 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.011 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.011 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.011 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:40.011 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:34.823317+0000", 2024-04-19T01:04:40.012 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.877e-06 2024-04-19T01:04:40.012 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.013 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.013 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:40.013 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:34.823343+0000", 2024-04-19T01:04:40.013 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6551999999999999e-05 2024-04-19T01:04:40.013 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.013 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.013 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:40.013 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:34.823435+0000", 2024-04-19T01:04:40.013 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.1479999999999998e-05 2024-04-19T01:04:40.014 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.014 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.014 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:40.014 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:34.823458+0000", 2024-04-19T01:04:40.014 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.2976000000000002e-05 2024-04-19T01:04:40.014 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.014 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.014 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:40.014 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:34.823614+0000", 2024-04-19T01:04:40.016 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000155909 2024-04-19T01:04:40.016 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.016 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.016 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:40.016 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:34.823639+0000", 2024-04-19T01:04:40.016 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5463e-05 2024-04-19T01:04:40.016 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.016 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.016 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:40.016 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:34.829989+0000", 2024-04-19T01:04:40.017 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0063495060000000004 2024-04-19T01:04:40.017 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.017 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.017 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:40.017 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:34.830087+0000", 2024-04-19T01:04:40.017 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.8740000000000007e-05 2024-04-19T01:04:40.017 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.017 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:40.017 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.017 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.017 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.017 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.as1 65/31 ECSubWrite(tid=107, reqid=client.4604.0:478, at_version=65'26, trim_to=0'0, roll_forward_to=64'24))", 2024-04-19T01:04:40.017 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:35.639127+0000", 2024-04-19T01:04:40.017 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 4.2252036530000003, 2024-04-19T01:04:40.018 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011274793, 2024-04-19T01:04:40.018 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:40.018 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:40.018 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:40.018 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:40.018 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.018 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:40.018 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639127+0000", 2024-04-19T01:04:40.018 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.018 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.018 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.018 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:40.018 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639127+0000", 2024-04-19T01:04:40.018 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.019 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.019 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.019 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:40.019 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639128+0000", 2024-04-19T01:04:40.019 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6000000000000004e-07 2024-04-19T01:04:40.019 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.020 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.020 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:40.020 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639372+0000", 2024-04-19T01:04:40.020 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00024466199999999998 2024-04-19T01:04:40.020 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.020 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.020 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:40.020 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639401+0000", 2024-04-19T01:04:40.020 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8282000000000001e-05 2024-04-19T01:04:40.020 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.020 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.021 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:40.021 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639446+0000", 2024-04-19T01:04:40.021 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5513000000000003e-05 2024-04-19T01:04:40.021 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.021 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.021 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:40.021 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639612+0000", 2024-04-19T01:04:40.021 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000166069 2024-04-19T01:04:40.021 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.021 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.021 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:40.021 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639626+0000", 2024-04-19T01:04:40.021 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3811999999999999e-05 2024-04-19T01:04:40.021 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.022 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.022 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:40.022 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.650359+0000", 2024-04-19T01:04:40.022 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010732858 2024-04-19T01:04:40.022 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.022 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.022 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:40.022 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.650402+0000", 2024-04-19T01:04:40.022 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3037000000000001e-05 2024-04-19T01:04:40.022 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.022 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:40.022 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.022 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.023 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.023 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.as1 65/31 ECSubWrite(tid=108, reqid=client.4604.0:479, at_version=65'27, trim_to=0'0, roll_forward_to=64'24))", 2024-04-19T01:04:40.023 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:35.654833+0000", 2024-04-19T01:04:40.023 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 4.2094980059999996, 2024-04-19T01:04:40.023 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0075784720000000002, 2024-04-19T01:04:40.023 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:40.023 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:40.023 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:40.023 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:40.023 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.023 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:40.024 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.654833+0000", 2024-04-19T01:04:40.024 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.024 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.024 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.024 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:40.024 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.654833+0000", 2024-04-19T01:04:40.024 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.024 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.024 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.024 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:40.024 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.654834+0000", 2024-04-19T01:04:40.024 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.296e-06 2024-04-19T01:04:40.024 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.024 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.025 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:40.025 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.654940+0000", 2024-04-19T01:04:40.025 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000106012 2024-04-19T01:04:40.025 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.025 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.025 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:40.025 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.654965+0000", 2024-04-19T01:04:40.025 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.472e-05 2024-04-19T01:04:40.025 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.025 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.025 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:40.025 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.654978+0000", 2024-04-19T01:04:40.025 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3274e-05 2024-04-19T01:04:40.025 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.026 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.026 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:40.026 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.655133+0000", 2024-04-19T01:04:40.026 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000154716 2024-04-19T01:04:40.026 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.026 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.026 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:40.026 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.655146+0000", 2024-04-19T01:04:40.026 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2958e-05 2024-04-19T01:04:40.026 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.026 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.026 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:40.026 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.662355+0000", 2024-04-19T01:04:40.026 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0072095470000000002 2024-04-19T01:04:40.026 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.027 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.027 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:40.027 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.662411+0000", 2024-04-19T01:04:40.027 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.5949000000000003e-05 2024-04-19T01:04:40.027 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.027 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:40.027 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.027 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.027 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:40.027 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:40.029 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-19T01:04:40.031 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:04:40.032+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:04:40.129 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:04:40.129+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:04:40.158 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:40.159 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:04:40.159 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2024-04-19T01:04:40.159 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:40.173 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-19T01:04:40.231 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:04:40.232+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:04:40.302 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:40.303 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:04:40.303 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2024-04-19T01:04:40.303 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2024-04-19T01:04:40.303 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:40.315 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-19T01:04:40.333 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:04:40.332+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:04:40.434 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:04:40.433+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:04:40.445 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:40.445 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2024-04-19T01:04:40.445 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2024-04-19T01:04:40.445 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-04-19T01:04:40.445 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.445 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:251 3.5s0 3:a8ddfdee:::smithi15528762-22:head [copy-from ver 6] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:40.445 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.978778+0000", 2024-04-19T01:04:40.446 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 26.464409855, 2024-04-19T01:04:40.446 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.036510621, 2024-04-19T01:04:40.446 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:40.446 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:40.446 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:40.446 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:40.446 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:40.446 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:40.446 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 251 2024-04-19T01:04:40.446 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.446 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:40.446 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.447 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:40.447 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.978778+0000", 2024-04-19T01:04:40.447 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.447 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.447 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.447 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:40.447 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.978778+0000", 2024-04-19T01:04:40.447 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.447 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.448 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.448 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:40.448 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.978780+0000", 2024-04-19T01:04:40.448 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4729999999999999e-06 2024-04-19T01:04:40.448 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.448 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.448 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:40.448 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.978790+0000", 2024-04-19T01:04:40.448 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0319999999999999e-05 2024-04-19T01:04:40.448 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.448 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.449 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:40.449 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.978814+0000", 2024-04-19T01:04:40.449 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3805999999999999e-05 2024-04-19T01:04:40.449 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.449 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.449 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:40.449 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.978828+0000", 2024-04-19T01:04:40.449 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3594000000000001e-05 2024-04-19T01:04:40.449 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.449 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.449 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:40.449 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.978916+0000", 2024-04-19T01:04:40.450 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.8331000000000005e-05 2024-04-19T01:04:40.450 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.450 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.450 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:40.450 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.979042+0000", 2024-04-19T01:04:40.450 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000125592 2024-04-19T01:04:40.450 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.450 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.450 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:40.450 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.984647+0000", 2024-04-19T01:04:40.450 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0056049150000000002 2024-04-19T01:04:40.453 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.453 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.453 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:40.454 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.011228+0000", 2024-04-19T01:04:40.454 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.026581707 2024-04-19T01:04:40.454 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.454 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.454 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:40.454 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.015199+0000", 2024-04-19T01:04:40.454 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0039706569999999998 2024-04-19T01:04:40.454 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.454 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.454 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:40.454 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.015288+0000", 2024-04-19T01:04:40.454 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9226000000000003e-05 2024-04-19T01:04:40.454 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.454 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:40.455 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.455 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.455 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.455 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:252 3.5s0 3:a8ddfdee:::smithi15528762-22:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:40.455 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.978852+0000", 2024-04-19T01:04:40.455 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 26.464335985999998, 2024-04-19T01:04:40.455 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.024670299, 2024-04-19T01:04:40.455 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:40.455 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:40.455 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:40.455 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:40.455 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:40.455 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:40.455 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 252 2024-04-19T01:04:40.456 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.456 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:40.456 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.456 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:40.456 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.978852+0000", 2024-04-19T01:04:40.456 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.456 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.456 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.456 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:40.456 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.978852+0000", 2024-04-19T01:04:40.456 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.456 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.456 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.457 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:40.457 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.978852+0000", 2024-04-19T01:04:40.457 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8899999999999999e-07 2024-04-19T01:04:40.457 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.457 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.457 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:40.457 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.978855+0000", 2024-04-19T01:04:40.457 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.1310000000000002e-06 2024-04-19T01:04:40.457 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.457 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.457 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:40.457 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.978870+0000", 2024-04-19T01:04:40.457 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4994000000000001e-05 2024-04-19T01:04:40.458 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.458 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.458 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:40.458 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.978875+0000", 2024-04-19T01:04:40.458 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6179999999999997e-06 2024-04-19T01:04:40.458 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.458 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.458 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:40.458 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.979272+0000", 2024-04-19T01:04:40.458 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00039709000000000001 2024-04-19T01:04:40.458 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.458 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.459 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for blocked object", 2024-04-19T01:04:40.459 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.979339+0000", 2024-04-19T01:04:40.459 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.7161000000000003e-05 2024-04-19T01:04:40.459 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.459 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.459 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:40.459 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.003307+0000", 2024-04-19T01:04:40.459 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.023967677999999999 2024-04-19T01:04:40.459 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.459 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.459 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:40.459 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.003402+0000", 2024-04-19T01:04:40.460 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.5583999999999998e-05 2024-04-19T01:04:40.460 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.460 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.460 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:40.460 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.003522+0000", 2024-04-19T01:04:40.460 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000119454 2024-04-19T01:04:40.460 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.460 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:40.460 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.460 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.460 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.460 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:258 3.5s0 3:a03c5c89:::smithi15528762-37:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:40.460 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.985209+0000", 2024-04-19T01:04:40.460 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 26.457979015999999, 2024-04-19T01:04:40.461 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.026491132000000001, 2024-04-19T01:04:40.461 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:40.461 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:40.461 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:40.461 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:40.461 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:40.461 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:40.461 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 258 2024-04-19T01:04:40.461 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.461 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:40.461 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.461 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:40.461 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985209+0000", 2024-04-19T01:04:40.461 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.462 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.462 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.462 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:40.462 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985209+0000", 2024-04-19T01:04:40.462 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.462 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.462 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.462 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:40.462 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985209+0000", 2024-04-19T01:04:40.462 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2600000000000002e-07 2024-04-19T01:04:40.462 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.462 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.462 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:40.462 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985214+0000", 2024-04-19T01:04:40.462 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7110000000000003e-06 2024-04-19T01:04:40.463 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.463 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.463 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:40.463 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985225+0000", 2024-04-19T01:04:40.463 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1382e-05 2024-04-19T01:04:40.463 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.463 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.463 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:40.463 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985233+0000", 2024-04-19T01:04:40.463 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.418e-06 2024-04-19T01:04:40.463 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.463 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.463 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:40.464 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.011520+0000", 2024-04-19T01:04:40.464 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.026287015 2024-04-19T01:04:40.464 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.464 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.464 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:40.464 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.011700+0000", 2024-04-19T01:04:40.464 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00017987999999999999 2024-04-19T01:04:40.464 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.464 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:40.464 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.464 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.464 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.464 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:259 3.5s0 3:a03c5c89:::smithi15528762-37:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:40.464 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.985245+0000", 2024-04-19T01:04:40.465 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 26.457942895999999, 2024-04-19T01:04:40.465 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.026550810000000001, 2024-04-19T01:04:40.465 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:40.465 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:40.465 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:40.465 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:40.465 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:40.465 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:40.465 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 259 2024-04-19T01:04:40.465 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.465 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:40.465 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.465 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:40.465 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985245+0000", 2024-04-19T01:04:40.466 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.466 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.466 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.466 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:40.466 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985245+0000", 2024-04-19T01:04:40.466 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.466 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.466 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.466 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:40.466 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985245+0000", 2024-04-19T01:04:40.466 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6000000000000004e-07 2024-04-19T01:04:40.466 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.466 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.467 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:40.467 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985249+0000", 2024-04-19T01:04:40.467 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.036e-06 2024-04-19T01:04:40.467 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.467 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.467 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:40.467 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985258+0000", 2024-04-19T01:04:40.467 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.0569999999999999e-06 2024-04-19T01:04:40.467 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.467 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.467 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:40.467 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985266+0000", 2024-04-19T01:04:40.467 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.0339999999999997e-06 2024-04-19T01:04:40.467 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.467 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.468 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:40.468 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.011729+0000", 2024-04-19T01:04:40.468 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.02646308 2024-04-19T01:04:40.468 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.468 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.468 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:40.468 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.011795+0000", 2024-04-19T01:04:40.468 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.6043000000000004e-05 2024-04-19T01:04:40.468 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.468 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:40.468 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.468 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.468 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.468 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:260 3.5s0 3:a626fd0d:::smithi15528762-17:head [read 0~278528,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:40.469 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.985277+0000", 2024-04-19T01:04:40.469 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 26.457910930000001, 2024-04-19T01:04:40.469 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.029721959999999999, 2024-04-19T01:04:40.469 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:40.469 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:40.469 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:40.469 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:40.469 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:40.469 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:40.469 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 260 2024-04-19T01:04:40.469 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.470 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:40.470 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.470 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:40.470 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985277+0000", 2024-04-19T01:04:40.470 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.470 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.470 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.470 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:40.470 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985277+0000", 2024-04-19T01:04:40.470 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.470 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.471 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.471 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:40.471 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985277+0000", 2024-04-19T01:04:40.471 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8800000000000002e-07 2024-04-19T01:04:40.471 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.471 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.471 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:40.471 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985286+0000", 2024-04-19T01:04:40.471 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.8009999999999993e-06 2024-04-19T01:04:40.471 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.471 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.471 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:40.472 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985297+0000", 2024-04-19T01:04:40.472 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1321000000000001e-05 2024-04-19T01:04:40.472 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.472 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.472 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:40.472 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985305+0000", 2024-04-19T01:04:40.472 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.3390000000000004e-06 2024-04-19T01:04:40.472 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.472 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.472 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:40.472 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.011821+0000", 2024-04-19T01:04:40.472 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.026516310000000001 2024-04-19T01:04:40.472 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.472 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.472 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:40.473 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.011905+0000", 2024-04-19T01:04:40.473 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.4538999999999998e-05 2024-04-19T01:04:40.473 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.473 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.473 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:40.473 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.014999+0000", 2024-04-19T01:04:40.473 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0030930620000000002 2024-04-19T01:04:40.473 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.473 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:40.473 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.473 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.473 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.473 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:261 3.5s0 3:a626fd0d:::smithi15528762-17:head [read 0~278528,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:40.473 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.985317+0000", 2024-04-19T01:04:40.474 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 26.457870492000001, 2024-04-19T01:04:40.474 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.030889950999999999, 2024-04-19T01:04:40.474 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:40.474 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:40.474 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:40.474 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:40.474 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:40.474 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:40.474 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 261 2024-04-19T01:04:40.474 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.474 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:40.474 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.474 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:40.474 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985317+0000", 2024-04-19T01:04:40.475 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.475 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.475 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.475 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:40.475 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985317+0000", 2024-04-19T01:04:40.475 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.475 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.475 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.475 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:40.475 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985318+0000", 2024-04-19T01:04:40.475 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6100000000000001e-07 2024-04-19T01:04:40.475 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.475 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.475 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:40.476 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985321+0000", 2024-04-19T01:04:40.476 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5930000000000002e-06 2024-04-19T01:04:40.476 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.476 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.476 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:40.476 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985329+0000", 2024-04-19T01:04:40.476 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.9699999999999999e-06 2024-04-19T01:04:40.476 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.476 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.476 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:40.476 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985336+0000", 2024-04-19T01:04:40.476 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5300000000000002e-06 2024-04-19T01:04:40.476 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.476 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.477 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:40.477 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.012184+0000", 2024-04-19T01:04:40.477 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.026848627 2024-04-19T01:04:40.477 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.477 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.477 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:40.477 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.012257+0000", 2024-04-19T01:04:40.477 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2534999999999996e-05 2024-04-19T01:04:40.477 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.477 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.477 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:40.477 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.016207+0000", 2024-04-19T01:04:40.477 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0039501349999999996 2024-04-19T01:04:40.477 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.478 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:40.478 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.478 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.478 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.478 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:262 3.5s0 3:a626fd0d:::smithi15528762-17:head [read 0~278528,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:40.478 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.985345+0000", 2024-04-19T01:04:40.478 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 26.457842844000002, 2024-04-19T01:04:40.478 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.031344419999999998, 2024-04-19T01:04:40.478 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:40.478 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:40.478 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:40.478 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:40.478 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:40.479 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:40.479 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 262 2024-04-19T01:04:40.479 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.479 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:40.479 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.479 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:40.479 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985345+0000", 2024-04-19T01:04:40.479 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.479 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.479 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.479 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:40.480 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985345+0000", 2024-04-19T01:04:40.480 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.480 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.480 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.480 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:40.480 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985345+0000", 2024-04-19T01:04:40.480 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3799999999999997e-07 2024-04-19T01:04:40.480 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.480 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.480 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:40.480 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985358+0000", 2024-04-19T01:04:40.480 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2969999999999999e-05 2024-04-19T01:04:40.480 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.480 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.480 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:40.481 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985369+0000", 2024-04-19T01:04:40.481 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0472000000000001e-05 2024-04-19T01:04:40.481 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.481 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.481 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:40.481 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985378+0000", 2024-04-19T01:04:40.481 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.8019999999999998e-06 2024-04-19T01:04:40.481 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.481 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.481 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:40.481 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.012481+0000", 2024-04-19T01:04:40.481 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.027103136 2024-04-19T01:04:40.481 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.482 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.482 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:40.482 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.012531+0000", 2024-04-19T01:04:40.482 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0189999999999999e-05 2024-04-19T01:04:40.482 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.482 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.482 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:40.482 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.016689+0000", 2024-04-19T01:04:40.482 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0041582119999999997 2024-04-19T01:04:40.482 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.482 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:40.482 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.482 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.483 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.483 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds4 43/31 ECSubWrite(tid=38, reqid=client.4604.0:253, at_version=43'14, trim_to=0'0, roll_forward_to=43'12))", 2024-04-19T01:04:40.483 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.991581+0000", 2024-04-19T01:04:40.483 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 26.451606774999998, 2024-04-19T01:04:40.483 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.019891641000000002, 2024-04-19T01:04:40.483 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:40.483 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:40.483 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:40.483 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:40.483 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.483 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:40.483 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991581+0000", 2024-04-19T01:04:40.483 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.484 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.484 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.484 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:40.484 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991581+0000", 2024-04-19T01:04:40.484 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.484 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.484 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.484 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:40.484 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991582+0000", 2024-04-19T01:04:40.484 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7430000000000001e-06 2024-04-19T01:04:40.484 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.484 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.484 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:40.485 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991739+0000", 2024-04-19T01:04:40.485 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015636 2024-04-19T01:04:40.485 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.485 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.485 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:40.485 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991764+0000", 2024-04-19T01:04:40.486 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5626000000000002e-05 2024-04-19T01:04:40.486 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.486 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.486 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:40.486 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991776+0000", 2024-04-19T01:04:40.486 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1154e-05 2024-04-19T01:04:40.486 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.486 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.486 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:40.486 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.003746+0000", 2024-04-19T01:04:40.486 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011970404 2024-04-19T01:04:40.486 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.486 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.486 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:40.487 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.003754+0000", 2024-04-19T01:04:40.487 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.9270000000000005e-06 2024-04-19T01:04:40.487 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.487 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.487 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:40.487 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.011430+0000", 2024-04-19T01:04:40.487 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0076758590000000002 2024-04-19T01:04:40.487 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.487 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.487 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:40.487 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.011472+0000", 2024-04-19T01:04:40.487 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2568000000000003e-05 2024-04-19T01:04:40.487 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.488 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:40.488 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.488 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.488 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.488 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.as6 48/31 ECSubWrite(tid=71, reqid=client.4604.0:354, at_version=48'10, trim_to=0'0, roll_forward_to=34'8))", 2024-04-19T01:04:40.488 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.560650+0000", 2024-04-19T01:04:40.488 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 21.882537678999999, 2024-04-19T01:04:40.488 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.020025294999999999, 2024-04-19T01:04:40.488 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:40.488 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:40.488 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:40.488 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:40.488 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.488 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:40.489 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.560650+0000", 2024-04-19T01:04:40.489 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.489 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.489 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.489 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:40.489 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.560650+0000", 2024-04-19T01:04:40.489 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.489 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.489 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.489 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:40.489 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.560651+0000", 2024-04-19T01:04:40.489 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3990000000000001e-06 2024-04-19T01:04:40.489 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.489 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.490 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:40.490 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.561126+0000", 2024-04-19T01:04:40.490 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00047482999999999998 2024-04-19T01:04:40.490 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.490 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.490 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:40.490 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.561147+0000", 2024-04-19T01:04:40.490 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1036e-05 2024-04-19T01:04:40.490 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.490 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.490 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:40.490 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.561163+0000", 2024-04-19T01:04:40.490 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6059e-05 2024-04-19T01:04:40.490 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.490 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.491 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:40.491 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.561254+0000", 2024-04-19T01:04:40.491 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.0653000000000001e-05 2024-04-19T01:04:40.491 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.491 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.491 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:40.491 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.561268+0000", 2024-04-19T01:04:40.491 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4242e-05 2024-04-19T01:04:40.491 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.491 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.491 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:40.491 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.580610+0000", 2024-04-19T01:04:40.491 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.019342301999999999 2024-04-19T01:04:40.491 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.492 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.492 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:40.492 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.580675+0000", 2024-04-19T01:04:40.492 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4774e-05 2024-04-19T01:04:40.492 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.492 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:40.492 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.492 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.492 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.492 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:423 3.5s0 3:a03c5c89:::smithi15528762-37:head [append 0~655360] snapc e=[d] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-19T01:04:40.492 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:29.537904+0000", 2024-04-19T01:04:40.493 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 10.905283651, 2024-04-19T01:04:40.493 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.049520382000000002, 2024-04-19T01:04:40.493 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:40.493 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:40.493 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:40.493 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:40.493 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:40.493 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:40.493 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 423 2024-04-19T01:04:40.493 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.493 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:40.493 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.493 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:40.494 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.537904+0000", 2024-04-19T01:04:40.494 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.494 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.494 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.494 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:40.494 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.537904+0000", 2024-04-19T01:04:40.494 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.494 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.495 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.495 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:40.495 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.537906+0000", 2024-04-19T01:04:40.495 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7859999999999999e-06 2024-04-19T01:04:40.495 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.495 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.495 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:40.495 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.538813+0000", 2024-04-19T01:04:40.495 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00090743099999999999 2024-04-19T01:04:40.495 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.495 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.495 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:40.495 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.538828+0000", 2024-04-19T01:04:40.496 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5206e-05 2024-04-19T01:04:40.496 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.496 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.496 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:40.496 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.538838+0000", 2024-04-19T01:04:40.496 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.3070000000000008e-06 2024-04-19T01:04:40.496 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.496 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.496 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:40.496 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.538910+0000", 2024-04-19T01:04:40.496 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2206000000000006e-05 2024-04-19T01:04:40.496 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.496 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.496 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:40.497 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.539016+0000", 2024-04-19T01:04:40.497 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010665199999999999 2024-04-19T01:04:40.497 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.497 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.497 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:40.497 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.540080+0000", 2024-04-19T01:04:40.497 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001063273 2024-04-19T01:04:40.497 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.497 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.497 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:40.497 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.545973+0000", 2024-04-19T01:04:40.497 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0058930570000000002 2024-04-19T01:04:40.497 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.497 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.498 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:40.498 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.587359+0000", 2024-04-19T01:04:40.498 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.041386128000000001 2024-04-19T01:04:40.498 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.498 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.498 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:40.498 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.587424+0000", 2024-04-19T01:04:40.498 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5335999999999999e-05 2024-04-19T01:04:40.498 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.498 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:40.498 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.498 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.498 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.498 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:424 3.5s0 3:a03c5c89:::smithi15528762-37:head [append 0~638976] snapc e=[d] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-19T01:04:40.499 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:29.545481+0000", 2024-04-19T01:04:40.499 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 10.897706662999999, 2024-04-19T01:04:40.499 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.042187482999999998, 2024-04-19T01:04:40.501 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:40.501 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:40.501 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:40.501 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:40.502 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:40.502 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:40.502 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 424 2024-04-19T01:04:40.502 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.502 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:40.502 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.502 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:40.502 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.545481+0000", 2024-04-19T01:04:40.502 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.502 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.502 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.503 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:40.503 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.545481+0000", 2024-04-19T01:04:40.503 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.503 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.503 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.504 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:40.504 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.545483+0000", 2024-04-19T01:04:40.504 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3879999999999998e-06 2024-04-19T01:04:40.504 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.504 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.504 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:40.504 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.546298+0000", 2024-04-19T01:04:40.504 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00081462099999999996 2024-04-19T01:04:40.504 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.504 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.504 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:40.504 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.546314+0000", 2024-04-19T01:04:40.505 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5722e-05 2024-04-19T01:04:40.505 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.505 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.505 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:40.505 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.546320+0000", 2024-04-19T01:04:40.505 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0939999999999996e-06 2024-04-19T01:04:40.506 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.506 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.506 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:40.506 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.546381+0000", 2024-04-19T01:04:40.506 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1696e-05 2024-04-19T01:04:40.506 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.506 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.506 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:40.506 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.546497+0000", 2024-04-19T01:04:40.506 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000115903 2024-04-19T01:04:40.506 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.506 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.506 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:40.506 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547522+0000", 2024-04-19T01:04:40.507 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010249289999999999 2024-04-19T01:04:40.507 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.507 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.507 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:40.507 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.556940+0000", 2024-04-19T01:04:40.507 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0094178899999999999 2024-04-19T01:04:40.507 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.507 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.507 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:40.507 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.587578+0000", 2024-04-19T01:04:40.507 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.030638021000000001 2024-04-19T01:04:40.507 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.507 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.507 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:40.507 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.587668+0000", 2024-04-19T01:04:40.508 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.0218999999999998e-05 2024-04-19T01:04:40.508 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.508 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:40.508 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.508 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.508 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.508 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:425 3.5s0 3:a03c5c89:::smithi15528762-37:head [append 0~131072] snapc e=[d] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-19T01:04:40.508 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:29.547018+0000", 2024-04-19T01:04:40.508 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 10.896169221999999, 2024-04-19T01:04:40.508 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.040774228000000003, 2024-04-19T01:04:40.508 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:40.508 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:40.509 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:40.509 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:40.509 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:40.509 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:40.509 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 425 2024-04-19T01:04:40.509 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.509 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:40.509 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.509 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:40.509 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547018+0000", 2024-04-19T01:04:40.509 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.509 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.510 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.510 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:40.510 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547018+0000", 2024-04-19T01:04:40.510 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.510 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.510 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.510 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:40.510 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547020+0000", 2024-04-19T01:04:40.510 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3710000000000001e-06 2024-04-19T01:04:40.510 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.510 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.510 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:40.510 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547148+0000", 2024-04-19T01:04:40.510 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012848699999999999 2024-04-19T01:04:40.511 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.511 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.511 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:40.511 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547162+0000", 2024-04-19T01:04:40.511 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4248e-05 2024-04-19T01:04:40.511 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.511 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.511 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:40.511 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547170+0000", 2024-04-19T01:04:40.511 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2069999999999998e-06 2024-04-19T01:04:40.511 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.511 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.511 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:40.511 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.554554+0000", 2024-04-19T01:04:40.512 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0073842159999999999 2024-04-19T01:04:40.512 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.512 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.512 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:40.512 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.554666+0000", 2024-04-19T01:04:40.513 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011222699999999999 2024-04-19T01:04:40.513 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.513 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.513 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:40.513 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.555080+0000", 2024-04-19T01:04:40.513 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00041430500000000001 2024-04-19T01:04:40.513 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.513 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.514 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:40.514 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.557075+0000", 2024-04-19T01:04:40.514 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001994198 2024-04-19T01:04:40.514 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.514 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.514 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:40.514 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.587746+0000", 2024-04-19T01:04:40.514 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.030671032000000001 2024-04-19T01:04:40.514 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.514 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.514 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:40.514 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.587793+0000", 2024-04-19T01:04:40.514 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6937000000000001e-05 2024-04-19T01:04:40.514 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.515 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:40.515 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.515 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.515 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.515 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:426 3.5s0 3:a03c5c89:::smithi15528762-37:head [setxattr _header (57) in=7b] snapc e=[d] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-19T01:04:40.515 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:29.547183+0000", 2024-04-19T01:04:40.515 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 10.896004387, 2024-04-19T01:04:40.515 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.040837638000000002, 2024-04-19T01:04:40.515 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:40.515 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:40.515 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:40.515 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:40.515 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:40.515 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:40.516 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 426 2024-04-19T01:04:40.516 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.516 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:40.516 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.516 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:40.516 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547183+0000", 2024-04-19T01:04:40.516 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.516 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.516 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.516 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:40.516 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547183+0000", 2024-04-19T01:04:40.516 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.516 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.516 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.516 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:40.517 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547184+0000", 2024-04-19T01:04:40.517 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.6899999999999997e-07 2024-04-19T01:04:40.517 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.517 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.517 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:40.517 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547188+0000", 2024-04-19T01:04:40.517 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4150000000000001e-06 2024-04-19T01:04:40.517 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.517 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.517 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:40.517 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547196+0000", 2024-04-19T01:04:40.517 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.8660000000000006e-06 2024-04-19T01:04:40.517 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.518 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.518 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:40.518 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547203+0000", 2024-04-19T01:04:40.518 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.0550000000000002e-06 2024-04-19T01:04:40.518 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.518 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.518 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:40.518 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.556074+0000", 2024-04-19T01:04:40.518 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0088710260000000006 2024-04-19T01:04:40.518 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.518 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.519 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:40.519 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.556179+0000", 2024-04-19T01:04:40.519 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000104947 2024-04-19T01:04:40.519 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.519 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.519 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:40.519 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.556694+0000", 2024-04-19T01:04:40.519 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000514563 2024-04-19T01:04:40.519 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.519 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.519 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:40.519 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.557301+0000", 2024-04-19T01:04:40.519 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00060763899999999999 2024-04-19T01:04:40.519 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.519 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.520 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:40.520 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.587923+0000", 2024-04-19T01:04:40.520 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.030621770999999999 2024-04-19T01:04:40.520 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.520 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.520 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:40.520 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.588021+0000", 2024-04-19T01:04:40.520 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7687000000000002e-05 2024-04-19T01:04:40.520 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.520 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:40.520 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.520 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.520 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.520 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:427 3.5s0 3:a03c5c89:::smithi15528762-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e59)", 2024-04-19T01:04:40.520 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:29.547214+0000", 2024-04-19T01:04:40.521 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 10.895973872999999, 2024-04-19T01:04:40.521 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.043029057000000002, 2024-04-19T01:04:40.521 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:40.521 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:40.521 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:40.521 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:40.521 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:40.521 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:40.521 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 427 2024-04-19T01:04:40.521 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.521 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:40.521 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.522 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:40.522 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547214+0000", 2024-04-19T01:04:40.522 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.522 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.522 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.522 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:40.522 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547214+0000", 2024-04-19T01:04:40.522 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.522 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.522 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.522 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:40.523 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547214+0000", 2024-04-19T01:04:40.523 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3799999999999998e-07 2024-04-19T01:04:40.523 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.523 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.524 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:40.524 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547216+0000", 2024-04-19T01:04:40.524 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.131e-06 2024-04-19T01:04:40.524 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.524 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.524 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:40.524 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547223+0000", 2024-04-19T01:04:40.524 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2790000000000004e-06 2024-04-19T01:04:40.524 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.524 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.524 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:40.524 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547226+0000", 2024-04-19T01:04:40.524 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4529999999999998e-06 2024-04-19T01:04:40.524 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.525 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.525 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:40.525 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.556968+0000", 2024-04-19T01:04:40.525 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0097420590000000008 2024-04-19T01:04:40.525 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.525 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.525 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:04:40.526 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.557060+0000", 2024-04-19T01:04:40.526 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.1807999999999999e-05 2024-04-19T01:04:40.526 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.526 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.526 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:40.526 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.588480+0000", 2024-04-19T01:04:40.526 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.031420333000000002 2024-04-19T01:04:40.526 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.526 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.526 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:40.526 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.588538+0000", 2024-04-19T01:04:40.526 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.7652000000000003e-05 2024-04-19T01:04:40.526 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.526 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.527 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:40.527 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.590243+0000", 2024-04-19T01:04:40.527 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0017049039999999999 2024-04-19T01:04:40.527 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.527 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:40.527 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.527 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.527 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.527 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.as6 65/31 ECSubWrite(tid=108, reqid=client.4604.0:479, at_version=65'27, trim_to=0'0, roll_forward_to=64'24))", 2024-04-19T01:04:40.527 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:35.654664+0000", 2024-04-19T01:04:40.527 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 4.7885233329999997, 2024-04-19T01:04:40.527 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.020133563, 2024-04-19T01:04:40.527 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:40.527 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:40.527 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:40.528 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:40.528 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.528 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:40.528 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.654664+0000", 2024-04-19T01:04:40.528 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.528 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.528 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.528 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:40.528 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.654664+0000", 2024-04-19T01:04:40.528 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.528 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.528 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.528 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:40.528 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.654666+0000", 2024-04-19T01:04:40.529 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7680000000000001e-06 2024-04-19T01:04:40.529 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.529 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.529 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:40.529 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.654876+0000", 2024-04-19T01:04:40.529 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00020975200000000001 2024-04-19T01:04:40.529 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.529 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.529 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:40.529 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.654898+0000", 2024-04-19T01:04:40.529 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.2059999999999999e-05 2024-04-19T01:04:40.529 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.529 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.529 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:40.529 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.654916+0000", 2024-04-19T01:04:40.530 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7770000000000001e-05 2024-04-19T01:04:40.530 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.530 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.530 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:40.530 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.655073+0000", 2024-04-19T01:04:40.530 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015758600000000001 2024-04-19T01:04:40.530 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.530 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.530 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:40.530 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.655103+0000", 2024-04-19T01:04:40.530 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9646e-05 2024-04-19T01:04:40.531 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.532 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.532 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:40.532 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.674745+0000", 2024-04-19T01:04:40.532 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.019641812000000002 2024-04-19T01:04:40.532 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.532 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.532 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:40.532 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.674798+0000", 2024-04-19T01:04:40.532 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3168999999999999e-05 2024-04-19T01:04:40.532 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.532 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:40.532 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.532 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.533 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.533 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:486 3.5s0 3:abe8a6c4:::smithi15528762-18:head [append 0~786432] snapc 12=[12,11,10] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-19T01:04:40.533 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:35.656780+0000", 2024-04-19T01:04:40.533 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 4.7864074639999998, 2024-04-19T01:04:40.533 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.036473492000000003, 2024-04-19T01:04:40.533 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:40.533 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:40.533 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:40.533 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:40.534 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:40.534 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:40.534 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 486 2024-04-19T01:04:40.534 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.534 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:40.534 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.534 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:40.534 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.656780+0000", 2024-04-19T01:04:40.534 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.534 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.534 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.534 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:40.534 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.656780+0000", 2024-04-19T01:04:40.535 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.535 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.535 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.535 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:40.535 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.656783+0000", 2024-04-19T01:04:40.535 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5390000000000002e-06 2024-04-19T01:04:40.535 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.535 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.535 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:40.535 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.658177+0000", 2024-04-19T01:04:40.535 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0013948890000000001 2024-04-19T01:04:40.535 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.535 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.535 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:40.535 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.658198+0000", 2024-04-19T01:04:40.536 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0183e-05 2024-04-19T01:04:40.536 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.536 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.536 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:40.536 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.658206+0000", 2024-04-19T01:04:40.536 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.789e-06 2024-04-19T01:04:40.536 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.536 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.536 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:40.536 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.658295+0000", 2024-04-19T01:04:40.536 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.8221999999999997e-05 2024-04-19T01:04:40.536 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.536 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.536 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:40.536 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.658393+0000", 2024-04-19T01:04:40.537 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.8469999999999997e-05 2024-04-19T01:04:40.537 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.537 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.537 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:40.537 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.659728+0000", 2024-04-19T01:04:40.537 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001335188 2024-04-19T01:04:40.537 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.537 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.537 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:40.537 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.683791+0000", 2024-04-19T01:04:40.537 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.024062798999999999 2024-04-19T01:04:40.537 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.537 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.537 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:40.537 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.692985+0000", 2024-04-19T01:04:40.538 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0091939940000000005 2024-04-19T01:04:40.538 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.538 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.538 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:40.538 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.693254+0000", 2024-04-19T01:04:40.538 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00026841899999999999 2024-04-19T01:04:40.538 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.538 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:40.538 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.538 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.538 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.538 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:487 3.5s0 3:abe8a6c4:::smithi15528762-18:head [append 0~491520] snapc 12=[12,11,10] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-19T01:04:40.538 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:35.665403+0000", 2024-04-19T01:04:40.538 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 4.7777846559999997, 2024-04-19T01:04:40.539 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.028697970999999999, 2024-04-19T01:04:40.539 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:40.539 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:40.539 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:40.539 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:40.539 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:40.539 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:40.539 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 487 2024-04-19T01:04:40.539 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.539 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:40.539 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.539 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:40.539 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.665403+0000", 2024-04-19T01:04:40.539 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.539 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.540 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.540 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:40.540 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.665403+0000", 2024-04-19T01:04:40.540 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.540 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.540 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.540 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:40.540 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.665406+0000", 2024-04-19T01:04:40.540 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.717e-06 2024-04-19T01:04:40.540 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.540 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.541 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:40.541 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.666514+0000", 2024-04-19T01:04:40.541 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001108436 2024-04-19T01:04:40.541 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.541 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.541 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:40.541 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.666548+0000", 2024-04-19T01:04:40.541 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.3578999999999998e-05 2024-04-19T01:04:40.541 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.541 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.541 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:40.541 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.666569+0000", 2024-04-19T01:04:40.541 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1299000000000001e-05 2024-04-19T01:04:40.541 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.541 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.542 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:40.542 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.670488+0000", 2024-04-19T01:04:40.542 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0039193420000000001 2024-04-19T01:04:40.542 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.542 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.542 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:40.542 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.670756+0000", 2024-04-19T01:04:40.542 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00026774900000000001 2024-04-19T01:04:40.542 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.542 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.542 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:40.542 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.672698+0000", 2024-04-19T01:04:40.542 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001941953 2024-04-19T01:04:40.543 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.543 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.543 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:40.543 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.692679+0000", 2024-04-19T01:04:40.543 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.019980976000000001 2024-04-19T01:04:40.543 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.543 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.543 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:40.543 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.693987+0000", 2024-04-19T01:04:40.543 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0013085659999999999 2024-04-19T01:04:40.543 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.543 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.543 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:40.543 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.694101+0000", 2024-04-19T01:04:40.544 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000113354 2024-04-19T01:04:40.544 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.544 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:40.544 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.544 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.544 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.544 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:488 3.5s0 3:abe8a6c4:::smithi15528762-18:head [append 0~278528] snapc 12=[12,11,10] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-19T01:04:40.544 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:35.669725+0000", 2024-04-19T01:04:40.544 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 4.7734620850000002, 2024-04-19T01:04:40.545 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.025088414, 2024-04-19T01:04:40.545 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:40.545 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:40.545 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:40.545 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:40.545 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:40.545 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:40.545 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 488 2024-04-19T01:04:40.545 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.545 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:40.545 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.545 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:40.545 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.669725+0000", 2024-04-19T01:04:40.545 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.545 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.546 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.546 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:40.546 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.669725+0000", 2024-04-19T01:04:40.546 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.546 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.546 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.546 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:40.546 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.669728+0000", 2024-04-19T01:04:40.546 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4499999999999998e-06 2024-04-19T01:04:40.546 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.546 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.546 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:40.546 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.670117+0000", 2024-04-19T01:04:40.546 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00038870099999999998 2024-04-19T01:04:40.547 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.547 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.547 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:40.547 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.670167+0000", 2024-04-19T01:04:40.547 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0355000000000002e-05 2024-04-19T01:04:40.547 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.547 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.547 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:40.547 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.670187+0000", 2024-04-19T01:04:40.547 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0499000000000001e-05 2024-04-19T01:04:40.547 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.547 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.547 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:40.547 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.684202+0000", 2024-04-19T01:04:40.547 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014014571 2024-04-19T01:04:40.548 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.548 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.548 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:40.548 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.684363+0000", 2024-04-19T01:04:40.548 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00016092499999999999 2024-04-19T01:04:40.548 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.548 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.548 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:40.548 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.685478+0000", 2024-04-19T01:04:40.548 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0011147080000000001 2024-04-19T01:04:40.548 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.548 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.548 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:40.548 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.692682+0000", 2024-04-19T01:04:40.549 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.007204762 2024-04-19T01:04:40.549 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.549 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.549 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:40.549 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.694708+0000", 2024-04-19T01:04:40.549 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0020260069999999998 2024-04-19T01:04:40.549 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.549 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.549 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:40.549 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.694814+0000", 2024-04-19T01:04:40.549 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000105436 2024-04-19T01:04:40.549 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.549 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:40.549 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.549 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.550 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.550 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:489 3.5s0 3:abe8a6c4:::smithi15528762-18:head [setxattr _header (57) in=7b] snapc 12=[12,11,10] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-19T01:04:40.550 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:35.670219+0000", 2024-04-19T01:04:40.550 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 4.7729681079999997, 2024-04-19T01:04:40.550 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.024973339000000001, 2024-04-19T01:04:40.550 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:40.550 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:40.550 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:40.550 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:40.550 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:40.550 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:40.550 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 489 2024-04-19T01:04:40.550 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.550 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:40.551 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.551 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:40.551 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.670219+0000", 2024-04-19T01:04:40.551 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.551 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.551 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.551 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:40.551 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.670219+0000", 2024-04-19T01:04:40.551 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.551 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.551 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.551 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:40.551 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.670221+0000", 2024-04-19T01:04:40.551 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.564e-06 2024-04-19T01:04:40.551 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.552 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.552 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:40.552 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.670230+0000", 2024-04-19T01:04:40.552 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4259999999999992e-06 2024-04-19T01:04:40.552 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.552 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.552 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:40.552 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.670256+0000", 2024-04-19T01:04:40.552 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5366000000000001e-05 2024-04-19T01:04:40.552 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.552 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.552 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:40.552 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.670338+0000", 2024-04-19T01:04:40.552 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1750999999999997e-05 2024-04-19T01:04:40.552 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.553 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.553 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:40.553 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.690787+0000", 2024-04-19T01:04:40.553 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.020449821 2024-04-19T01:04:40.553 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.553 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.553 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:40.553 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.691071+0000", 2024-04-19T01:04:40.553 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00028331100000000001 2024-04-19T01:04:40.553 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.553 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.553 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:40.553 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.692018+0000", 2024-04-19T01:04:40.553 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00094739800000000001 2024-04-19T01:04:40.553 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.554 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.554 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:40.554 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.693827+0000", 2024-04-19T01:04:40.554 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0018090109999999999 2024-04-19T01:04:40.554 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.554 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.554 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:40.554 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.695089+0000", 2024-04-19T01:04:40.555 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001262064 2024-04-19T01:04:40.555 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.555 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.555 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:40.555 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.695193+0000", 2024-04-19T01:04:40.555 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000103627 2024-04-19T01:04:40.555 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.555 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:40.555 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.555 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.555 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.555 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:490 3.5s0 3:abe8a6c4:::smithi15528762-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-04-19T01:04:40.555 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:35.670369+0000", 2024-04-19T01:04:40.555 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 4.7728186089999998, 2024-04-19T01:04:40.555 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.027880933, 2024-04-19T01:04:40.556 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:40.556 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:40.556 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:40.556 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:40.556 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:40.556 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:40.556 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 490 2024-04-19T01:04:40.556 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.556 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:40.556 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.556 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:40.556 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.670369+0000", 2024-04-19T01:04:40.556 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.556 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.556 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.557 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:40.557 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.670369+0000", 2024-04-19T01:04:40.557 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:40.557 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.557 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.557 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:40.557 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.670370+0000", 2024-04-19T01:04:40.557 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5e-06 2024-04-19T01:04:40.557 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.557 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.557 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:40.557 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.670379+0000", 2024-04-19T01:04:40.557 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.4300000000000006e-06 2024-04-19T01:04:40.557 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.558 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.558 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:40.558 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.670406+0000", 2024-04-19T01:04:40.558 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7155999999999999e-05 2024-04-19T01:04:40.558 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.558 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.558 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:40.558 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.670424+0000", 2024-04-19T01:04:40.558 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7762e-05 2024-04-19T01:04:40.558 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.558 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.558 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:40.558 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.695891+0000", 2024-04-19T01:04:40.558 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.025467467000000001 2024-04-19T01:04:40.559 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.559 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.559 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:40.559 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.696040+0000", 2024-04-19T01:04:40.559 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014830900000000001 2024-04-19T01:04:40.559 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:40.559 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:40.559 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:40.559 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.698250+0000", 2024-04-19T01:04:40.559 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0022103090000000001 2024-04-19T01:04:40.559 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.559 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:40.559 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.559 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:40.559 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:40.560 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:40.561 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-19T01:04:40.564 INFO:tasks.rados.rados.0.smithi155.stdout:212: done (0 left) 2024-04-19T01:04:40.564 INFO:tasks.rados.rados.0.smithi155.stdout:213: snap_create 2024-04-19T01:04:40.566 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-04-19T01:04:40.567 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.3 to 1 (10000) 2024-04-19T01:04:40.568 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-04-19T01:04:40.568 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:04:40.562+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:04:40.663 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:04:40.664+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:04:40.731 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:40.732 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:04:40.732 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2024-04-19T01:04:40.732 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:40.752 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-04-19T01:04:40.765 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:04:40.762+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:04:40.865 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:04:40.864+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:04:40.870 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:40.870 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:04:40.870 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2024-04-19T01:04:40.870 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2024-04-19T01:04:40.870 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:40.884 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-04-19T01:04:40.966 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:04:40.966+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:04:41.005 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:41.049 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2024-04-19T01:04:41.049 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2024-04-19T01:04:41.049 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-04-19T01:04:41.050 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.050 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:253 3.ds0 3:b9317ffa:::smithi15528762-1:head [append 0~704512] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:41.050 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.983022+0000", 2024-04-19T01:04:41.050 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 27.018661581, 2024-04-19T01:04:41.050 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.034825673000000001, 2024-04-19T01:04:41.050 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:41.050 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:41.050 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:41.050 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:41.054 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:41.054 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:41.055 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 253 2024-04-19T01:04:41.055 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.055 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:41.055 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.055 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:41.055 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.983022+0000", 2024-04-19T01:04:41.055 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:41.055 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.055 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.055 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:41.055 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.983022+0000", 2024-04-19T01:04:41.055 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:41.055 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.055 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.056 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:41.056 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.983023+0000", 2024-04-19T01:04:41.056 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.6500000000000008e-07 2024-04-19T01:04:41.056 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.056 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.056 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:41.056 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.983925+0000", 2024-04-19T01:04:41.056 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000902439 2024-04-19T01:04:41.056 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.056 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.056 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:41.056 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.983943+0000", 2024-04-19T01:04:41.057 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7975e-05 2024-04-19T01:04:41.057 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.057 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.057 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:41.057 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.983952+0000", 2024-04-19T01:04:41.057 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.4270000000000008e-06 2024-04-19T01:04:41.057 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.057 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.057 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:41.057 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.984029+0000", 2024-04-19T01:04:41.057 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7631000000000003e-05 2024-04-19T01:04:41.057 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.057 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.058 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:41.058 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.984150+0000", 2024-04-19T01:04:41.058 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000120548 2024-04-19T01:04:41.058 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.058 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.058 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:41.058 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.986166+0000", 2024-04-19T01:04:41.058 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0020163410000000001 2024-04-19T01:04:41.058 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.058 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.058 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:41.058 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.001084+0000", 2024-04-19T01:04:41.058 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014917872 2024-04-19T01:04:41.058 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.059 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.059 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:41.059 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.017779+0000", 2024-04-19T01:04:41.059 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.016694817000000001 2024-04-19T01:04:41.059 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.059 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.059 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:41.059 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.017848+0000", 2024-04-19T01:04:41.059 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.8658000000000006e-05 2024-04-19T01:04:41.059 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:41.059 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:41.059 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:41.060 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.060 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.060 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:254 3.ds0 3:b9317ffa:::smithi15528762-1:head [append 0~376832] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:41.060 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.985096+0000", 2024-04-19T01:04:41.060 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 27.016587965999999, 2024-04-19T01:04:41.060 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.043912868000000001, 2024-04-19T01:04:41.060 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:41.060 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:41.060 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:41.060 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:41.060 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:41.060 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:41.060 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 254 2024-04-19T01:04:41.060 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.061 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:41.061 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.061 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:41.061 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985096+0000", 2024-04-19T01:04:41.061 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:41.061 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.061 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.063 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:41.063 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985096+0000", 2024-04-19T01:04:41.063 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:41.063 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.063 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.063 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:41.063 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985098+0000", 2024-04-19T01:04:41.063 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0540000000000002e-06 2024-04-19T01:04:41.063 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.063 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.063 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:41.063 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985632+0000", 2024-04-19T01:04:41.064 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00053471200000000001 2024-04-19T01:04:41.064 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.064 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.064 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:41.064 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985650+0000", 2024-04-19T01:04:41.064 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7217000000000001e-05 2024-04-19T01:04:41.064 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.064 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.064 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:41.065 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985658+0000", 2024-04-19T01:04:41.065 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.8689999999999993e-06 2024-04-19T01:04:41.065 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.065 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.065 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:41.065 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.995027+0000", 2024-04-19T01:04:41.065 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0093689159999999997 2024-04-19T01:04:41.065 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.066 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.066 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:41.066 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.995141+0000", 2024-04-19T01:04:41.066 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011387000000000001 2024-04-19T01:04:41.066 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.066 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.066 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:41.066 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.996005+0000", 2024-04-19T01:04:41.066 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00086397999999999998 2024-04-19T01:04:41.066 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.066 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.066 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:41.066 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.002138+0000", 2024-04-19T01:04:41.073 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.006133182 2024-04-19T01:04:41.073 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.074 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.074 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:41.074 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.028913+0000", 2024-04-19T01:04:41.074 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.026774686999999998 2024-04-19T01:04:41.074 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.074 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.074 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:41.074 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.029008+0000", 2024-04-19T01:04:41.074 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.5381e-05 2024-04-19T01:04:41.074 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:41.076 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:41.076 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:41.076 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.076 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.076 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:255 3.ds0 3:b9317ffa:::smithi15528762-1:head [setxattr _header (55) in=7b] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:41.076 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.985699+0000", 2024-04-19T01:04:41.076 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 27.015984427999999, 2024-04-19T01:04:41.076 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.043535886000000003, 2024-04-19T01:04:41.076 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:41.077 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:41.077 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:41.077 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:41.077 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:41.077 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:41.077 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 255 2024-04-19T01:04:41.077 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.077 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:41.077 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.077 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:41.082 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985699+0000", 2024-04-19T01:04:41.082 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:41.082 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.082 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.082 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:41.082 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985699+0000", 2024-04-19T01:04:41.082 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:41.082 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.083 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.083 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:41.083 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985700+0000", 2024-04-19T01:04:41.083 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4199999999999995e-07 2024-04-19T01:04:41.083 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.083 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.083 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:41.083 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985705+0000", 2024-04-19T01:04:41.083 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7960000000000003e-06 2024-04-19T01:04:41.083 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.083 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.083 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:41.083 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985715+0000", 2024-04-19T01:04:41.084 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0285999999999999e-05 2024-04-19T01:04:41.084 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.084 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.084 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:41.084 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985735+0000", 2024-04-19T01:04:41.084 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9666000000000001e-05 2024-04-19T01:04:41.084 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.084 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.084 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:41.084 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.001214+0000", 2024-04-19T01:04:41.085 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015479694 2024-04-19T01:04:41.085 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.085 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.085 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:41.085 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.001417+0000", 2024-04-19T01:04:41.085 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00020274099999999999 2024-04-19T01:04:41.085 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.085 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.085 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:41.085 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.001931+0000", 2024-04-19T01:04:41.085 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000513599 2024-04-19T01:04:41.085 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.085 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.085 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:41.085 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.002446+0000", 2024-04-19T01:04:41.086 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00051521100000000001 2024-04-19T01:04:41.086 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.086 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.086 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:41.086 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.029136+0000", 2024-04-19T01:04:41.093 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.026690523000000001 2024-04-19T01:04:41.093 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.093 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.093 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:41.093 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.029235+0000", 2024-04-19T01:04:41.093 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.8727999999999994e-05 2024-04-19T01:04:41.093 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:41.093 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:41.093 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:41.093 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.093 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.093 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:256 3.ds0 3:b9317ffa:::smithi15528762-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:41.094 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.985744+0000", 2024-04-19T01:04:41.094 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 27.015939641999999, 2024-04-19T01:04:41.094 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.046574693, 2024-04-19T01:04:41.094 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:41.094 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:41.094 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:41.094 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:41.094 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:41.094 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:41.094 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 256 2024-04-19T01:04:41.094 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.094 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:41.094 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.094 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:41.095 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985744+0000", 2024-04-19T01:04:41.095 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:41.095 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.095 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.095 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:41.095 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985744+0000", 2024-04-19T01:04:41.095 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:41.095 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.095 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.095 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:41.095 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985744+0000", 2024-04-19T01:04:41.095 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1699999999999999e-07 2024-04-19T01:04:41.095 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.095 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.095 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:41.096 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985747+0000", 2024-04-19T01:04:41.096 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3429999999999998e-06 2024-04-19T01:04:41.096 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.096 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.096 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:41.096 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985753+0000", 2024-04-19T01:04:41.096 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8619999999999999e-06 2024-04-19T01:04:41.096 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.096 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.096 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:41.096 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985756+0000", 2024-04-19T01:04:41.096 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8820000000000003e-06 2024-04-19T01:04:41.096 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.096 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.097 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:41.097 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.002179+0000", 2024-04-19T01:04:41.097 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.01642302 2024-04-19T01:04:41.097 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.097 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.097 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:04:41.097 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.002278+0000", 2024-04-19T01:04:41.097 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.8548000000000006e-05 2024-04-19T01:04:41.097 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.098 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.098 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:41.098 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.029285+0000", 2024-04-19T01:04:41.098 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.027006935999999999 2024-04-19T01:04:41.098 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.098 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.098 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:41.098 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.029400+0000", 2024-04-19T01:04:41.098 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000115438 2024-04-19T01:04:41.098 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.098 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.098 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:41.098 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.032319+0000", 2024-04-19T01:04:41.099 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0029182470000000001 2024-04-19T01:04:41.099 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:41.099 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:41.099 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:41.099 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.099 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.099 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:271 3.ds0 3:bf91dcb5:::smithi15528762-2:head [copy-from ver 2] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:41.099 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:14.014802+0000", 2024-04-19T01:04:41.099 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 26.986881177000001, 2024-04-19T01:04:41.099 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015357356000000001, 2024-04-19T01:04:41.099 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:41.099 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:41.099 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:41.099 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:41.099 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:41.100 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:41.100 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 271 2024-04-19T01:04:41.100 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.100 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:41.100 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.100 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:41.100 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.014802+0000", 2024-04-19T01:04:41.100 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:41.100 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.100 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.100 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:41.100 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.014802+0000", 2024-04-19T01:04:41.100 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:41.100 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.100 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.101 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:41.101 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.014804+0000", 2024-04-19T01:04:41.101 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7570000000000001e-06 2024-04-19T01:04:41.101 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.101 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.101 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:41.101 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.014811+0000", 2024-04-19T01:04:41.101 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.3529999999999997e-06 2024-04-19T01:04:41.101 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.101 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.101 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:41.101 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.014822+0000", 2024-04-19T01:04:41.101 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0098000000000001e-05 2024-04-19T01:04:41.101 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.101 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.102 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:41.102 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.014826+0000", 2024-04-19T01:04:41.102 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8350000000000003e-06 2024-04-19T01:04:41.102 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.102 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.102 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:41.102 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.014864+0000", 2024-04-19T01:04:41.102 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7364999999999999e-05 2024-04-19T01:04:41.102 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.102 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.102 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:41.102 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.014966+0000", 2024-04-19T01:04:41.102 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000102012 2024-04-19T01:04:41.102 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.103 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.103 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:41.103 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.018689+0000", 2024-04-19T01:04:41.103 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0037234669999999998 2024-04-19T01:04:41.103 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.103 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.103 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:41.103 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.021559+0000", 2024-04-19T01:04:41.103 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0028693740000000001 2024-04-19T01:04:41.103 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.103 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.103 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:41.103 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.030075+0000", 2024-04-19T01:04:41.103 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0085168429999999996 2024-04-19T01:04:41.104 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.104 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.104 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:41.104 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.030160+0000", 2024-04-19T01:04:41.104 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.4252000000000006e-05 2024-04-19T01:04:41.104 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:41.104 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:41.104 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:41.104 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.104 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.104 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:330 3.ds0 3:bf91dcb5:::smithi15528762-2:head [append 0~507904] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:04:41.104 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.674394+0000", 2024-04-19T01:04:41.104 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 23.327289490999998, 2024-04-19T01:04:41.104 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.036711424999999999, 2024-04-19T01:04:41.105 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:41.105 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:41.105 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:41.105 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:41.105 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:41.105 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:41.105 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 330 2024-04-19T01:04:41.105 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.105 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:41.105 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.105 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:41.105 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.674394+0000", 2024-04-19T01:04:41.105 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:41.105 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.105 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.106 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:41.106 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.674394+0000", 2024-04-19T01:04:41.106 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:41.106 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.106 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.106 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:41.106 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.674396+0000", 2024-04-19T01:04:41.106 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3870000000000002e-06 2024-04-19T01:04:41.106 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.106 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.106 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:41.106 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.675539+0000", 2024-04-19T01:04:41.106 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001142321 2024-04-19T01:04:41.106 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.107 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.107 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:41.107 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.675564+0000", 2024-04-19T01:04:41.107 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5126e-05 2024-04-19T01:04:41.107 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.107 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.107 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:41.107 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.675575+0000", 2024-04-19T01:04:41.107 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0951999999999999e-05 2024-04-19T01:04:41.107 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.107 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.107 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:41.107 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.675641+0000", 2024-04-19T01:04:41.107 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5914999999999995e-05 2024-04-19T01:04:41.107 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.108 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.108 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:41.108 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.675713+0000", 2024-04-19T01:04:41.108 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2169000000000006e-05 2024-04-19T01:04:41.108 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.108 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.108 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:41.108 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.676428+0000", 2024-04-19T01:04:41.108 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00071535000000000001 2024-04-19T01:04:41.108 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.108 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.108 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:41.108 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680290+0000", 2024-04-19T01:04:41.108 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003861532 2024-04-19T01:04:41.108 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.109 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.109 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:41.109 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.711050+0000", 2024-04-19T01:04:41.109 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.030759916000000002 2024-04-19T01:04:41.109 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.109 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.109 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:41.109 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.711106+0000", 2024-04-19T01:04:41.109 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.5757000000000002e-05 2024-04-19T01:04:41.109 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:41.109 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:41.110 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:41.110 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.110 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.110 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:331 3.ds0 3:bf91dcb5:::smithi15528762-2:head [append 0~507904] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:04:41.110 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.680101+0000", 2024-04-19T01:04:41.110 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 23.321582712000001, 2024-04-19T01:04:41.110 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.031204012, 2024-04-19T01:04:41.110 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:41.110 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:41.110 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:41.110 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:41.110 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:41.110 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:41.110 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 331 2024-04-19T01:04:41.111 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.111 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:41.111 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.111 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:41.111 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680101+0000", 2024-04-19T01:04:41.111 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:41.111 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.111 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.111 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:41.111 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680101+0000", 2024-04-19T01:04:41.111 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:41.111 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.111 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.111 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:41.112 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680103+0000", 2024-04-19T01:04:41.112 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.956e-06 2024-04-19T01:04:41.112 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.112 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.112 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:41.112 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680444+0000", 2024-04-19T01:04:41.112 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00034149199999999999 2024-04-19T01:04:41.112 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.112 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.112 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:41.112 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680460+0000", 2024-04-19T01:04:41.112 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5554e-05 2024-04-19T01:04:41.112 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.112 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.112 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:41.113 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680471+0000", 2024-04-19T01:04:41.113 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0852e-05 2024-04-19T01:04:41.113 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.113 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.113 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:41.113 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680539+0000", 2024-04-19T01:04:41.113 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.8619000000000001e-05 2024-04-19T01:04:41.113 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.113 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.113 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:41.113 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680652+0000", 2024-04-19T01:04:41.113 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011258499999999999 2024-04-19T01:04:41.113 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.113 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.113 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:41.114 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.681392+0000", 2024-04-19T01:04:41.114 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00074039899999999998 2024-04-19T01:04:41.114 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.114 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.114 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:41.114 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.691230+0000", 2024-04-19T01:04:41.114 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0098373260000000004 2024-04-19T01:04:41.114 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.114 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.114 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:41.114 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.711264+0000", 2024-04-19T01:04:41.115 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.020034501999999999 2024-04-19T01:04:41.115 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.115 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.115 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:41.115 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.711305+0000", 2024-04-19T01:04:41.115 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0726999999999998e-05 2024-04-19T01:04:41.115 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:41.115 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:41.115 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:41.115 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.116 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.116 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:332 3.ds0 3:bf91dcb5:::smithi15528762-2:head [append 0~262144] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:04:41.116 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.682638+0000", 2024-04-19T01:04:41.116 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 23.319045608, 2024-04-19T01:04:41.116 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.029333649999999999, 2024-04-19T01:04:41.116 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:41.116 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:41.116 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:41.116 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:41.116 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:41.116 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:41.116 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 332 2024-04-19T01:04:41.116 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.116 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:41.117 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.117 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:41.123 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682638+0000", 2024-04-19T01:04:41.123 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:41.123 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.123 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.124 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:41.124 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682638+0000", 2024-04-19T01:04:41.124 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:41.124 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.124 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.124 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:41.125 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682640+0000", 2024-04-19T01:04:41.125 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5990000000000001e-06 2024-04-19T01:04:41.125 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.125 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.126 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:41.126 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682770+0000", 2024-04-19T01:04:41.126 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013076200000000001 2024-04-19T01:04:41.126 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.127 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.127 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:41.127 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682784+0000", 2024-04-19T01:04:41.127 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3786e-05 2024-04-19T01:04:41.127 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.128 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.128 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:41.128 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682792+0000", 2024-04-19T01:04:41.128 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.3000000000000002e-06 2024-04-19T01:04:41.128 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.128 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.129 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:41.129 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.686722+0000", 2024-04-19T01:04:41.129 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0039299749999999996 2024-04-19T01:04:41.129 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.129 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.129 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:41.129 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.686804+0000", 2024-04-19T01:04:41.129 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1588000000000006e-05 2024-04-19T01:04:41.129 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.129 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.129 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:41.129 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.687521+0000", 2024-04-19T01:04:41.129 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00071744300000000001 2024-04-19T01:04:41.129 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.130 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.130 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:41.130 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.691494+0000", 2024-04-19T01:04:41.130 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0039721239999999996 2024-04-19T01:04:41.130 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.130 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.130 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:41.130 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.711934+0000", 2024-04-19T01:04:41.130 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.020440649000000002 2024-04-19T01:04:41.130 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.130 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.130 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:41.130 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.711972+0000", 2024-04-19T01:04:41.130 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7423999999999999e-05 2024-04-19T01:04:41.131 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:41.131 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:41.131 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:41.131 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.131 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.131 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:333 3.ds0 3:bf91dcb5:::smithi15528762-2:head [setxattr _header (55) in=7b] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:04:41.131 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.682805+0000", 2024-04-19T01:04:41.131 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 23.318878207000001, 2024-04-19T01:04:41.131 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.029376039, 2024-04-19T01:04:41.131 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:41.131 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:41.131 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:41.131 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:41.132 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:41.132 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:41.132 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 333 2024-04-19T01:04:41.132 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.132 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:41.132 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.132 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:41.132 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682805+0000", 2024-04-19T01:04:41.132 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:41.133 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.133 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.133 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:41.133 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682805+0000", 2024-04-19T01:04:41.133 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:41.133 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.133 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.133 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:41.133 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682806+0000", 2024-04-19T01:04:41.133 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4099999999999999e-07 2024-04-19T01:04:41.134 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.134 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.134 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:41.134 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682815+0000", 2024-04-19T01:04:41.134 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.516e-06 2024-04-19T01:04:41.134 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.134 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.134 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:41.134 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682823+0000", 2024-04-19T01:04:41.134 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6459999999999993e-06 2024-04-19T01:04:41.135 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.135 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.135 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:41.135 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682832+0000", 2024-04-19T01:04:41.135 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.6260000000000001e-06 2024-04-19T01:04:41.135 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.135 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.135 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:41.135 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.690350+0000", 2024-04-19T01:04:41.135 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0075183730000000001 2024-04-19T01:04:41.136 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.136 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.136 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:41.136 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.690456+0000", 2024-04-19T01:04:41.136 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010617599999999999 2024-04-19T01:04:41.136 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.136 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.136 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:41.136 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.690951+0000", 2024-04-19T01:04:41.136 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00049519699999999995 2024-04-19T01:04:41.136 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.136 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.136 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:41.136 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.693060+0000", 2024-04-19T01:04:41.136 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0021084239999999998 2024-04-19T01:04:41.137 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.137 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.137 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:41.137 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.712088+0000", 2024-04-19T01:04:41.137 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.019028024000000001 2024-04-19T01:04:41.137 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.137 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.137 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:41.137 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.712181+0000", 2024-04-19T01:04:41.137 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.3516000000000004e-05 2024-04-19T01:04:41.137 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:41.137 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:41.137 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:41.137 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.138 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.138 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:334 3.ds0 3:bf91dcb5:::smithi15528762-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:04:41.138 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.682840+0000", 2024-04-19T01:04:41.138 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 23.318844058, 2024-04-19T01:04:41.138 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.031162179000000002, 2024-04-19T01:04:41.138 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:41.138 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:41.138 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:41.138 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:41.138 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:41.138 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:41.138 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 334 2024-04-19T01:04:41.138 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.138 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:41.139 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.139 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:41.139 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682840+0000", 2024-04-19T01:04:41.139 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:41.139 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.139 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.139 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:41.139 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682840+0000", 2024-04-19T01:04:41.139 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:41.139 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.139 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.139 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:41.139 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682840+0000", 2024-04-19T01:04:41.139 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4899999999999995e-07 2024-04-19T01:04:41.139 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.140 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.140 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:41.140 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682848+0000", 2024-04-19T01:04:41.140 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5270000000000001e-06 2024-04-19T01:04:41.140 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.140 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.140 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:41.140 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682855+0000", 2024-04-19T01:04:41.140 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.4889999999999998e-06 2024-04-19T01:04:41.140 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.140 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.140 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:41.140 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682860+0000", 2024-04-19T01:04:41.140 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.164e-06 2024-04-19T01:04:41.140 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.141 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.141 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:41.141 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.691525+0000", 2024-04-19T01:04:41.141 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0086643239999999993 2024-04-19T01:04:41.141 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.141 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.141 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:04:41.141 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.691618+0000", 2024-04-19T01:04:41.141 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2983000000000001e-05 2024-04-19T01:04:41.141 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.141 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.141 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:41.141 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.712661+0000", 2024-04-19T01:04:41.141 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021043777 2024-04-19T01:04:41.142 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.142 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.142 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:41.142 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.712733+0000", 2024-04-19T01:04:41.142 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2063000000000004e-05 2024-04-19T01:04:41.142 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.142 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.142 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:41.142 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.714002+0000", 2024-04-19T01:04:41.142 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0012683029999999999 2024-04-19T01:04:41.142 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:41.142 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:41.142 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:41.142 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.143 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.143 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:354 3.as0 3:56c32ee7:::smithi15528762-47:head [copy-from ver 35] snapc 9=[8,7] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:04:41.143 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.544030+0000", 2024-04-19T01:04:41.143 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 22.457654031000001, 2024-04-19T01:04:41.143 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.049063250000000003, 2024-04-19T01:04:41.143 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:41.143 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:41.143 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:41.143 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:41.144 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:41.144 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:41.144 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 354 2024-04-19T01:04:41.144 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.144 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:41.144 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.144 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:41.144 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544030+0000", 2024-04-19T01:04:41.144 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:41.144 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.144 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.144 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:41.144 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544030+0000", 2024-04-19T01:04:41.145 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:41.145 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.145 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.145 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:41.145 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544033+0000", 2024-04-19T01:04:41.145 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.506e-06 2024-04-19T01:04:41.145 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.145 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.145 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:41.145 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544046+0000", 2024-04-19T01:04:41.145 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3208e-05 2024-04-19T01:04:41.145 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.145 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.145 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:41.146 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544082+0000", 2024-04-19T01:04:41.146 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5509999999999997e-05 2024-04-19T01:04:41.146 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.146 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.146 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:41.146 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544101+0000", 2024-04-19T01:04:41.146 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9500999999999998e-05 2024-04-19T01:04:41.146 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.146 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.146 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:41.146 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544230+0000", 2024-04-19T01:04:41.146 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000129114 2024-04-19T01:04:41.146 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.146 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.147 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:41.147 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544428+0000", 2024-04-19T01:04:41.147 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00019776799999999999 2024-04-19T01:04:41.147 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.147 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.147 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:41.147 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.556395+0000", 2024-04-19T01:04:41.147 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011967261 2024-04-19T01:04:41.147 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.147 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.147 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:41.147 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.574011+0000", 2024-04-19T01:04:41.147 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017615654000000001 2024-04-19T01:04:41.147 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.147 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.148 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:41.148 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.592920+0000", 2024-04-19T01:04:41.148 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018908812000000001 2024-04-19T01:04:41.148 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.148 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.148 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:41.148 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.593093+0000", 2024-04-19T01:04:41.148 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00017291600000000001 2024-04-19T01:04:41.148 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:41.148 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:41.148 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:41.148 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.148 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.148 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:355 3.as0 3:56c32ee7:::smithi15528762-47:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:04:41.148 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.544150+0000", 2024-04-19T01:04:41.149 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 22.457533259000002, 2024-04-19T01:04:41.149 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.029722035000000001, 2024-04-19T01:04:41.149 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:41.149 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:41.149 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:41.149 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:41.149 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:41.149 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:41.149 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 355 2024-04-19T01:04:41.149 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.149 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:41.149 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.149 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:41.149 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544150+0000", 2024-04-19T01:04:41.149 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:41.150 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.150 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.150 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:41.150 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544150+0000", 2024-04-19T01:04:41.150 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:41.150 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.150 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.150 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:41.150 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544152+0000", 2024-04-19T01:04:41.150 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.28e-06 2024-04-19T01:04:41.150 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.150 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.150 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:41.151 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544158+0000", 2024-04-19T01:04:41.151 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4160000000000001e-06 2024-04-19T01:04:41.151 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.151 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.151 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:41.151 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544178+0000", 2024-04-19T01:04:41.151 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0054000000000001e-05 2024-04-19T01:04:41.151 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.151 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.151 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:41.151 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544195+0000", 2024-04-19T01:04:41.151 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6835999999999999e-05 2024-04-19T01:04:41.151 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.151 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.151 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:41.152 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544717+0000", 2024-04-19T01:04:41.152 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00052258800000000005 2024-04-19T01:04:41.152 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.152 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.152 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for blocked object", 2024-04-19T01:04:41.152 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544835+0000", 2024-04-19T01:04:41.152 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000117448 2024-04-19T01:04:41.152 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.152 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.152 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:41.152 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.573609+0000", 2024-04-19T01:04:41.152 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.028774094 2024-04-19T01:04:41.152 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.152 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.152 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:41.153 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.573721+0000", 2024-04-19T01:04:41.153 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011205900000000001 2024-04-19T01:04:41.153 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.153 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.153 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:41.153 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.573872+0000", 2024-04-19T01:04:41.153 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015126000000000001 2024-04-19T01:04:41.153 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:41.153 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:41.153 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:41.153 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.153 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.153 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:478 3.as0 3:5a488dbb:::smithi15528762-13:head [append 0~770048] snapc 12=[12,11,10] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-19T01:04:41.153 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:35.631704+0000", 2024-04-19T01:04:41.154 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.3699800309999999, 2024-04-19T01:04:41.154 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.031533347000000003, 2024-04-19T01:04:41.154 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:41.154 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:41.154 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:41.154 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:41.154 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:41.154 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:41.154 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 478 2024-04-19T01:04:41.154 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.154 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:41.154 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.154 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:41.154 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.631704+0000", 2024-04-19T01:04:41.155 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:41.155 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.155 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.155 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:41.155 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.631704+0000", 2024-04-19T01:04:41.155 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:41.155 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.155 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.155 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:41.155 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.631707+0000", 2024-04-19T01:04:41.155 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0800000000000002e-06 2024-04-19T01:04:41.155 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.155 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.155 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:41.155 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.632909+0000", 2024-04-19T01:04:41.156 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0012019070000000001 2024-04-19T01:04:41.156 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.156 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.156 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:41.156 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.632934+0000", 2024-04-19T01:04:41.156 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5289000000000001e-05 2024-04-19T01:04:41.156 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.156 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.156 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:41.156 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.632945+0000", 2024-04-19T01:04:41.156 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1046999999999999e-05 2024-04-19T01:04:41.156 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.156 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.156 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:41.156 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.633050+0000", 2024-04-19T01:04:41.157 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010516 2024-04-19T01:04:41.157 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.157 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.157 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op must wait for map", 2024-04-19T01:04:41.157 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.633065+0000", 2024-04-19T01:04:41.157 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4765e-05 2024-04-19T01:04:41.157 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.157 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.157 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:41.157 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.636005+0000", 2024-04-19T01:04:41.157 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0029403530000000002 2024-04-19T01:04:41.157 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.157 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.157 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:41.158 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.636188+0000", 2024-04-19T01:04:41.158 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00018290099999999999 2024-04-19T01:04:41.158 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.158 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.158 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:41.158 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.638553+0000", 2024-04-19T01:04:41.158 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0023651800000000001 2024-04-19T01:04:41.158 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.158 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.158 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:41.158 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.662256+0000", 2024-04-19T01:04:41.158 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.023702524999999999 2024-04-19T01:04:41.158 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.158 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.159 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:41.159 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.662993+0000", 2024-04-19T01:04:41.159 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00073735000000000001 2024-04-19T01:04:41.159 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.159 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.159 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:41.159 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.663237+0000", 2024-04-19T01:04:41.159 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00024379 2024-04-19T01:04:41.159 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:41.159 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:41.159 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:41.159 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.159 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.159 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:479 3.as0 3:5a488dbb:::smithi15528762-13:head [append 0~475136] snapc 12=[12,11,10] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-19T01:04:41.159 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:35.633850+0000", 2024-04-19T01:04:41.160 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.3678337730000001, 2024-04-19T01:04:41.160 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.042497337000000003, 2024-04-19T01:04:41.160 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:41.160 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:41.160 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:41.160 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:41.160 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:41.160 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:41.160 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 479 2024-04-19T01:04:41.160 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.160 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:41.160 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.160 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:41.160 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.633850+0000", 2024-04-19T01:04:41.160 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:41.161 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.161 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.161 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:41.161 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.633850+0000", 2024-04-19T01:04:41.161 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:41.161 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.161 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.161 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:41.161 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.633852+0000", 2024-04-19T01:04:41.161 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.26e-06 2024-04-19T01:04:41.161 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.161 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.161 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:41.161 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.634421+0000", 2024-04-19T01:04:41.161 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000569058 2024-04-19T01:04:41.162 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.162 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.162 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:41.162 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.634441+0000", 2024-04-19T01:04:41.162 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0169e-05 2024-04-19T01:04:41.162 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.162 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.162 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:41.162 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.634452+0000", 2024-04-19T01:04:41.162 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0963e-05 2024-04-19T01:04:41.162 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.162 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.162 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:41.162 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.634540+0000", 2024-04-19T01:04:41.163 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.7563000000000002e-05 2024-04-19T01:04:41.163 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.163 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.163 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:04:41.163 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.634550+0000", 2024-04-19T01:04:41.163 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0491e-05 2024-04-19T01:04:41.163 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.163 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.163 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:41.163 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.653619+0000", 2024-04-19T01:04:41.163 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.019068284000000001 2024-04-19T01:04:41.163 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.163 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.164 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:41.164 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.653716+0000", 2024-04-19T01:04:41.164 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7546000000000004e-05 2024-04-19T01:04:41.164 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.164 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.164 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:41.164 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.654525+0000", 2024-04-19T01:04:41.164 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00080861300000000002 2024-04-19T01:04:41.164 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.164 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.164 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:41.164 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.674163+0000", 2024-04-19T01:04:41.164 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.019638666999999999 2024-04-19T01:04:41.164 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.164 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.164 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:41.165 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.676228+0000", 2024-04-19T01:04:41.165 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0020650379999999999 2024-04-19T01:04:41.165 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.165 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.165 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:41.165 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.676347+0000", 2024-04-19T01:04:41.165 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011868500000000001 2024-04-19T01:04:41.165 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:41.165 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:41.165 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:41.165 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.165 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.165 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:480 3.as0 3:5a488dbb:::smithi15528762-13:head [append 0~294912] snapc 12=[12,11,10] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-19T01:04:41.165 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:35.639539+0000", 2024-04-19T01:04:41.166 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.3621441389999998, 2024-04-19T01:04:41.166 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.038046241000000001, 2024-04-19T01:04:41.166 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:41.166 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:41.166 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:41.166 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:41.166 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:41.166 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:41.166 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 480 2024-04-19T01:04:41.166 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.166 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:41.166 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.166 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:41.166 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639539+0000", 2024-04-19T01:04:41.166 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:41.167 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.167 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.167 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:41.167 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639539+0000", 2024-04-19T01:04:41.167 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:41.167 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.167 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.167 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:41.167 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639541+0000", 2024-04-19T01:04:41.167 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.922e-06 2024-04-19T01:04:41.167 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.167 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.167 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:41.167 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639847+0000", 2024-04-19T01:04:41.167 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00030593599999999998 2024-04-19T01:04:41.168 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.168 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.168 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:41.168 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639886+0000", 2024-04-19T01:04:41.168 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9171000000000002e-05 2024-04-19T01:04:41.168 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.168 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.168 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:41.168 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639898+0000", 2024-04-19T01:04:41.168 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1103e-05 2024-04-19T01:04:41.168 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.168 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.168 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:41.168 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.663566+0000", 2024-04-19T01:04:41.168 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.023668392 2024-04-19T01:04:41.169 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.169 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.169 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:41.169 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.663853+0000", 2024-04-19T01:04:41.169 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00028731200000000002 2024-04-19T01:04:41.169 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.169 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.169 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:41.169 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.665713+0000", 2024-04-19T01:04:41.169 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001860109 2024-04-19T01:04:41.169 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.169 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.169 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:41.169 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.675595+0000", 2024-04-19T01:04:41.169 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0098813939999999999 2024-04-19T01:04:41.170 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.170 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.170 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:41.170 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.677466+0000", 2024-04-19T01:04:41.170 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0018710199999999999 2024-04-19T01:04:41.170 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.170 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.170 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:41.170 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.677586+0000", 2024-04-19T01:04:41.170 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000119882 2024-04-19T01:04:41.170 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:41.170 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:41.170 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:41.170 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.171 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.171 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:481 3.as0 3:5a488dbb:::smithi15528762-13:head [setxattr _header (57) in=7b] snapc 12=[12,11,10] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-19T01:04:41.171 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:35.639915+0000", 2024-04-19T01:04:41.171 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.3617689180000001, 2024-04-19T01:04:41.171 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.038026319000000003, 2024-04-19T01:04:41.171 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:41.171 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:41.171 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:41.171 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:41.171 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:41.171 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:41.171 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 481 2024-04-19T01:04:41.171 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.171 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:41.171 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.172 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:41.172 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639915+0000", 2024-04-19T01:04:41.172 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:41.172 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.172 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.172 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:41.172 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639915+0000", 2024-04-19T01:04:41.172 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:41.172 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.172 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.172 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:41.172 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639915+0000", 2024-04-19T01:04:41.172 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.0800000000000004e-07 2024-04-19T01:04:41.172 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.173 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.173 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:41.173 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639919+0000", 2024-04-19T01:04:41.173 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0589999999999996e-06 2024-04-19T01:04:41.173 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.173 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.173 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:41.173 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639929+0000", 2024-04-19T01:04:41.173 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.9590000000000007e-06 2024-04-19T01:04:41.173 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.173 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.173 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:41.173 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639936+0000", 2024-04-19T01:04:41.173 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2480000000000001e-06 2024-04-19T01:04:41.174 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.174 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.174 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:41.174 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.674290+0000", 2024-04-19T01:04:41.174 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.034354279000000001 2024-04-19T01:04:41.174 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.174 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.174 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:41.174 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.674404+0000", 2024-04-19T01:04:41.174 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000114189 2024-04-19T01:04:41.174 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.174 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.174 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:41.174 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.675012+0000", 2024-04-19T01:04:41.175 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00060776199999999997 2024-04-19T01:04:41.175 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.175 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.175 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:41.175 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.676365+0000", 2024-04-19T01:04:41.175 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0013529849999999999 2024-04-19T01:04:41.175 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.175 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.175 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:41.175 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.677805+0000", 2024-04-19T01:04:41.175 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014399160000000001 2024-04-19T01:04:41.175 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.175 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.175 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:41.175 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.677941+0000", 2024-04-19T01:04:41.176 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013621400000000001 2024-04-19T01:04:41.176 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:41.176 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:41.176 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:41.176 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.176 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.176 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:482 3.as0 3:5a488dbb:::smithi15528762-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-04-19T01:04:41.176 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:35.639943+0000", 2024-04-19T01:04:41.176 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.3617407200000002, 2024-04-19T01:04:41.176 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.040997591999999999, 2024-04-19T01:04:41.176 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:41.176 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:41.176 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:41.176 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:41.176 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:41.177 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:41.177 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 482 2024-04-19T01:04:41.177 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.177 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:41.177 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.177 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:41.177 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639943+0000", 2024-04-19T01:04:41.177 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:41.177 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.177 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.177 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:41.177 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639943+0000", 2024-04-19T01:04:41.177 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:41.177 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.177 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.178 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:41.178 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639943+0000", 2024-04-19T01:04:41.178 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.89e-07 2024-04-19T01:04:41.178 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.178 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.178 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:41.178 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639945+0000", 2024-04-19T01:04:41.178 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9870000000000002e-06 2024-04-19T01:04:41.178 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.178 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.178 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:41.178 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639951+0000", 2024-04-19T01:04:41.178 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.9479999999999996e-06 2024-04-19T01:04:41.178 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.178 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.179 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:41.179 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639956+0000", 2024-04-19T01:04:41.179 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8040000000000001e-06 2024-04-19T01:04:41.179 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.179 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.179 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:41.179 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.676400+0000", 2024-04-19T01:04:41.179 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.036444184999999997 2024-04-19T01:04:41.179 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.179 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.179 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:04:41.179 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.676473+0000", 2024-04-19T01:04:41.179 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2704000000000001e-05 2024-04-19T01:04:41.179 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.179 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.180 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:41.180 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.678615+0000", 2024-04-19T01:04:41.180 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0021422030000000001 2024-04-19T01:04:41.180 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.180 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.180 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:41.180 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.678811+0000", 2024-04-19T01:04:41.180 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000196026 2024-04-19T01:04:41.180 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.180 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.180 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:41.180 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.680940+0000", 2024-04-19T01:04:41.180 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0021293459999999998 2024-04-19T01:04:41.180 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:41.180 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:41.181 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:41.181 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.181 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.181 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.as0 65/31 ECSubWriteReply(tid=107, last_complete=65'26, committed=1, applied=1))", 2024-04-19T01:04:41.181 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:35.650186+0000", 2024-04-19T01:04:41.181 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.3514975219999998, 2024-04-19T01:04:41.181 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012079777999999999, 2024-04-19T01:04:41.181 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:41.181 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:41.181 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:41.181 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:41.182 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.182 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:41.182 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.650186+0000", 2024-04-19T01:04:41.182 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:41.182 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.182 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.182 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:41.182 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.650186+0000", 2024-04-19T01:04:41.182 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:41.182 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.182 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.182 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:41.183 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.650187+0000", 2024-04-19T01:04:41.183 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2440000000000001e-06 2024-04-19T01:04:41.183 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.183 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.183 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:41.183 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.650194+0000", 2024-04-19T01:04:41.183 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.6159999999999998e-06 2024-04-19T01:04:41.183 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.183 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.183 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:41.183 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.650210+0000", 2024-04-19T01:04:41.183 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6432000000000001e-05 2024-04-19T01:04:41.183 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.183 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.184 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:41.184 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.650222+0000", 2024-04-19T01:04:41.184 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1429e-05 2024-04-19T01:04:41.184 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.184 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.184 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:41.184 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.662033+0000", 2024-04-19T01:04:41.184 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011811210000000001 2024-04-19T01:04:41.184 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.184 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.184 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:41.184 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.662266+0000", 2024-04-19T01:04:41.184 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000232847 2024-04-19T01:04:41.184 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:41.185 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:41.185 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:41.185 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.185 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.185 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.as0 65/31 ECSubWriteReply(tid=107, last_complete=65'26, committed=1, applied=1))", 2024-04-19T01:04:41.185 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:35.650469+0000", 2024-04-19T01:04:41.185 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.3512144859999999, 2024-04-19T01:04:41.185 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011774705, 2024-04-19T01:04:41.185 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:41.185 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:41.185 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:41.185 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:41.185 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.185 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:41.185 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.650469+0000", 2024-04-19T01:04:41.186 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:41.186 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.186 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.186 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:41.186 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.650469+0000", 2024-04-19T01:04:41.186 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:41.186 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.186 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.186 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:41.186 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.650470+0000", 2024-04-19T01:04:41.186 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.141e-06 2024-04-19T01:04:41.186 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.186 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.186 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:41.187 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.650476+0000", 2024-04-19T01:04:41.187 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2469999999999997e-06 2024-04-19T01:04:41.187 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.187 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.187 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:41.187 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.650493+0000", 2024-04-19T01:04:41.187 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6422e-05 2024-04-19T01:04:41.187 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.187 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.187 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:41.187 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.650501+0000", 2024-04-19T01:04:41.187 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.2849999999999995e-06 2024-04-19T01:04:41.187 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.187 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.187 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:41.187 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.662180+0000", 2024-04-19T01:04:41.188 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011679314999999999 2024-04-19T01:04:41.188 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.188 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.188 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:41.188 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.662244+0000", 2024-04-19T01:04:41.188 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3294999999999996e-05 2024-04-19T01:04:41.188 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:41.188 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:41.188 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:41.188 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.188 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.188 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.as0 65/31 ECSubWriteReply(tid=108, last_complete=65'27, committed=1, applied=1))", 2024-04-19T01:04:41.188 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:35.664806+0000", 2024-04-19T01:04:41.188 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.3368778260000003, 2024-04-19T01:04:41.189 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010692602000000001, 2024-04-19T01:04:41.189 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:41.189 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:41.189 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:41.189 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:41.189 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.189 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:41.189 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.664806+0000", 2024-04-19T01:04:41.189 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:41.189 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.189 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.189 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:41.189 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.664806+0000", 2024-04-19T01:04:41.189 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:41.189 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.189 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.190 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:41.190 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.664807+0000", 2024-04-19T01:04:41.190 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.64e-06 2024-04-19T01:04:41.190 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.190 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.190 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:41.190 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.664819+0000", 2024-04-19T01:04:41.190 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1202999999999999e-05 2024-04-19T01:04:41.190 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.190 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.190 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:41.190 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.664853+0000", 2024-04-19T01:04:41.190 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4375000000000002e-05 2024-04-19T01:04:41.190 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.191 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.191 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:41.191 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.664876+0000", 2024-04-19T01:04:41.191 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3008999999999999e-05 2024-04-19T01:04:41.191 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.191 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.191 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:41.191 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.675445+0000", 2024-04-19T01:04:41.191 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010569423999999999 2024-04-19T01:04:41.191 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:41.191 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:41.191 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:41.191 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.675498+0000", 2024-04-19T01:04:41.191 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2951000000000002e-05 2024-04-19T01:04:41.191 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:41.192 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:41.192 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:41.192 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:41.192 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:41.192 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:41.192 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-19T01:04:41.194 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:04:41.194+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:04:41.194 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-19T01:04:41.294 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:04:41.293+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:04:41.325 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:41.326 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [], 2024-04-19T01:04:41.326 INFO:teuthology.orchestra.run.smithi110.stdout: "num_ops": 0 2024-04-19T01:04:41.326 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:41.339 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-19T01:04:41.392 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-04-19T01:04:41.393 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T01:04:41.395 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:04:41.394+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:04:41.411 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-04-19T01:04:41.411 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T01:04:41.432 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-04-19T01:04:41.433 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T01:04:41.456 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-04-19T01:04:41.456 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T01:04:41.460 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:41.460 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:41.474 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-19T01:04:41.475 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-04-19T01:04:41.475 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T01:04:41.495 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-04-19T01:04:41.495 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T01:04:41.496 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:04:41.497+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:04:41.503 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2024-04-19T01:04:41.503 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.4 to 1 (10000) 2024-04-19T01:04:41.510 INFO:tasks.rados.rados.0.smithi155.stdout:213: done (0 left) 2024-04-19T01:04:41.510 INFO:tasks.rados.rados.0.smithi155.stdout:214: copy_from oid 15 from oid 2 current snap is 21 2024-04-19T01:04:41.510 INFO:tasks.rados.rados.0.smithi155.stdout:215: copy_from oid 12 from oid 7 current snap is 21 2024-04-19T01:04:41.510 INFO:tasks.rados.rados.0.smithi155.stdout:216: copy_from oid 13 from oid 9 current snap is 21 2024-04-19T01:04:41.510 INFO:tasks.rados.rados.0.smithi155.stdout:217: rmattr oid 46 current snap is 21 2024-04-19T01:04:41.510 INFO:tasks.rados.rados.0.smithi155.stdout:217: done (3 left) 2024-04-19T01:04:41.510 INFO:tasks.rados.rados.0.smithi155.stdout:218: snap_remove snap 20 2024-04-19T01:04:41.525 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-04-19T01:04:41.526 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T01:04:41.535 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-04-19T01:04:41.563 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-04-19T01:04:41.564 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T01:04:41.582 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2024-04-19T01:04:41.582 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T01:04:41.597 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:04:41.594+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:04:41.601 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2024-04-19T01:04:41.601 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T01:04:41.622 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2024-04-19T01:04:41.623 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T01:04:41.640 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2024-04-19T01:04:41.641 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T01:04:41.651 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:41.651 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:41.657 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2024-04-19T01:04:41.657 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-19T01:04:41.665 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-19T01:04:41.675 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2024-04-19T01:04:41.676 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-19T01:04:41.696 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2024-04-19T01:04:41.696 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-19T01:04:41.708 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:04:41.706+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:04:41.716 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2024-04-19T01:04:41.717 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-19T01:04:41.799 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:04:41.801+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:04:41.825 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:41.825 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:41.839 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-19T01:04:41.900 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:04:41.898+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:04:41.961 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:41.961 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:41.976 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-19T01:04:42.001 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:04:42.000+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:04:42.109 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:04:42.105+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:04:42.140 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:42.140 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:42.154 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-19T01:04:42.206 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:04:42.208+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:04:42.276 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:42.276 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:42.290 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-19T01:04:42.307 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:04:42.305+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:04:42.407 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:42.408 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:42.408 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:04:42.406+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:04:42.421 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-19T01:04:42.508 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:42.506+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:42.513 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.5 to 1 (10000) 2024-04-19T01:04:42.518 INFO:tasks.rados.rados.0.smithi155.stdout:218: done (3 left) 2024-04-19T01:04:42.519 INFO:tasks.rados.rados.0.smithi155.stdout:219: read oid 43 snap -1 2024-04-19T01:04:42.519 INFO:tasks.rados.rados.0.smithi155.stdout:219: expect (ObjNum 12 snap 0 seq_num 12) 2024-04-19T01:04:42.519 INFO:tasks.rados.rados.0.smithi155.stdout:220: delete oid 11 current snap is 21 2024-04-19T01:04:42.519 INFO:tasks.rados.rados.0.smithi155.stdout:216: finishing copy_from racing read to smithi15528762-13 2024-04-19T01:04:42.519 INFO:tasks.rados.rados.0.smithi155.stdout:216: finishing copy_from to smithi15528762-13 2024-04-19T01:04:42.519 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 13 v 2 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-04-19T01:04:42.519 INFO:tasks.rados.rados.0.smithi155.stdout:215: finishing copy_from racing read to smithi15528762-12 2024-04-19T01:04:42.519 INFO:tasks.rados.rados.0.smithi155.stdout:214: finishing copy_from racing read to smithi15528762-15 2024-04-19T01:04:42.519 INFO:tasks.rados.rados.0.smithi155.stdout:215: finishing copy_from to smithi15528762-12 2024-04-19T01:04:42.519 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 12 v 38 (ObjNum 95 snap 19 seq_num 95) dirty exists 2024-04-19T01:04:42.519 INFO:tasks.rados.rados.0.smithi155.stdout:214: finishing copy_from to smithi15528762-15 2024-04-19T01:04:42.519 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 15 v 52 (ObjNum 86 snap 15 seq_num 86) dirty exists 2024-04-19T01:04:42.525 INFO:tasks.rados.rados.0.smithi155.stdout:219: expect (ObjNum 12 snap 0 seq_num 12) 2024-04-19T01:04:42.540 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-04-19T01:04:42.547 INFO:tasks.rados.rados.0.smithi155.stdout:214: done (4 left) 2024-04-19T01:04:42.547 INFO:tasks.rados.rados.0.smithi155.stdout:215: done (3 left) 2024-04-19T01:04:42.547 INFO:tasks.rados.rados.0.smithi155.stdout:216: done (2 left) 2024-04-19T01:04:42.547 INFO:tasks.rados.rados.0.smithi155.stdout:219: done (1 left) 2024-04-19T01:04:42.547 INFO:tasks.rados.rados.0.smithi155.stdout:220: done (0 left) 2024-04-19T01:04:42.547 INFO:tasks.rados.rados.0.smithi155.stdout:221: delete oid 21 current snap is 21 2024-04-19T01:04:42.557 INFO:tasks.rados.rados.0.smithi155.stdout:221: done (0 left) 2024-04-19T01:04:42.557 INFO:tasks.rados.rados.0.smithi155.stdout:222: delete oid 13 current snap is 21 2024-04-19T01:04:42.560 INFO:tasks.rados.rados.0.smithi155.stdout:222: done (0 left) 2024-04-19T01:04:42.561 INFO:tasks.rados.rados.0.smithi155.stdout:223: delete oid 37 current snap is 21 2024-04-19T01:04:42.568 INFO:tasks.rados.rados.0.smithi155.stdout:223: done (0 left) 2024-04-19T01:04:42.568 INFO:tasks.rados.rados.0.smithi155.stdout:224: read oid 11 snap 19 2024-04-19T01:04:42.568 INFO:tasks.rados.rados.0.smithi155.stdout:224: expect (ObjNum 10 snap 0 seq_num 10) 2024-04-19T01:04:42.568 INFO:tasks.rados.rados.0.smithi155.stdout:225: read oid 7 snap 17 2024-04-19T01:04:42.569 INFO:tasks.rados.rados.0.smithi155.stdout:225: expect (ObjNum 80 snap 13 seq_num 80) 2024-04-19T01:04:42.569 INFO:tasks.rados.rados.0.smithi155.stdout:226: read oid 19 snap -1 2024-04-19T01:04:42.569 INFO:tasks.rados.rados.0.smithi155.stdout:226: expect (ObjNum 56 snap 5 seq_num 56) 2024-04-19T01:04:42.569 INFO:tasks.rados.rados.0.smithi155.stdout:227: snap_create 2024-04-19T01:04:42.577 INFO:tasks.rados.rados.0.smithi155.stdout:224: expect (ObjNum 10 snap 0 seq_num 10) 2024-04-19T01:04:42.591 INFO:tasks.rados.rados.0.smithi155.stdout:226: expect (ObjNum 56 snap 5 seq_num 56) 2024-04-19T01:04:42.593 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:42.593 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:42.613 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-04-19T01:04:42.651 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:04:42.650+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:04:42.655 INFO:tasks.rados.rados.0.smithi155.stdout:225: expect (ObjNum 80 snap 13 seq_num 80) 2024-04-19T01:04:42.716 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:04:42.717+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:04:42.773 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:42.773 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:42.789 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-04-19T01:04:42.823 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:04:42.818+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:04:42.915 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:42.915 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:42.921 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:04:42.921+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:04:42.929 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-04-19T01:04:43.022 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:43.023+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:43.050 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:43.051 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:43.065 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-19T01:04:43.123 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:04:43.122+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:04:43.196 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:43.196 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:43.215 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-19T01:04:43.233 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:04:43.225+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:04:43.327 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:04:43.327+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:04:43.340 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:43.340 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:43.359 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-19T01:04:43.430 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:43.426+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:43.516 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.6 to 1 (10000) 2024-04-19T01:04:43.521 INFO:tasks.rados.rados.0.smithi155.stdout:224: done (3 left) 2024-04-19T01:04:43.521 INFO:tasks.rados.rados.0.smithi155.stdout:225: done (2 left) 2024-04-19T01:04:43.521 INFO:tasks.rados.rados.0.smithi155.stdout:226: done (1 left) 2024-04-19T01:04:43.521 INFO:tasks.rados.rados.0.smithi155.stdout:227: done (0 left) 2024-04-19T01:04:43.521 INFO:tasks.rados.rados.0.smithi155.stdout:228: snap_remove snap 17 2024-04-19T01:04:43.531 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:43.532 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:43.532 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:04:43.527+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:04:43.537 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-19T01:04:43.541 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-04-19T01:04:43.544 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-19T01:04:43.630 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:04:43.631+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:04:43.709 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:43.709 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:43.724 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-19T01:04:43.734 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-19T01:04:43.735 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:04:43.730+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:04:43.833 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:04:43.833+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:04:43.875 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-04-19T01:04:43.875 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T01:04:43.887 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:43.887 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:43.896 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-04-19T01:04:43.897 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T01:04:43.902 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-19T01:04:43.914 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-04-19T01:04:43.914 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T01:04:43.934 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:04:43.934+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:04:43.939 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-04-19T01:04:43.939 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T01:04:43.951 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-04-19T01:04:43.951 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T01:04:43.972 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-04-19T01:04:43.972 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T01:04:43.993 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-04-19T01:04:43.994 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T01:04:44.016 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-04-19T01:04:44.017 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T01:04:44.028 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:04:44.028 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-04-19T01:04:44.028 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-04-19T01:04:44.029 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-04-19T01:04:44.029 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.029 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:240 3.1s0 3:82ff4f48:::smithi15528762-41:head [read 0~688128,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:44.029 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.861180+0000", 2024-04-19T01:04:44.029 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 30.166359580000002, 2024-04-19T01:04:44.029 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.026860730999999999, 2024-04-19T01:04:44.030 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:44.030 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:44.030 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-04-19T01:04:44.030 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:04:44.030 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:04:44.030 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:44.031 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 240 2024-04-19T01:04:44.031 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.031 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:44.031 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.031 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:44.031 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861180+0000", 2024-04-19T01:04:44.032 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.032 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.032 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.032 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:44.032 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861180+0000", 2024-04-19T01:04:44.032 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.032 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.033 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.033 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:44.034 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861182+0000", 2024-04-19T01:04:44.034 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.654e-06 2024-04-19T01:04:44.036 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.036 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.037 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:44.037 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861212+0000", 2024-04-19T01:04:44.037 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0511999999999999e-05 2024-04-19T01:04:44.037 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.037 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.037 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:44.038 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861234+0000", 2024-04-19T01:04:44.038 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2135999999999999e-05 2024-04-19T01:04:44.038 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.039 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.040 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:44.040 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861250+0000", 2024-04-19T01:04:44.040 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6141999999999999e-05 2024-04-19T01:04:44.041 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.041 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.041 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.041 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.865146+0000", 2024-04-19T01:04:44.041 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0038953350000000002 2024-04-19T01:04:44.041 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.041 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.042 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:04:44.042 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.865375+0000", 2024-04-19T01:04:44.042 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00022880900000000001 2024-04-19T01:04:44.042 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.042 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.042 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:44.043 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.888041+0000", 2024-04-19T01:04:44.043 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022666143 2024-04-19T01:04:44.043 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.044 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.044 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.044 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.045 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.045 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:241 3.1s0 3:82ff4f48:::smithi15528762-41:head [read 0~688128,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:44.045 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.861274+0000", 2024-04-19T01:04:44.045 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 30.166265732999999, 2024-04-19T01:04:44.045 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.027990521000000001, 2024-04-19T01:04:44.045 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:44.046 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:44.062 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-04-19T01:04:44.062 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:04:44.062 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:04:44.063 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:44.063 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 241 2024-04-19T01:04:44.063 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.063 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:44.063 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.063 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:44.064 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861274+0000", 2024-04-19T01:04:44.064 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.064 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.064 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.064 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:44.071 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861274+0000", 2024-04-19T01:04:44.071 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.071 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.071 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.071 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:44.072 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861275+0000", 2024-04-19T01:04:44.072 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3030000000000001e-06 2024-04-19T01:04:44.072 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.072 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.072 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:44.072 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861282+0000", 2024-04-19T01:04:44.073 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8569999999999997e-06 2024-04-19T01:04:44.073 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.073 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.073 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:44.073 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861300+0000", 2024-04-19T01:04:44.073 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7898999999999999e-05 2024-04-19T01:04:44.074 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.074 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.074 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:44.074 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861314+0000", 2024-04-19T01:04:44.074 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.364e-05 2024-04-19T01:04:44.074 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.074 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.075 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.075 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.868359+0000", 2024-04-19T01:04:44.085 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0070459019999999997 2024-04-19T01:04:44.085 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.086 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.086 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:04:44.086 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.868594+0000", 2024-04-19T01:04:44.086 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000234942 2024-04-19T01:04:44.086 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.086 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.086 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:44.086 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.889264+0000", 2024-04-19T01:04:44.086 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020669977999999999 2024-04-19T01:04:44.087 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.087 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.087 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.087 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s6 43/31 ECSubWrite(tid=67, reqid=osd.13.0:67, at_version=43'39, trim_to=0'0, roll_forward_to=41'37))", 2024-04-19T01:04:44.087 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.867247+0000", 2024-04-19T01:04:44.087 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 30.160292977000001, 2024-04-19T01:04:44.087 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019643094999999999, 2024-04-19T01:04:44.087 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:44.087 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:44.087 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:44.087 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:44.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.087 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:44.088 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.867247+0000", 2024-04-19T01:04:44.088 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.088 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.088 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.088 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:44.088 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.867247+0000", 2024-04-19T01:04:44.088 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.088 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.088 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.088 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:44.088 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.867247+0000", 2024-04-19T01:04:44.088 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7299999999999996e-07 2024-04-19T01:04:44.088 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.088 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.089 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:44.089 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.867324+0000", 2024-04-19T01:04:44.089 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6981e-05 2024-04-19T01:04:44.089 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.089 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.089 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:44.089 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.867357+0000", 2024-04-19T01:04:44.089 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3269000000000003e-05 2024-04-19T01:04:44.089 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.089 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.089 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:44.089 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.867380+0000", 2024-04-19T01:04:44.089 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2272999999999998e-05 2024-04-19T01:04:44.089 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.089 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.090 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.090 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.870949+0000", 2024-04-19T01:04:44.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0035693259999999998 2024-04-19T01:04:44.090 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.090 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.090 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:44.090 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.870973+0000", 2024-04-19T01:04:44.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3859e-05 2024-04-19T01:04:44.090 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.090 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.090 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:44.090 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.886787+0000", 2024-04-19T01:04:44.091 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0158142 2024-04-19T01:04:44.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.091 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.091 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:44.091 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.886890+0000", 2024-04-19T01:04:44.091 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000102614 2024-04-19T01:04:44.091 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.091 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.091 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.091 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.091 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s6 43/31 ECSubWrite(tid=69, reqid=osd.13.0:69, at_version=43'41, trim_to=0'0, roll_forward_to=41'37))", 2024-04-19T01:04:44.091 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.867419+0000", 2024-04-19T01:04:44.091 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 30.160121006000001, 2024-04-19T01:04:44.092 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019827727999999999, 2024-04-19T01:04:44.092 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:44.092 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:44.092 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:44.092 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:44.092 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.092 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:44.092 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.867419+0000", 2024-04-19T01:04:44.092 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.092 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.092 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.092 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:44.092 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.867419+0000", 2024-04-19T01:04:44.092 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.093 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.093 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.093 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:44.093 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.867420+0000", 2024-04-19T01:04:44.093 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.04e-06 2024-04-19T01:04:44.093 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.093 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.093 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:44.093 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.867470+0000", 2024-04-19T01:04:44.093 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9914999999999999e-05 2024-04-19T01:04:44.093 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.093 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.093 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:44.093 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.867498+0000", 2024-04-19T01:04:44.093 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8969e-05 2024-04-19T01:04:44.094 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.094 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.094 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:44.094 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.867517+0000", 2024-04-19T01:04:44.094 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8675999999999999e-05 2024-04-19T01:04:44.094 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.094 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.094 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.094 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.871964+0000", 2024-04-19T01:04:44.094 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0044469260000000004 2024-04-19T01:04:44.094 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.094 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.094 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:44.094 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.871987+0000", 2024-04-19T01:04:44.095 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2521999999999999e-05 2024-04-19T01:04:44.095 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.095 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.095 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:44.095 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.887168+0000", 2024-04-19T01:04:44.095 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015181019 2024-04-19T01:04:44.095 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.095 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.095 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:44.095 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.887246+0000", 2024-04-19T01:04:44.095 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8660999999999998e-05 2024-04-19T01:04:44.095 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.095 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.095 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.095 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.096 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.096 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s1 43/31 ECSubWrite(tid=43, reqid=client.4604.0:251, at_version=43'29, trim_to=0'0, roll_forward_to=43'27))", 2024-04-19T01:04:44.096 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.985924+0000", 2024-04-19T01:04:44.096 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 30.041615279999998, 2024-04-19T01:04:44.096 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.029591623000000001, 2024-04-19T01:04:44.096 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:44.096 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:44.096 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:44.096 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:44.096 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.096 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:44.096 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.985924+0000", 2024-04-19T01:04:44.096 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.096 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.096 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.097 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:44.097 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.985924+0000", 2024-04-19T01:04:44.097 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.097 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.097 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.097 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:44.097 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.985927+0000", 2024-04-19T01:04:44.097 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3460000000000001e-06 2024-04-19T01:04:44.097 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.097 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.097 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:44.097 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986452+0000", 2024-04-19T01:04:44.097 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00052555199999999999 2024-04-19T01:04:44.097 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.097 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.098 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:44.098 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986541+0000", 2024-04-19T01:04:44.098 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8352999999999999e-05 2024-04-19T01:04:44.098 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.098 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.098 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:44.098 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986597+0000", 2024-04-19T01:04:44.098 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6144999999999999e-05 2024-04-19T01:04:44.098 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.098 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.098 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.098 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986776+0000", 2024-04-19T01:04:44.098 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017914999999999999 2024-04-19T01:04:44.098 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.098 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.099 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:44.099 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986811+0000", 2024-04-19T01:04:44.099 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5233e-05 2024-04-19T01:04:44.099 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.099 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.099 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:44.099 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.015444+0000", 2024-04-19T01:04:44.099 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.028632676999999999 2024-04-19T01:04:44.099 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.099 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.099 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:44.099 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.015516+0000", 2024-04-19T01:04:44.099 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2167000000000002e-05 2024-04-19T01:04:44.099 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.099 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.100 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.100 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.100 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.100 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds6 43/31 ECSubWrite(tid=38, reqid=client.4604.0:253, at_version=43'14, trim_to=0'0, roll_forward_to=43'12))", 2024-04-19T01:04:44.100 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.987449+0000", 2024-04-19T01:04:44.100 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 30.0400904, 2024-04-19T01:04:44.100 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.027803888999999998, 2024-04-19T01:04:44.100 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:44.100 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:44.100 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:44.100 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:44.100 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.100 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:44.100 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987449+0000", 2024-04-19T01:04:44.100 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.101 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.101 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.101 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:44.101 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987449+0000", 2024-04-19T01:04:44.101 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.101 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.101 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.101 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:44.101 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987451+0000", 2024-04-19T01:04:44.101 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2929999999999999e-06 2024-04-19T01:04:44.101 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.101 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.101 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:44.101 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988097+0000", 2024-04-19T01:04:44.101 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00064595099999999997 2024-04-19T01:04:44.102 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.102 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.102 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:44.102 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988140+0000", 2024-04-19T01:04:44.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2809000000000003e-05 2024-04-19T01:04:44.102 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.102 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.102 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:44.102 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988168+0000", 2024-04-19T01:04:44.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7498999999999999e-05 2024-04-19T01:04:44.102 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.102 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.102 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.102 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988319+0000", 2024-04-19T01:04:44.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000151621 2024-04-19T01:04:44.103 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.103 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:44.103 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988345+0000", 2024-04-19T01:04:44.103 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5846e-05 2024-04-19T01:04:44.103 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.103 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:44.103 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.015183+0000", 2024-04-19T01:04:44.103 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.026837928 2024-04-19T01:04:44.103 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.103 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:44.103 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.015253+0000", 2024-04-19T01:04:44.103 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9942000000000002e-05 2024-04-19T01:04:44.104 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.104 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.104 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.104 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.104 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.104 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:289 3.1s0 3:8ea800b0:::smithi15528762-8:head [stat,delete] snapc 7=[7,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-19T01:04:44.104 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.872650+0000", 2024-04-19T01:04:44.104 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 29.154889512, 2024-04-19T01:04:44.104 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.043399042999999998, 2024-04-19T01:04:44.104 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:44.104 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:44.104 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:44.104 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:04:44.104 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:04:44.104 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:44.105 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 289 2024-04-19T01:04:44.105 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.105 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:44.105 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.105 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:44.105 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.872650+0000", 2024-04-19T01:04:44.105 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.105 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.105 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.105 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:44.105 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.872650+0000", 2024-04-19T01:04:44.105 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.105 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.105 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.106 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:44.106 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.872653+0000", 2024-04-19T01:04:44.106 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.23e-06 2024-04-19T01:04:44.106 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.106 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.106 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:44.106 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.872669+0000", 2024-04-19T01:04:44.106 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5293000000000001e-05 2024-04-19T01:04:44.106 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.106 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.106 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:44.106 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.872701+0000", 2024-04-19T01:04:44.106 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2391999999999998e-05 2024-04-19T01:04:44.106 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.106 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.107 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:44.107 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.872721+0000", 2024-04-19T01:04:44.107 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0400000000000001e-05 2024-04-19T01:04:44.107 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.107 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.107 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.107 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.872889+0000", 2024-04-19T01:04:44.107 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016723000000000001 2024-04-19T01:04:44.107 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.107 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.107 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:04:44.107 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.873172+0000", 2024-04-19T01:04:44.107 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000283338 2024-04-19T01:04:44.107 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.107 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.108 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:44.108 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.875406+0000", 2024-04-19T01:04:44.108 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022335839999999998 2024-04-19T01:04:44.108 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.108 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.108 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:44.108 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.914580+0000", 2024-04-19T01:04:44.108 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.039174253999999999 2024-04-19T01:04:44.108 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.108 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.108 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-04-19T01:04:44.108 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.914900+0000", 2024-04-19T01:04:44.108 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000320663 2024-04-19T01:04:44.108 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.108 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.109 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:44.109 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.916049+0000", 2024-04-19T01:04:44.109 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001148659 2024-04-19T01:04:44.109 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.109 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.109 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.109 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.109 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.109 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s1 44/31 ECSubWrite(tid=49, reqid=osd.8.0:49, at_version=44'33, trim_to=0'0, roll_forward_to=43'29))", 2024-04-19T01:04:44.109 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.876436+0000", 2024-04-19T01:04:44.109 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 29.151103829, 2024-04-19T01:04:44.109 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024659179, 2024-04-19T01:04:44.109 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:44.109 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:44.109 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:44.110 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:44.110 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.110 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:44.110 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876436+0000", 2024-04-19T01:04:44.110 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.110 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.110 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.110 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:44.110 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876436+0000", 2024-04-19T01:04:44.110 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.110 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.110 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.110 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:44.110 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876437+0000", 2024-04-19T01:04:44.110 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2860000000000001e-06 2024-04-19T01:04:44.111 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.111 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.111 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:44.111 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876485+0000", 2024-04-19T01:04:44.111 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8294999999999998e-05 2024-04-19T01:04:44.111 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.111 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.111 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:44.111 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876512+0000", 2024-04-19T01:04:44.111 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6363000000000002e-05 2024-04-19T01:04:44.111 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.111 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.111 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:44.111 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876531+0000", 2024-04-19T01:04:44.111 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8902999999999998e-05 2024-04-19T01:04:44.112 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.112 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.112 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.112 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.877917+0000", 2024-04-19T01:04:44.112 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0013865870000000001 2024-04-19T01:04:44.112 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.112 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.112 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:44.112 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.877944+0000", 2024-04-19T01:04:44.112 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6553000000000001e-05 2024-04-19T01:04:44.112 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.112 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.112 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:44.112 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.901009+0000", 2024-04-19T01:04:44.112 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023064772000000001 2024-04-19T01:04:44.113 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.113 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.113 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:44.113 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.901095+0000", 2024-04-19T01:04:44.113 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6420000000000003e-05 2024-04-19T01:04:44.113 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.113 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.113 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.113 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.113 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.113 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s6 44/31 ECSubWrite(tid=82, reqid=client.4604.0:280, at_version=44'50, trim_to=0'0, roll_forward_to=43'48))", 2024-04-19T01:04:44.113 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.882093+0000", 2024-04-19T01:04:44.113 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 29.145446218, 2024-04-19T01:04:44.113 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.042938993000000002, 2024-04-19T01:04:44.113 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:44.114 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:44.114 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:44.114 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:44.114 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.114 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:44.114 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882093+0000", 2024-04-19T01:04:44.114 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.114 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.114 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.114 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:44.114 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882093+0000", 2024-04-19T01:04:44.114 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.114 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.114 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.114 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:44.115 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882094+0000", 2024-04-19T01:04:44.115 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0130000000000001e-06 2024-04-19T01:04:44.115 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.115 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.115 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:44.115 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882590+0000", 2024-04-19T01:04:44.115 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00049558999999999996 2024-04-19T01:04:44.115 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.115 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.115 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:44.115 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882631+0000", 2024-04-19T01:04:44.115 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1399000000000001e-05 2024-04-19T01:04:44.115 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.115 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.115 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:44.116 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882654+0000", 2024-04-19T01:04:44.116 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2535999999999999e-05 2024-04-19T01:04:44.116 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.116 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.116 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.116 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882871+0000", 2024-04-19T01:04:44.116 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021730999999999999 2024-04-19T01:04:44.116 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.116 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.116 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:44.116 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882896+0000", 2024-04-19T01:04:44.116 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5111999999999999e-05 2024-04-19T01:04:44.116 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.116 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.116 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:44.117 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.924961+0000", 2024-04-19T01:04:44.117 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.042065022000000001 2024-04-19T01:04:44.117 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.117 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.117 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:44.117 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.925032+0000", 2024-04-19T01:04:44.117 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1011000000000002e-05 2024-04-19T01:04:44.117 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.117 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.117 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.118 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.118 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.118 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s6 44/31 ECSubWrite(tid=84, reqid=client.4604.0:281, at_version=44'51, trim_to=0'0, roll_forward_to=43'48))", 2024-04-19T01:04:44.118 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.907467+0000", 2024-04-19T01:04:44.118 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 29.120072688, 2024-04-19T01:04:44.118 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018626453000000001, 2024-04-19T01:04:44.118 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:44.118 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:44.118 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:44.118 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:44.118 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.118 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:44.118 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.907467+0000", 2024-04-19T01:04:44.118 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.119 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.119 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.119 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:44.119 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.907467+0000", 2024-04-19T01:04:44.119 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.119 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.119 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.119 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:44.119 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.907468+0000", 2024-04-19T01:04:44.119 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3230000000000001e-06 2024-04-19T01:04:44.119 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.119 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.119 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:44.119 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.907725+0000", 2024-04-19T01:04:44.119 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00025646499999999999 2024-04-19T01:04:44.120 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.120 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.120 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:44.120 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.907754+0000", 2024-04-19T01:04:44.120 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9799e-05 2024-04-19T01:04:44.120 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.120 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.120 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:44.120 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.907776+0000", 2024-04-19T01:04:44.120 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1849999999999999e-05 2024-04-19T01:04:44.120 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.121 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.121 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.121 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.914523+0000", 2024-04-19T01:04:44.121 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0067470459999999996 2024-04-19T01:04:44.121 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.121 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.121 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:44.121 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.914550+0000", 2024-04-19T01:04:44.121 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6259e-05 2024-04-19T01:04:44.121 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.121 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.121 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:44.121 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.926057+0000", 2024-04-19T01:04:44.121 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011507224999999999 2024-04-19T01:04:44.122 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.122 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.122 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:44.122 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.926093+0000", 2024-04-19T01:04:44.122 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6486000000000003e-05 2024-04-19T01:04:44.122 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.122 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.122 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.122 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.122 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.122 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.0s6 44/31 ECSubWrite(tid=25, reqid=client.4604.0:284, at_version=44'22, trim_to=0'0, roll_forward_to=44'20))", 2024-04-19T01:04:44.123 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.921980+0000", 2024-04-19T01:04:44.123 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 29.105559945, 2024-04-19T01:04:44.123 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.036689795999999997, 2024-04-19T01:04:44.123 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:44.123 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:44.123 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:44.123 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:44.123 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.123 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:44.123 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.921980+0000", 2024-04-19T01:04:44.123 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.123 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.123 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.123 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:44.123 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.921980+0000", 2024-04-19T01:04:44.124 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.124 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.124 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.124 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:44.124 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.921981+0000", 2024-04-19T01:04:44.124 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.378e-06 2024-04-19T01:04:44.124 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.124 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.124 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:44.124 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.923355+0000", 2024-04-19T01:04:44.124 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0013742820000000001 2024-04-19T01:04:44.124 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.124 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.124 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:44.124 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.923387+0000", 2024-04-19T01:04:44.125 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1702999999999998e-05 2024-04-19T01:04:44.125 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.125 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.125 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:44.125 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.923405+0000", 2024-04-19T01:04:44.125 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8071999999999999e-05 2024-04-19T01:04:44.125 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.125 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.125 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.125 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.923699+0000", 2024-04-19T01:04:44.125 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029384700000000002 2024-04-19T01:04:44.125 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.125 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.125 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:44.125 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.923727+0000", 2024-04-19T01:04:44.126 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8085999999999998e-05 2024-04-19T01:04:44.126 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.126 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.126 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:44.126 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.958618+0000", 2024-04-19T01:04:44.126 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.034891349000000002 2024-04-19T01:04:44.126 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.126 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.126 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:44.126 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.958669+0000", 2024-04-19T01:04:44.126 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1078999999999997e-05 2024-04-19T01:04:44.126 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.126 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.126 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.127 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.127 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.127 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds6 47/31 ECSubWrite(tid=61, reqid=client.4604.0:330, at_version=47'32, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:04:44.127 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.677697+0000", 2024-04-19T01:04:44.127 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 26.349843001, 2024-04-19T01:04:44.127 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.034267463999999997, 2024-04-19T01:04:44.127 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:44.127 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:44.127 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:44.127 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:44.127 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.127 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:44.127 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.677697+0000", 2024-04-19T01:04:44.127 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.127 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.128 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.128 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:44.128 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.677697+0000", 2024-04-19T01:04:44.128 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.128 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.128 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.128 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:44.128 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.677699+0000", 2024-04-19T01:04:44.128 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3929999999999998e-06 2024-04-19T01:04:44.128 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.128 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.128 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:44.128 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678182+0000", 2024-04-19T01:04:44.128 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00048326199999999998 2024-04-19T01:04:44.128 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.129 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.129 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:44.129 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678227+0000", 2024-04-19T01:04:44.129 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4301999999999997e-05 2024-04-19T01:04:44.129 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.129 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.129 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:44.129 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678254+0000", 2024-04-19T01:04:44.129 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7396e-05 2024-04-19T01:04:44.129 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.129 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.129 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.129 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678409+0000", 2024-04-19T01:04:44.129 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000154759 2024-04-19T01:04:44.129 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.130 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.130 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:44.130 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678438+0000", 2024-04-19T01:04:44.130 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8926000000000001e-05 2024-04-19T01:04:44.130 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.130 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.130 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:44.130 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711880+0000", 2024-04-19T01:04:44.130 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.033442209000000001 2024-04-19T01:04:44.130 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.130 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.130 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:44.130 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711964+0000", 2024-04-19T01:04:44.130 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4216999999999997e-05 2024-04-19T01:04:44.131 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.131 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.131 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.131 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.131 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.131 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds6 47/31 ECSubWrite(tid=62, reqid=client.4604.0:331, at_version=47'33, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:04:44.131 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.682679+0000", 2024-04-19T01:04:44.131 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 26.344860611000001, 2024-04-19T01:04:44.131 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.029354201999999999, 2024-04-19T01:04:44.131 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:44.131 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:44.131 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:44.131 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:44.131 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.131 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:44.131 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682679+0000", 2024-04-19T01:04:44.132 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.132 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.132 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.132 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:44.132 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682679+0000", 2024-04-19T01:04:44.132 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.132 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.132 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.132 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:44.132 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682681+0000", 2024-04-19T01:04:44.132 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7799999999999999e-06 2024-04-19T01:04:44.132 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.132 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.132 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:44.133 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682945+0000", 2024-04-19T01:04:44.133 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000263875 2024-04-19T01:04:44.133 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.133 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.133 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:44.133 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682975+0000", 2024-04-19T01:04:44.133 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0216e-05 2024-04-19T01:04:44.133 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.133 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.133 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:44.133 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682990+0000", 2024-04-19T01:04:44.133 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5333e-05 2024-04-19T01:04:44.133 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.133 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.134 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.134 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692463+0000", 2024-04-19T01:04:44.134 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0094724660000000006 2024-04-19T01:04:44.134 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.134 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.134 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:44.134 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692479+0000", 2024-04-19T01:04:44.134 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5892e-05 2024-04-19T01:04:44.134 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.134 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.134 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:44.134 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711976+0000", 2024-04-19T01:04:44.134 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019497012000000001 2024-04-19T01:04:44.134 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.134 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.135 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:44.135 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.712033+0000", 2024-04-19T01:04:44.135 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7627999999999998e-05 2024-04-19T01:04:44.135 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.135 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.135 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.135 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.135 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.135 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds6 47/31 ECSubWrite(tid=63, reqid=client.4604.0:332, at_version=47'34, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:04:44.135 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.688944+0000", 2024-04-19T01:04:44.135 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 26.33859575, 2024-04-19T01:04:44.135 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023924708999999999, 2024-04-19T01:04:44.135 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:44.135 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:44.135 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:44.136 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:44.136 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.136 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:44.136 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.688944+0000", 2024-04-19T01:04:44.136 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.136 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.136 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.136 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:44.136 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.688944+0000", 2024-04-19T01:04:44.136 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.136 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.136 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.136 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:44.143 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.688945+0000", 2024-04-19T01:04:44.143 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5540000000000001e-06 2024-04-19T01:04:44.143 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.143 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.143 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:44.144 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.689042+0000", 2024-04-19T01:04:44.144 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6903000000000004e-05 2024-04-19T01:04:44.144 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.144 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.144 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:44.144 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.689071+0000", 2024-04-19T01:04:44.144 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8249e-05 2024-04-19T01:04:44.144 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.144 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.144 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:44.144 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.689092+0000", 2024-04-19T01:04:44.144 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1685999999999999e-05 2024-04-19T01:04:44.144 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.144 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.145 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.145 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.705134+0000", 2024-04-19T01:04:44.145 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016041454 2024-04-19T01:04:44.145 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.145 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.145 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:44.145 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.705155+0000", 2024-04-19T01:04:44.145 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.177e-05 2024-04-19T01:04:44.145 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.145 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.145 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:44.145 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.712801+0000", 2024-04-19T01:04:44.145 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0076452489999999998 2024-04-19T01:04:44.145 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.145 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.145 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:44.146 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.712869+0000", 2024-04-19T01:04:44.146 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7843999999999996e-05 2024-04-19T01:04:44.146 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.146 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.146 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.146 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.146 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.146 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds6 47/31 ECSubWrite(tid=64, reqid=client.4604.0:333, at_version=47'35, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:04:44.146 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.692117+0000", 2024-04-19T01:04:44.146 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 26.335422916999999, 2024-04-19T01:04:44.146 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020827234, 2024-04-19T01:04:44.146 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:44.146 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:44.146 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:44.147 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:44.147 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.147 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:44.147 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692117+0000", 2024-04-19T01:04:44.147 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.147 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.147 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.147 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:44.147 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692117+0000", 2024-04-19T01:04:44.147 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.147 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.147 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.147 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:44.147 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692118+0000", 2024-04-19T01:04:44.147 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7489999999999999e-06 2024-04-19T01:04:44.148 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.148 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.148 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:44.148 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692163+0000", 2024-04-19T01:04:44.148 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4342999999999999e-05 2024-04-19T01:04:44.148 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.148 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.148 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:44.148 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692204+0000", 2024-04-19T01:04:44.148 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1588999999999998e-05 2024-04-19T01:04:44.148 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.148 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.148 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:44.148 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692227+0000", 2024-04-19T01:04:44.148 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3059000000000001e-05 2024-04-19T01:04:44.149 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.149 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.149 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.149 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711560+0000", 2024-04-19T01:04:44.149 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019332623 2024-04-19T01:04:44.149 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.149 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.149 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:44.149 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711575+0000", 2024-04-19T01:04:44.149 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5218999999999999e-05 2024-04-19T01:04:44.150 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.150 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.150 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:44.150 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.712879+0000", 2024-04-19T01:04:44.150 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001303397 2024-04-19T01:04:44.150 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.150 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.150 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:44.150 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.712944+0000", 2024-04-19T01:04:44.150 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5254999999999998e-05 2024-04-19T01:04:44.150 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.150 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.150 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.150 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.151 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.151 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s1 59/31 ECSubWrite(tid=66, reqid=client.4604.0:423, at_version=59'51, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:04:44.151 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.541655+0000", 2024-04-19T01:04:44.151 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 14.485884564999999, 2024-04-19T01:04:44.151 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.038825746000000001, 2024-04-19T01:04:44.151 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:44.151 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:44.151 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:44.151 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:44.151 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.151 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:44.151 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.541655+0000", 2024-04-19T01:04:44.151 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.151 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.151 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.152 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:44.152 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.541655+0000", 2024-04-19T01:04:44.152 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.152 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.152 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.152 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:44.152 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.541656+0000", 2024-04-19T01:04:44.152 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1699999999999998e-07 2024-04-19T01:04:44.152 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.152 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.152 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:44.152 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.544278+0000", 2024-04-19T01:04:44.152 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002622728 2024-04-19T01:04:44.152 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.152 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.153 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:44.153 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.544314+0000", 2024-04-19T01:04:44.153 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5751999999999999e-05 2024-04-19T01:04:44.153 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.153 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.153 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:44.153 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.544338+0000", 2024-04-19T01:04:44.153 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4003e-05 2024-04-19T01:04:44.153 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.153 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.153 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.153 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.544577+0000", 2024-04-19T01:04:44.153 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00023931500000000001 2024-04-19T01:04:44.153 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.153 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.154 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:44.154 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.544603+0000", 2024-04-19T01:04:44.154 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5854999999999999e-05 2024-04-19T01:04:44.154 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.154 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.154 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:44.154 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.580392+0000", 2024-04-19T01:04:44.154 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.035788477999999999 2024-04-19T01:04:44.154 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.154 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.154 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:44.154 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.580481+0000", 2024-04-19T01:04:44.154 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8998000000000004e-05 2024-04-19T01:04:44.154 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.155 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.155 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.155 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.155 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.155 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s1 59/31 ECSubWrite(tid=67, reqid=client.4604.0:424, at_version=59'52, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:04:44.164 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.549050+0000", 2024-04-19T01:04:44.164 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 14.478489193, 2024-04-19T01:04:44.164 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.031505582999999997, 2024-04-19T01:04:44.164 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:44.164 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:44.165 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:44.165 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:44.165 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.165 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:44.165 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549050+0000", 2024-04-19T01:04:44.166 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.166 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.166 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.166 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:44.166 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549050+0000", 2024-04-19T01:04:44.166 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.167 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.167 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.167 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:44.167 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549052+0000", 2024-04-19T01:04:44.167 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0669999999999999e-06 2024-04-19T01:04:44.167 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.181 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.181 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:44.181 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.550013+0000", 2024-04-19T01:04:44.181 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00096065699999999996 2024-04-19T01:04:44.181 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.182 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.182 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:44.182 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.550048+0000", 2024-04-19T01:04:44.182 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4535000000000001e-05 2024-04-19T01:04:44.182 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.182 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.186 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:44.186 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.550071+0000", 2024-04-19T01:04:44.186 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3057e-05 2024-04-19T01:04:44.187 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.187 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.187 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.187 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.561847+0000", 2024-04-19T01:04:44.187 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.01177603 2024-04-19T01:04:44.187 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.188 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.188 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:44.188 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.561873+0000", 2024-04-19T01:04:44.188 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6679999999999999e-05 2024-04-19T01:04:44.188 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.188 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.188 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:44.189 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.580493+0000", 2024-04-19T01:04:44.189 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018619125 2024-04-19T01:04:44.189 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.189 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.194 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:44.194 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.580556+0000", 2024-04-19T01:04:44.195 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3431999999999998e-05 2024-04-19T01:04:44.195 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.195 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.195 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.196 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.196 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.196 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s1 59/31 ECSubWrite(tid=68, reqid=client.4604.0:425, at_version=59'53, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:04:44.196 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.556579+0000", 2024-04-19T01:04:44.196 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 14.470960869000001, 2024-04-19T01:04:44.196 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024080832999999999, 2024-04-19T01:04:44.197 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:44.197 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:44.197 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:44.197 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:44.197 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.197 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:44.198 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556579+0000", 2024-04-19T01:04:44.198 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.198 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.198 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.198 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:44.198 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556579+0000", 2024-04-19T01:04:44.199 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.199 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.199 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.199 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:44.200 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556581+0000", 2024-04-19T01:04:44.200 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0700000000000001e-06 2024-04-19T01:04:44.200 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.202 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.202 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:44.202 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556696+0000", 2024-04-19T01:04:44.202 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011476099999999999 2024-04-19T01:04:44.202 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.202 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.202 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:44.202 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556723+0000", 2024-04-19T01:04:44.202 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.764e-05 2024-04-19T01:04:44.202 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.203 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.203 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:44.203 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556744+0000", 2024-04-19T01:04:44.203 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1229e-05 2024-04-19T01:04:44.203 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.203 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.203 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.203 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.573868+0000", 2024-04-19T01:04:44.204 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017123104 2024-04-19T01:04:44.204 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.204 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.204 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:44.204 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.573896+0000", 2024-04-19T01:04:44.204 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8526000000000001e-05 2024-04-19T01:04:44.204 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.204 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.204 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:44.205 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.580562+0000", 2024-04-19T01:04:44.205 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0066657180000000002 2024-04-19T01:04:44.205 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.205 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.205 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:44.205 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.580660+0000", 2024-04-19T01:04:44.205 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7785e-05 2024-04-19T01:04:44.205 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.205 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.205 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.206 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.206 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.206 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s1 59/31 ECSubWrite(tid=69, reqid=client.4604.0:426, at_version=59'54, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:04:44.206 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.558218+0000", 2024-04-19T01:04:44.206 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 14.469321641000001, 2024-04-19T01:04:44.206 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024150672000000001, 2024-04-19T01:04:44.206 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:44.206 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:44.206 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:44.206 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:44.206 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.206 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:44.206 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558218+0000", 2024-04-19T01:04:44.206 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.207 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.207 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.207 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:44.207 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558218+0000", 2024-04-19T01:04:44.207 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.207 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.207 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.207 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:44.207 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558219+0000", 2024-04-19T01:04:44.207 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7700000000000004e-07 2024-04-19T01:04:44.207 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.207 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.207 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:44.207 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558256+0000", 2024-04-19T01:04:44.208 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6981999999999999e-05 2024-04-19T01:04:44.208 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.208 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.208 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:44.208 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558275+0000", 2024-04-19T01:04:44.208 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9097e-05 2024-04-19T01:04:44.208 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.208 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.208 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:44.208 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558323+0000", 2024-04-19T01:04:44.208 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7961999999999999e-05 2024-04-19T01:04:44.208 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.208 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.208 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.209 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.579875+0000", 2024-04-19T01:04:44.209 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.021552616 2024-04-19T01:04:44.209 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.209 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.209 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:44.209 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.579901+0000", 2024-04-19T01:04:44.209 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6049000000000001e-05 2024-04-19T01:04:44.209 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.209 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.209 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:44.209 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.582293+0000", 2024-04-19T01:04:44.209 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0023912080000000001 2024-04-19T01:04:44.209 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.209 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.210 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:44.210 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.582369+0000", 2024-04-19T01:04:44.210 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5981000000000003e-05 2024-04-19T01:04:44.210 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.210 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.210 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.210 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.211 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.211 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s6 63/31 ECSubWrite(tid=152, reqid=client.4604.0:465, at_version=63'100, trim_to=0'0, roll_forward_to=62'98))", 2024-04-19T01:04:44.211 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:33.566850+0000", 2024-04-19T01:04:44.211 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.460689780999999, 2024-04-19T01:04:44.211 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019378244999999999, 2024-04-19T01:04:44.211 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:44.211 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:44.211 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:44.211 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:44.211 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.212 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:44.212 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:33.566850+0000", 2024-04-19T01:04:44.212 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.212 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.212 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.212 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:44.212 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:33.566850+0000", 2024-04-19T01:04:44.212 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.212 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.212 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.212 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:44.212 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:33.566850+0000", 2024-04-19T01:04:44.212 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7300000000000001e-07 2024-04-19T01:04:44.212 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.212 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.213 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:44.213 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:33.567763+0000", 2024-04-19T01:04:44.213 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00091228399999999997 2024-04-19T01:04:44.213 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.213 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.213 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:44.213 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:33.567802+0000", 2024-04-19T01:04:44.213 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9622000000000001e-05 2024-04-19T01:04:44.213 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.213 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.213 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:44.213 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:33.567827+0000", 2024-04-19T01:04:44.213 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4797e-05 2024-04-19T01:04:44.213 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.213 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.214 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.214 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:33.568026+0000", 2024-04-19T01:04:44.214 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019901900000000001 2024-04-19T01:04:44.214 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.214 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.214 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:44.214 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:33.568083+0000", 2024-04-19T01:04:44.214 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6570999999999998e-05 2024-04-19T01:04:44.214 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.214 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.214 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:44.214 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:33.586143+0000", 2024-04-19T01:04:44.214 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018060321000000001 2024-04-19T01:04:44.214 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.215 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.215 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:44.215 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:33.586228+0000", 2024-04-19T01:04:44.215 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5158e-05 2024-04-19T01:04:44.215 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.215 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.215 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.215 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.215 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.215 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:04:44.216 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2024-04-19T01:04:44.217 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T01:04:44.217 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2024-04-19T01:04:44.219 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T01:04:44.219 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-19T01:04:44.220 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:04:44.215+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:04:44.221 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2024-04-19T01:04:44.221 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T01:04:44.222 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2024-04-19T01:04:44.222 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T01:04:44.223 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2024-04-19T01:04:44.224 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-19T01:04:44.226 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2024-04-19T01:04:44.226 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-19T01:04:44.227 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2024-04-19T01:04:44.228 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-19T01:04:44.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:04:44.194+0000 7effb77fe640 0 -- 172.21.15.33:0/1514611831 >> [v2:172.21.15.155:6807/741132602,v1:172.21.15.155:6810/741132602] conn(0x7effb8226530 msgr2=0x7effb8228920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-19T01:04:44.230 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2024-04-19T01:04:44.318 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:44.318+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:44.400 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:04:44.400 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-04-19T01:04:44.400 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-04-19T01:04:44.400 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:04:44.414 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-19T01:04:44.419 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:04:44.421+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:04:44.518 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2024-04-19T01:04:44.518 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.7 to 1 (10000) 2024-04-19T01:04:44.520 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:04:44.518+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:04:44.524 INFO:tasks.rados.rados.0.smithi155.stdout:228: done (0 left) 2024-04-19T01:04:44.524 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 22 current snap is 22 2024-04-19T01:04:44.524 INFO:tasks.rados.rados.0.smithi155.stdout:229: seq_num 100 ranges {770048=524288,1294336=753664,2048000=409600} 2024-04-19T01:04:44.535 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:04:44.536 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-04-19T01:04:44.536 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-04-19T01:04:44.536 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-04-19T01:04:44.536 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:04:44.545 INFO:tasks.rados.rados.0.smithi155.stdout:229: writing smithi15528762-22 from 770048 to 1294336 tid 1 2024-04-19T01:04:44.556 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-04-19T01:04:44.559 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-19T01:04:44.560 INFO:tasks.rados.rados.0.smithi155.stdout:229: writing smithi15528762-22 from 1294336 to 2048000 tid 2 2024-04-19T01:04:44.573 INFO:tasks.rados.rados.0.smithi155.stdout:229: writing smithi15528762-22 from 2048000 to 2457600 tid 3 2024-04-19T01:04:44.573 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 44 current snap is 22 2024-04-19T01:04:44.573 INFO:tasks.rados.rados.0.smithi155.stdout:230: seq_num 101 ranges {737280=540672,1277952=507904,1785856=622592,2408448=32768} 2024-04-19T01:04:44.587 INFO:tasks.rados.rados.0.smithi155.stdout:230: writing smithi15528762-44 from 737280 to 1277952 tid 1 2024-04-19T01:04:44.597 INFO:tasks.rados.rados.0.smithi155.stdout:230: writing smithi15528762-44 from 1277952 to 1785856 tid 2 2024-04-19T01:04:44.600 INFO:tasks.rados.rados.0.smithi155.stdout:230: writing smithi15528762-44 from 1785856 to 2408448 tid 3 2024-04-19T01:04:44.601 INFO:tasks.rados.rados.0.smithi155.stdout:230: writing smithi15528762-44 from 2408448 to 2441216 tid 4 2024-04-19T01:04:44.602 INFO:tasks.rados.rados.0.smithi155.stdout:231: read oid 25 snap -1 2024-04-19T01:04:44.602 INFO:tasks.rados.rados.0.smithi155.stdout:231: expect (ObjNum 24 snap 0 seq_num 24) 2024-04-19T01:04:44.602 INFO:tasks.rados.rados.0.smithi155.stdout:232: read oid 17 snap -1 2024-04-19T01:04:44.602 INFO:tasks.rados.rados.0.smithi155.stdout:232: expect (ObjNum 16 snap 0 seq_num 16) 2024-04-19T01:04:44.602 INFO:tasks.rados.rados.0.smithi155.stdout:233: snap_create 2024-04-19T01:04:44.602 INFO:tasks.rados.rados.0.smithi155.stdout:229: finishing write tid 1 to smithi15528762-22 2024-04-19T01:04:44.602 INFO:tasks.rados.rados.0.smithi155.stdout:229: finishing write tid 2 to smithi15528762-22 2024-04-19T01:04:44.602 INFO:tasks.rados.rados.0.smithi155.stdout:229: finishing write tid 3 to smithi15528762-22 2024-04-19T01:04:44.602 INFO:tasks.rados.rados.0.smithi155.stdout:229: finishing write tid 4 to smithi15528762-22 2024-04-19T01:04:44.602 INFO:tasks.rados.rados.0.smithi155.stdout:229: finishing write tid 5 to smithi15528762-22 2024-04-19T01:04:44.602 INFO:tasks.rados.rados.0.smithi155.stdout:229: oid 22 updating version 0 to 70 2024-04-19T01:04:44.602 INFO:tasks.rados.rados.0.smithi155.stdout:229: oid 22 updating version 70 to 72 2024-04-19T01:04:44.602 INFO:tasks.rados.rados.0.smithi155.stdout:229: oid 22 version 72 is already newer than 67 2024-04-19T01:04:44.602 INFO:tasks.rados.rados.0.smithi155.stdout:229: oid 22 version 72 is already newer than 71 2024-04-19T01:04:44.603 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 22 v 72 (ObjNum 100 snap 22 seq_num 100) dirty exists 2024-04-19T01:04:44.603 INFO:tasks.rados.rados.0.smithi155.stdout:229: left oid 22 (ObjNum 100 snap 22 seq_num 100) 2024-04-19T01:04:44.604 INFO:tasks.rados.rados.0.smithi155.stdout:232: expect (ObjNum 16 snap 0 seq_num 16) 2024-04-19T01:04:44.613 INFO:tasks.rados.rados.0.smithi155.stdout:231: expect (ObjNum 24 snap 0 seq_num 24) 2024-04-19T01:04:44.624 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:04:44.621+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:04:44.627 INFO:tasks.rados.rados.0.smithi155.stdout:230: finishing write tid 1 to smithi15528762-44 2024-04-19T01:04:44.627 INFO:tasks.rados.rados.0.smithi155.stdout:230: finishing write tid 2 to smithi15528762-44 2024-04-19T01:04:44.637 INFO:tasks.rados.rados.0.smithi155.stdout:230: finishing write tid 3 to smithi15528762-44 2024-04-19T01:04:44.637 INFO:tasks.rados.rados.0.smithi155.stdout:230: finishing write tid 4 to smithi15528762-44 2024-04-19T01:04:44.642 INFO:tasks.rados.rados.0.smithi155.stdout:230: finishing write tid 5 to smithi15528762-44 2024-04-19T01:04:44.645 INFO:tasks.rados.rados.0.smithi155.stdout:230: finishing write tid 6 to smithi15528762-44 2024-04-19T01:04:44.645 INFO:tasks.rados.rados.0.smithi155.stdout:230: oid 44 updating version 0 to 114 2024-04-19T01:04:44.645 INFO:tasks.rados.rados.0.smithi155.stdout:230: oid 44 version 114 is already newer than 111 2024-04-19T01:04:44.645 INFO:tasks.rados.rados.0.smithi155.stdout:230: oid 44 version 114 is already newer than 112 2024-04-19T01:04:44.645 INFO:tasks.rados.rados.0.smithi155.stdout:230: oid 44 version 114 is already newer than 110 2024-04-19T01:04:44.645 INFO:tasks.rados.rados.0.smithi155.stdout:230: oid 44 version 114 is already newer than 113 2024-04-19T01:04:44.645 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 44 v 114 (ObjNum 101 snap 22 seq_num 101) dirty exists 2024-04-19T01:04:44.645 INFO:tasks.rados.rados.0.smithi155.stdout:230: left oid 44 (ObjNum 101 snap 22 seq_num 101) 2024-04-19T01:04:44.702 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:04:44.702 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-04-19T01:04:44.702 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-04-19T01:04:44.702 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-04-19T01:04:44.702 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.702 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s5 39/31 ECSubWrite(tid=32, reqid=client.4604.0:181, at_version=39'16, trim_to=0'0, roll_forward_to=38'14))", 2024-04-19T01:04:44.702 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:09.821218+0000", 2024-04-19T01:04:44.702 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 34.880833899000002, 2024-04-19T01:04:44.703 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024070148999999999, 2024-04-19T01:04:44.703 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:44.703 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:44.703 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:44.703 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:44.703 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.703 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:44.703 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.821218+0000", 2024-04-19T01:04:44.703 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.703 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.703 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.703 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:44.703 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.821218+0000", 2024-04-19T01:04:44.704 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.704 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.704 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.704 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:44.704 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.821218+0000", 2024-04-19T01:04:44.704 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6699999999999999e-07 2024-04-19T01:04:44.704 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.704 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.704 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:44.704 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.821605+0000", 2024-04-19T01:04:44.704 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00038695599999999999 2024-04-19T01:04:44.704 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.704 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.704 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:44.705 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.821641+0000", 2024-04-19T01:04:44.705 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5998000000000003e-05 2024-04-19T01:04:44.705 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.705 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.705 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:44.705 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.821671+0000", 2024-04-19T01:04:44.705 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9859e-05 2024-04-19T01:04:44.705 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.705 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.705 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.705 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.821894+0000", 2024-04-19T01:04:44.705 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000223365 2024-04-19T01:04:44.705 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.705 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.705 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:44.706 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.821923+0000", 2024-04-19T01:04:44.706 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8198999999999999e-05 2024-04-19T01:04:44.706 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.706 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.706 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:44.706 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.845179+0000", 2024-04-19T01:04:44.706 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023255983000000001 2024-04-19T01:04:44.706 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.706 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.706 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:44.706 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.845288+0000", 2024-04-19T01:04:44.706 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000109422 2024-04-19T01:04:44.706 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.707 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.707 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.707 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.707 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.707 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s5 43/31 ECSubWrite(tid=70, reqid=client.4604.0:235, at_version=43'43, trim_to=0'0, roll_forward_to=41'37))", 2024-04-19T01:04:44.707 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.873887+0000", 2024-04-19T01:04:44.707 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 30.828165218999999, 2024-04-19T01:04:44.707 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.026150459000000001, 2024-04-19T01:04:44.707 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:44.707 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:44.707 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:44.707 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:44.707 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.707 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:44.708 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.873887+0000", 2024-04-19T01:04:44.708 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.708 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.708 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.708 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:44.708 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.873887+0000", 2024-04-19T01:04:44.708 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.708 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.708 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.708 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:44.708 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.873889+0000", 2024-04-19T01:04:44.708 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3099999999999999e-06 2024-04-19T01:04:44.708 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.708 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.708 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:44.709 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.874386+0000", 2024-04-19T01:04:44.709 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00049737900000000005 2024-04-19T01:04:44.709 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.709 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.709 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:44.709 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.874427+0000", 2024-04-19T01:04:44.709 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0349000000000003e-05 2024-04-19T01:04:44.709 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.709 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.709 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:44.709 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.874452+0000", 2024-04-19T01:04:44.710 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4888999999999998e-05 2024-04-19T01:04:44.710 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.710 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.710 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.710 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.874622+0000", 2024-04-19T01:04:44.710 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017063399999999999 2024-04-19T01:04:44.710 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.710 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.710 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:44.710 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.874652+0000", 2024-04-19T01:04:44.710 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9329e-05 2024-04-19T01:04:44.710 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.710 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.710 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:44.711 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.899980+0000", 2024-04-19T01:04:44.711 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025328053999999999 2024-04-19T01:04:44.711 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.711 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.711 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:44.711 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.900037+0000", 2024-04-19T01:04:44.711 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7515000000000001e-05 2024-04-19T01:04:44.711 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.711 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.711 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.711 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.711 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.712 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 43/31 ECSubWrite(tid=43, reqid=client.4604.0:251, at_version=43'29, trim_to=0'0, roll_forward_to=43'27))", 2024-04-19T01:04:44.712 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.985908+0000", 2024-04-19T01:04:44.712 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 30.716144262, 2024-04-19T01:04:44.712 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023748525999999999, 2024-04-19T01:04:44.712 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:44.712 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:44.712 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:44.712 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:44.712 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.712 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:44.712 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.985908+0000", 2024-04-19T01:04:44.712 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.712 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.713 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.713 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:44.713 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.985908+0000", 2024-04-19T01:04:44.713 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.713 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.713 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.713 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:44.713 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.985909+0000", 2024-04-19T01:04:44.713 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.823e-06 2024-04-19T01:04:44.713 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.713 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.713 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:44.713 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986673+0000", 2024-04-19T01:04:44.714 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00076342100000000002 2024-04-19T01:04:44.714 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.714 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.714 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:44.714 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986704+0000", 2024-04-19T01:04:44.714 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1034999999999997e-05 2024-04-19T01:04:44.714 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.714 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.714 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:44.714 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986720+0000", 2024-04-19T01:04:44.714 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6093e-05 2024-04-19T01:04:44.714 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.714 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.715 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.715 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986887+0000", 2024-04-19T01:04:44.715 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000167524 2024-04-19T01:04:44.715 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.715 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.715 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:44.715 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986913+0000", 2024-04-19T01:04:44.715 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5326000000000001e-05 2024-04-19T01:04:44.715 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.715 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.715 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:44.715 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.009579+0000", 2024-04-19T01:04:44.715 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022666657 2024-04-19T01:04:44.715 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.716 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.716 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:44.716 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.009656+0000", 2024-04-19T01:04:44.716 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6647e-05 2024-04-19T01:04:44.716 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.716 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.716 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.716 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.716 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.716 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:276 3.4s0 3:25634c61:::smithi15528762-21:head [read 0~196608,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:44.716 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.015933+0000", 2024-04-19T01:04:44.716 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 30.686118802999999, 2024-04-19T01:04:44.716 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019236120999999998, 2024-04-19T01:04:44.717 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:44.717 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:44.717 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-04-19T01:04:44.717 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:04:44.717 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:04:44.717 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:44.717 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 276 2024-04-19T01:04:44.717 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.717 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:44.717 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.717 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:44.717 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.015933+0000", 2024-04-19T01:04:44.717 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.717 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.718 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.718 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:44.718 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.015933+0000", 2024-04-19T01:04:44.718 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.718 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.722 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.722 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:44.722 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.015936+0000", 2024-04-19T01:04:44.722 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.65e-06 2024-04-19T01:04:44.722 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.722 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.722 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:44.722 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.015946+0000", 2024-04-19T01:04:44.722 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0061000000000001e-05 2024-04-19T01:04:44.722 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.722 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.722 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:44.722 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.015971+0000", 2024-04-19T01:04:44.723 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5704999999999999e-05 2024-04-19T01:04:44.723 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.723 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.723 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:44.723 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.015984+0000", 2024-04-19T01:04:44.723 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2046e-05 2024-04-19T01:04:44.723 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.723 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.723 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.723 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.016097+0000", 2024-04-19T01:04:44.723 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011383999999999999 2024-04-19T01:04:44.724 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.724 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.724 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:04:44.724 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.016224+0000", 2024-04-19T01:04:44.724 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012657 2024-04-19T01:04:44.724 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.724 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.724 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:44.724 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.035169+0000", 2024-04-19T01:04:44.724 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018945249000000001 2024-04-19T01:04:44.724 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.724 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.724 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.724 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.724 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.725 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:277 3.4s0 3:25634c61:::smithi15528762-21:head [read 0~196608,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:44.725 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.016014+0000", 2024-04-19T01:04:44.725 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 30.686037674000001, 2024-04-19T01:04:44.725 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020020381, 2024-04-19T01:04:44.725 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:44.725 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:44.725 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-04-19T01:04:44.725 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:04:44.725 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:04:44.725 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:44.725 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 277 2024-04-19T01:04:44.725 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.725 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:44.725 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.725 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:44.726 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.016014+0000", 2024-04-19T01:04:44.726 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.726 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.726 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.726 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:44.726 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.016014+0000", 2024-04-19T01:04:44.726 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.726 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.726 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.726 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:44.726 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.016015+0000", 2024-04-19T01:04:44.726 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1999999999999998e-07 2024-04-19T01:04:44.726 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.726 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.727 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:44.727 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.016021+0000", 2024-04-19T01:04:44.727 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3600000000000001e-06 2024-04-19T01:04:44.727 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.727 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.727 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:44.727 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.016035+0000", 2024-04-19T01:04:44.727 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3947e-05 2024-04-19T01:04:44.727 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.727 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.727 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:44.727 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.016046+0000", 2024-04-19T01:04:44.727 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1029999999999999e-05 2024-04-19T01:04:44.727 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.727 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.727 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.728 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.016794+0000", 2024-04-19T01:04:44.728 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00074748000000000002 2024-04-19T01:04:44.728 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.728 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.728 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:04:44.728 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.016941+0000", 2024-04-19T01:04:44.728 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000147369 2024-04-19T01:04:44.728 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.728 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.728 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:44.728 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.036035+0000", 2024-04-19T01:04:44.728 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019093275 2024-04-19T01:04:44.728 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.728 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.729 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.729 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.729 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.729 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:278 3.4s0 3:25634c61:::smithi15528762-21:head [read 0~196608,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:44.729 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.016061+0000", 2024-04-19T01:04:44.729 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 30.685990406999998, 2024-04-19T01:04:44.729 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020558379000000002, 2024-04-19T01:04:44.729 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:44.729 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:44.730 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-04-19T01:04:44.731 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:04:44.731 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:04:44.731 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:44.731 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 278 2024-04-19T01:04:44.731 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.731 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:44.731 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.731 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:44.731 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.016061+0000", 2024-04-19T01:04:44.731 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.731 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.731 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.731 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:44.731 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.016061+0000", 2024-04-19T01:04:44.732 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.732 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.732 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.732 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:44.732 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.016062+0000", 2024-04-19T01:04:44.732 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9800000000000002e-07 2024-04-19T01:04:44.732 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.732 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.732 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:44.732 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.016067+0000", 2024-04-19T01:04:44.732 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0050000000000004e-06 2024-04-19T01:04:44.732 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.732 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.732 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:44.732 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.016079+0000", 2024-04-19T01:04:44.733 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1966e-05 2024-04-19T01:04:44.733 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.733 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.733 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:44.733 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.016089+0000", 2024-04-19T01:04:44.733 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.011e-05 2024-04-19T01:04:44.733 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.733 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.733 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.733 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.017458+0000", 2024-04-19T01:04:44.733 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001369035 2024-04-19T01:04:44.733 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.733 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.733 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:04:44.733 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.017608+0000", 2024-04-19T01:04:44.734 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000149133 2024-04-19T01:04:44.734 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.734 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.734 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:44.734 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.036620+0000", 2024-04-19T01:04:44.734 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019012232 2024-04-19T01:04:44.734 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.734 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.734 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.734 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.734 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.734 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s5 43/31 ECSubWrite(tid=76, reqid=client.4604.0:267, at_version=43'46, trim_to=0'0, roll_forward_to=43'45))", 2024-04-19T01:04:44.734 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.034778+0000", 2024-04-19T01:04:44.734 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 30.667273558000002, 2024-04-19T01:04:44.735 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.021558997, 2024-04-19T01:04:44.735 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:44.735 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:44.735 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:44.735 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:44.735 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.735 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:44.735 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.034778+0000", 2024-04-19T01:04:44.735 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.735 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.735 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.735 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:44.735 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.034778+0000", 2024-04-19T01:04:44.735 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.735 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.736 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.736 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:44.736 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.034779+0000", 2024-04-19T01:04:44.736 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8400000000000005e-07 2024-04-19T01:04:44.736 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.736 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.736 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:44.736 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.034950+0000", 2024-04-19T01:04:44.736 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017166300000000001 2024-04-19T01:04:44.736 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.736 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.736 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:44.736 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.034975+0000", 2024-04-19T01:04:44.736 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4861000000000001e-05 2024-04-19T01:04:44.737 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.737 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.737 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:44.737 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.034987+0000", 2024-04-19T01:04:44.737 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2e-05 2024-04-19T01:04:44.737 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.737 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.737 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.737 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.035082+0000", 2024-04-19T01:04:44.737 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4611000000000006e-05 2024-04-19T01:04:44.737 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.737 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.737 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:44.737 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.035097+0000", 2024-04-19T01:04:44.737 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5529999999999999e-05 2024-04-19T01:04:44.738 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.738 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.738 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:44.738 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.056282+0000", 2024-04-19T01:04:44.738 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.021184222999999999 2024-04-19T01:04:44.738 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.738 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.738 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:44.738 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.056337+0000", 2024-04-19T01:04:44.738 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5624999999999997e-05 2024-04-19T01:04:44.738 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.738 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.738 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.738 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.738 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.739 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s5 44/31 ECSubWrite(tid=82, reqid=client.4604.0:280, at_version=44'50, trim_to=0'0, roll_forward_to=43'48))", 2024-04-19T01:04:44.739 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.886693+0000", 2024-04-19T01:04:44.739 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 29.815359124, 2024-04-19T01:04:44.739 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.044983205999999998, 2024-04-19T01:04:44.739 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:44.739 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:44.739 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:44.739 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:44.739 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.739 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:44.739 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.886693+0000", 2024-04-19T01:04:44.739 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.739 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.739 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.739 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:44.740 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.886693+0000", 2024-04-19T01:04:44.740 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.740 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.740 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.740 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:44.740 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.886694+0000", 2024-04-19T01:04:44.740 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2699999999999998e-07 2024-04-19T01:04:44.740 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.740 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.740 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:44.740 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.887101+0000", 2024-04-19T01:04:44.740 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00040740299999999998 2024-04-19T01:04:44.740 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.740 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.741 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:44.741 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.887130+0000", 2024-04-19T01:04:44.741 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9479e-05 2024-04-19T01:04:44.741 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.741 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.741 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:44.741 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.887152+0000", 2024-04-19T01:04:44.741 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1708999999999998e-05 2024-04-19T01:04:44.741 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.741 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.741 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.741 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.887281+0000", 2024-04-19T01:04:44.741 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012845599999999999 2024-04-19T01:04:44.741 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.741 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.742 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:44.742 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.887296+0000", 2024-04-19T01:04:44.742 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5642999999999999e-05 2024-04-19T01:04:44.742 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.742 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.742 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:44.742 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.931601+0000", 2024-04-19T01:04:44.742 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.044304705 2024-04-19T01:04:44.742 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.742 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.742 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:44.742 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.931676+0000", 2024-04-19T01:04:44.742 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4983999999999999e-05 2024-04-19T01:04:44.742 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.743 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.743 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.743 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.743 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.743 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(osd.1.22:1 3.bs0 3:da0c9485:::smithi15528762-40:head [assert-version v10,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-19T01:04:44.743 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.891568+0000", 2024-04-19T01:04:44.743 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 29.810483463000001, 2024-04-19T01:04:44.743 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019029371, 2024-04-19T01:04:44.743 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:44.743 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:44.743 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-04-19T01:04:44.743 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:44.743 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.743 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:44.743 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.891568+0000", 2024-04-19T01:04:44.744 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.744 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.744 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.744 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:44.744 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.891568+0000", 2024-04-19T01:04:44.744 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.744 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.744 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.744 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:44.744 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.891572+0000", 2024-04-19T01:04:44.744 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.596e-06 2024-04-19T01:04:44.744 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.744 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.744 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:44.744 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.891587+0000", 2024-04-19T01:04:44.745 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5130000000000001e-05 2024-04-19T01:04:44.745 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.745 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.745 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:44.745 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.891620+0000", 2024-04-19T01:04:44.745 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3241999999999998e-05 2024-04-19T01:04:44.745 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.745 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.745 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:44.745 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.891641+0000", 2024-04-19T01:04:44.745 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0639e-05 2024-04-19T01:04:44.745 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.745 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.745 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.746 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.891804+0000", 2024-04-19T01:04:44.746 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000163182 2024-04-19T01:04:44.746 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.746 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.746 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-04-19T01:04:44.746 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.892069+0000", 2024-04-19T01:04:44.746 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026467100000000001 2024-04-19T01:04:44.746 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.746 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.746 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.746 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.903406+0000", 2024-04-19T01:04:44.746 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011336946000000001 2024-04-19T01:04:44.746 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.746 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.746 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:04:44.747 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.903666+0000", 2024-04-19T01:04:44.747 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026044099999999998 2024-04-19T01:04:44.747 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.747 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.747 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:44.747 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.910598+0000", 2024-04-19T01:04:44.747 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0069315239999999997 2024-04-19T01:04:44.747 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.747 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.747 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.747 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.747 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.747 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s5 44/31 ECSubWrite(tid=84, reqid=client.4604.0:281, at_version=44'51, trim_to=0'0, roll_forward_to=43'48))", 2024-04-19T01:04:44.747 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.908596+0000", 2024-04-19T01:04:44.747 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 29.793455627, 2024-04-19T01:04:44.748 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024500261999999998, 2024-04-19T01:04:44.748 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:44.748 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:44.748 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:44.748 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:44.748 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.748 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:44.748 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.908596+0000", 2024-04-19T01:04:44.748 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.748 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.748 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.748 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:44.748 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.908596+0000", 2024-04-19T01:04:44.748 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.749 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.749 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.749 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:44.749 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.908598+0000", 2024-04-19T01:04:44.749 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.327e-06 2024-04-19T01:04:44.749 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.749 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.749 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:44.749 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.908853+0000", 2024-04-19T01:04:44.749 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00025536100000000001 2024-04-19T01:04:44.749 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.749 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.749 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:44.749 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.908881+0000", 2024-04-19T01:04:44.750 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7889999999999999e-05 2024-04-19T01:04:44.750 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.750 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.750 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:44.750 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.908901+0000", 2024-04-19T01:04:44.750 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0112e-05 2024-04-19T01:04:44.750 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.750 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.750 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.750 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.921159+0000", 2024-04-19T01:04:44.750 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012258349 2024-04-19T01:04:44.750 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.750 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.750 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:44.750 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.921189+0000", 2024-04-19T01:04:44.751 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9743e-05 2024-04-19T01:04:44.751 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.751 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.751 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:44.751 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.933036+0000", 2024-04-19T01:04:44.751 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011847068000000001 2024-04-19T01:04:44.751 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.751 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.751 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:44.751 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.933096+0000", 2024-04-19T01:04:44.751 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0411999999999997e-05 2024-04-19T01:04:44.751 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.751 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.751 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.752 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.752 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.752 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.0s4 44/31 ECSubWrite(tid=25, reqid=client.4604.0:284, at_version=44'22, trim_to=0'0, roll_forward_to=44'20))", 2024-04-19T01:04:44.752 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.921604+0000", 2024-04-19T01:04:44.752 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 29.780447796000001, 2024-04-19T01:04:44.752 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023370134000000001, 2024-04-19T01:04:44.752 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:44.752 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:44.752 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:44.752 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:44.752 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.752 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:44.752 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.921604+0000", 2024-04-19T01:04:44.752 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.752 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.753 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.753 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:44.753 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.921604+0000", 2024-04-19T01:04:44.753 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.753 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.753 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.753 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:44.753 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.921606+0000", 2024-04-19T01:04:44.753 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.508e-06 2024-04-19T01:04:44.753 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.753 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.753 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:44.753 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.922858+0000", 2024-04-19T01:04:44.754 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0012527829999999999 2024-04-19T01:04:44.754 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.754 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.754 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:44.754 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.922889+0000", 2024-04-19T01:04:44.754 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0822999999999999e-05 2024-04-19T01:04:44.754 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.754 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.754 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:44.755 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.922907+0000", 2024-04-19T01:04:44.755 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8065999999999999e-05 2024-04-19T01:04:44.755 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.755 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.755 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.755 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.923065+0000", 2024-04-19T01:04:44.755 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015793799999999999 2024-04-19T01:04:44.756 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.756 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.756 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:44.756 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.923086+0000", 2024-04-19T01:04:44.756 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0874e-05 2024-04-19T01:04:44.756 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.756 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.756 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:44.756 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.944897+0000", 2024-04-19T01:04:44.757 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.021811185 2024-04-19T01:04:44.757 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.757 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.757 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:44.757 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.944974+0000", 2024-04-19T01:04:44.757 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6957000000000002e-05 2024-04-19T01:04:44.757 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.757 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.757 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.757 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.757 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.757 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 59/31 ECSubWrite(tid=66, reqid=client.4604.0:423, at_version=59'51, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:04:44.757 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.544994+0000", 2024-04-19T01:04:44.758 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.157058186, 2024-04-19T01:04:44.758 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.042329451999999997, 2024-04-19T01:04:44.758 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:44.758 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:44.758 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:44.758 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:44.758 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.758 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:44.758 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.544994+0000", 2024-04-19T01:04:44.759 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.759 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.759 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.759 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:44.759 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.544994+0000", 2024-04-19T01:04:44.759 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.759 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.759 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.759 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:44.759 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.544994+0000", 2024-04-19T01:04:44.759 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2399999999999997e-07 2024-04-19T01:04:44.759 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.759 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.759 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:44.759 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.545710+0000", 2024-04-19T01:04:44.760 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00071517300000000005 2024-04-19T01:04:44.760 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.760 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.760 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:44.760 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.545755+0000", 2024-04-19T01:04:44.760 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5105000000000003e-05 2024-04-19T01:04:44.760 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.760 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.760 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:44.760 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.545780+0000", 2024-04-19T01:04:44.760 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5052999999999999e-05 2024-04-19T01:04:44.760 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.760 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.760 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.760 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.546377+0000", 2024-04-19T01:04:44.761 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000597779 2024-04-19T01:04:44.761 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.761 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.761 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:44.761 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.546405+0000", 2024-04-19T01:04:44.761 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7809000000000001e-05 2024-04-19T01:04:44.761 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.761 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.761 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:44.761 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.587263+0000", 2024-04-19T01:04:44.761 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.040857841999999998 2024-04-19T01:04:44.761 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.761 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.761 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:44.761 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.587323+0000", 2024-04-19T01:04:44.762 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9966999999999997e-05 2024-04-19T01:04:44.762 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.762 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.762 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.762 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.762 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.762 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 59/31 ECSubWrite(tid=67, reqid=client.4604.0:424, at_version=59'52, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:04:44.762 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.549026+0000", 2024-04-19T01:04:44.762 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.153025505, 2024-04-19T01:04:44.762 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.038380015000000003, 2024-04-19T01:04:44.762 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:44.762 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:44.762 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:44.762 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:44.762 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.763 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:44.763 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549026+0000", 2024-04-19T01:04:44.763 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.763 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.763 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.763 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:44.763 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549026+0000", 2024-04-19T01:04:44.763 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.763 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.763 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.763 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:44.763 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549029+0000", 2024-04-19T01:04:44.764 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7269999999999999e-06 2024-04-19T01:04:44.764 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.764 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.764 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:44.764 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549687+0000", 2024-04-19T01:04:44.764 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00065801800000000001 2024-04-19T01:04:44.764 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.764 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.764 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:44.764 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549722+0000", 2024-04-19T01:04:44.764 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4882000000000002e-05 2024-04-19T01:04:44.764 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.764 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.764 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:44.765 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549745+0000", 2024-04-19T01:04:44.765 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2895999999999999e-05 2024-04-19T01:04:44.765 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.765 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.765 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.765 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.568051+0000", 2024-04-19T01:04:44.765 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018306368999999999 2024-04-19T01:04:44.765 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.765 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.765 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:44.765 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.568076+0000", 2024-04-19T01:04:44.765 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.493e-05 2024-04-19T01:04:44.765 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.765 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.765 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:44.766 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.587351+0000", 2024-04-19T01:04:44.766 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019274593999999999 2024-04-19T01:04:44.766 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.766 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.766 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:44.766 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.587406+0000", 2024-04-19T01:04:44.766 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5599000000000001e-05 2024-04-19T01:04:44.766 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.766 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.766 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.766 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.766 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.766 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s1 59/31 ECSubWrite(tid=223, reqid=osd.11.0:223, at_version=59'41, trim_to=0'0, roll_forward_to=58'39))", 2024-04-19T01:04:44.766 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.552920+0000", 2024-04-19T01:04:44.767 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.149132077000001, 2024-04-19T01:04:44.767 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019860096000000001, 2024-04-19T01:04:44.767 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:44.767 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:44.767 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:44.767 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:44.767 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.767 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:44.767 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.552920+0000", 2024-04-19T01:04:44.767 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.767 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.767 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.767 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:44.767 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.552920+0000", 2024-04-19T01:04:44.767 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.768 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.768 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.768 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:44.768 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.552921+0000", 2024-04-19T01:04:44.768 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3200000000000001e-06 2024-04-19T01:04:44.768 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.768 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.768 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:44.768 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.552984+0000", 2024-04-19T01:04:44.768 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2589000000000007e-05 2024-04-19T01:04:44.768 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.768 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.768 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:44.768 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.553019+0000", 2024-04-19T01:04:44.768 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5271999999999997e-05 2024-04-19T01:04:44.769 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.769 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.769 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:44.769 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.553042+0000", 2024-04-19T01:04:44.769 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3093000000000001e-05 2024-04-19T01:04:44.769 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.769 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.769 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.769 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.554395+0000", 2024-04-19T01:04:44.769 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0013527809999999999 2024-04-19T01:04:44.769 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.769 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.769 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:44.769 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.554422+0000", 2024-04-19T01:04:44.769 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7310000000000001e-05 2024-04-19T01:04:44.770 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.770 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.770 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:44.770 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.560791+0000", 2024-04-19T01:04:44.770 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0063686280000000003 2024-04-19T01:04:44.770 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.770 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.770 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:44.770 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.572780+0000", 2024-04-19T01:04:44.770 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011989102999999999 2024-04-19T01:04:44.770 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.770 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.770 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.770 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.770 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.771 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 59/31 ECSubWrite(tid=68, reqid=client.4604.0:425, at_version=59'53, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:04:44.771 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.556627+0000", 2024-04-19T01:04:44.771 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.145424524999999, 2024-04-19T01:04:44.771 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.030965293000000001, 2024-04-19T01:04:44.771 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:44.771 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:44.771 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:44.771 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:44.771 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.771 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:44.771 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556627+0000", 2024-04-19T01:04:44.771 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.771 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.771 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.771 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:44.772 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556627+0000", 2024-04-19T01:04:44.772 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.772 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.772 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.772 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:44.772 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556629+0000", 2024-04-19T01:04:44.772 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.917e-06 2024-04-19T01:04:44.772 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.772 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.772 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:44.772 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556748+0000", 2024-04-19T01:04:44.773 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000118773 2024-04-19T01:04:44.773 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.773 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.773 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:44.773 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556781+0000", 2024-04-19T01:04:44.773 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3025999999999999e-05 2024-04-19T01:04:44.773 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.773 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.774 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:44.774 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556804+0000", 2024-04-19T01:04:44.774 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3252999999999999e-05 2024-04-19T01:04:44.775 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.775 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.775 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.775 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.584574+0000", 2024-04-19T01:04:44.775 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.027770197999999999 2024-04-19T01:04:44.775 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.775 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.776 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:44.776 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.584600+0000", 2024-04-19T01:04:44.776 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5354000000000002e-05 2024-04-19T01:04:44.776 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.776 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.777 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:44.777 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.587504+0000", 2024-04-19T01:04:44.777 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0029039769999999999 2024-04-19T01:04:44.790 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.790 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.790 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:44.791 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.587593+0000", 2024-04-19T01:04:44.791 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8795000000000006e-05 2024-04-19T01:04:44.791 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.791 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.791 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.791 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.792 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.792 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 59/31 ECSubWrite(tid=69, reqid=client.4604.0:426, at_version=59'54, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:04:44.792 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.558167+0000", 2024-04-19T01:04:44.792 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.143884792, 2024-04-19T01:04:44.793 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.029869994, 2024-04-19T01:04:44.793 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:44.793 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:44.793 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:44.793 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:44.793 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.794 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:44.794 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558167+0000", 2024-04-19T01:04:44.794 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.794 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.794 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.794 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:44.794 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558167+0000", 2024-04-19T01:04:44.795 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.795 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.795 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.795 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:44.795 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558169+0000", 2024-04-19T01:04:44.795 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.697e-06 2024-04-19T01:04:44.795 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.796 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.796 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:44.797 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558209+0000", 2024-04-19T01:04:44.797 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0565999999999998e-05 2024-04-19T01:04:44.797 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.797 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.797 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:44.798 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558238+0000", 2024-04-19T01:04:44.798 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8799e-05 2024-04-19T01:04:44.798 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.798 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.798 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:44.800 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558259+0000", 2024-04-19T01:04:44.800 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0383000000000001e-05 2024-04-19T01:04:44.800 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.800 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.800 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.800 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.586922+0000", 2024-04-19T01:04:44.800 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.028663043999999999 2024-04-19T01:04:44.800 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.800 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.800 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:44.800 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.586944+0000", 2024-04-19T01:04:44.800 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2101999999999999e-05 2024-04-19T01:04:44.801 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.801 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.801 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:44.801 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.587979+0000", 2024-04-19T01:04:44.801 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0010351469999999999 2024-04-19T01:04:44.801 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.801 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.801 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:44.801 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.588037+0000", 2024-04-19T01:04:44.801 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8255999999999999e-05 2024-04-19T01:04:44.801 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.801 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.801 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.801 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.802 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.802 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s1 60/31 ECSubWrite(tid=225, reqid=client.4604.0:433, at_version=60'43, trim_to=0'0, roll_forward_to=59'41))", 2024-04-19T01:04:44.802 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:30.570751+0000", 2024-04-19T01:04:44.802 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 14.131301338, 2024-04-19T01:04:44.802 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025868410000000001, 2024-04-19T01:04:44.802 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:44.802 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:44.802 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:44.802 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:44.802 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.802 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:44.802 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:30.570751+0000", 2024-04-19T01:04:44.802 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.802 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.802 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.803 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:44.803 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:30.570751+0000", 2024-04-19T01:04:44.803 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.803 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.803 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:44.803 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:30.570751+0000", 2024-04-19T01:04:44.803 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8400000000000004e-07 2024-04-19T01:04:44.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.803 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.803 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:44.803 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:30.571444+0000", 2024-04-19T01:04:44.803 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00069287699999999995 2024-04-19T01:04:44.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:44.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:30.571485+0000", 2024-04-19T01:04:44.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0602999999999997e-05 2024-04-19T01:04:44.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:44.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:30.571508+0000", 2024-04-19T01:04:44.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3918999999999999e-05 2024-04-19T01:04:44.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.805 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:30.571644+0000", 2024-04-19T01:04:44.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000135847 2024-04-19T01:04:44.805 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.805 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:44.805 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:30.571664+0000", 2024-04-19T01:04:44.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9372999999999999e-05 2024-04-19T01:04:44.805 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.805 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:44.805 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:30.596521+0000", 2024-04-19T01:04:44.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024857055999999999 2024-04-19T01:04:44.805 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:44.806 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:30.596619+0000", 2024-04-19T01:04:44.806 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8151000000000003e-05 2024-04-19T01:04:44.806 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.806 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.806 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.806 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s5 63/31 ECSubWrite(tid=152, reqid=client.4604.0:465, at_version=63'100, trim_to=0'0, roll_forward_to=62'98))", 2024-04-19T01:04:44.806 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:33.566650+0000", 2024-04-19T01:04:44.806 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.135402122, 2024-04-19T01:04:44.806 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018633950999999999, 2024-04-19T01:04:44.806 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:44.806 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:44.807 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:44.807 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:44.807 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.807 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:44.807 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:33.566650+0000", 2024-04-19T01:04:44.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.807 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.807 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.807 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:44.807 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:33.566650+0000", 2024-04-19T01:04:44.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.807 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.807 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.807 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:44.807 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:33.566650+0000", 2024-04-19T01:04:44.808 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0999999999999999e-07 2024-04-19T01:04:44.808 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.808 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.808 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:44.808 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:33.567179+0000", 2024-04-19T01:04:44.808 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00052896900000000001 2024-04-19T01:04:44.808 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.808 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.808 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:44.808 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:33.567252+0000", 2024-04-19T01:04:44.808 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3195000000000006e-05 2024-04-19T01:04:44.808 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.808 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.808 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:44.809 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:33.567269+0000", 2024-04-19T01:04:44.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6969e-05 2024-04-19T01:04:44.809 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.809 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.809 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.809 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:33.567434+0000", 2024-04-19T01:04:44.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016497300000000001 2024-04-19T01:04:44.809 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.809 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.809 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:44.809 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:33.567464+0000", 2024-04-19T01:04:44.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0103e-05 2024-04-19T01:04:44.809 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.809 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.810 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:44.810 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:33.585186+0000", 2024-04-19T01:04:44.810 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017721614 2024-04-19T01:04:44.810 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.810 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.810 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:44.810 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:33.585284+0000", 2024-04-19T01:04:44.810 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7718000000000002e-05 2024-04-19T01:04:44.810 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.810 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.810 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.810 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.810 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.810 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:518 3.4s0 3:232ae717:::smithi15528762-15:head [copy-from ver 52] snapc 16=[16,15,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-04-19T01:04:44.810 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:41.513636+0000", 2024-04-19T01:04:44.811 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 3.1884156429999999, 2024-04-19T01:04:44.811 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.099118577999999999, 2024-04-19T01:04:44.811 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:44.811 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:44.811 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:44.811 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:04:44.811 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:04:44.811 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:44.811 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 518 2024-04-19T01:04:44.811 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.811 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:44.811 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.811 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:44.811 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513636+0000", 2024-04-19T01:04:44.811 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.812 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:44.812 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513636+0000", 2024-04-19T01:04:44.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.812 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:44.812 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513639+0000", 2024-04-19T01:04:44.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1240000000000001e-06 2024-04-19T01:04:44.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.812 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:44.812 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513655+0000", 2024-04-19T01:04:44.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6143000000000001e-05 2024-04-19T01:04:44.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:44.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513688+0000", 2024-04-19T01:04:44.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2396e-05 2024-04-19T01:04:44.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:44.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513705+0000", 2024-04-19T01:04:44.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7295999999999998e-05 2024-04-19T01:04:44.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513854+0000", 2024-04-19T01:04:44.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014896700000000001 2024-04-19T01:04:44.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op must wait for map", 2024-04-19T01:04:44.814 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513868+0000", 2024-04-19T01:04:44.814 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3604000000000001e-05 2024-04-19T01:04:44.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.814 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.519112+0000", 2024-04-19T01:04:44.814 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005243808 2024-04-19T01:04:44.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:04:44.814 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.519314+0000", 2024-04-19T01:04:44.815 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020256300000000001 2024-04-19T01:04:44.815 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.815 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.815 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:44.815 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.543242+0000", 2024-04-19T01:04:44.815 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023928362000000002 2024-04-19T01:04:44.815 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.815 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.816 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:44.816 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.591560+0000", 2024-04-19T01:04:44.816 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.048317225999999998 2024-04-19T01:04:44.816 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.816 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.816 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-04-19T01:04:44.816 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.612629+0000", 2024-04-19T01:04:44.816 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.021069623999999999 2024-04-19T01:04:44.816 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.816 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.816 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:44.816 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.612755+0000", 2024-04-19T01:04:44.816 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012546500000000001 2024-04-19T01:04:44.816 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.817 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.817 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.817 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:519 3.4s0 3:232ae717:::smithi15528762-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e71)", 2024-04-19T01:04:44.817 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:41.513738+0000", 2024-04-19T01:04:44.817 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 3.1883135139999998, 2024-04-19T01:04:44.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.077425087000000004, 2024-04-19T01:04:44.817 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:44.817 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:44.817 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-04-19T01:04:44.817 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:04:44.817 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:04:44.818 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:44.818 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 519 2024-04-19T01:04:44.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.818 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:44.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:44.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513738+0000", 2024-04-19T01:04:44.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:44.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513738+0000", 2024-04-19T01:04:44.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:44.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.819 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:44.819 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513740+0000", 2024-04-19T01:04:44.819 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2300000000000001e-06 2024-04-19T01:04:44.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.819 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:44.819 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513746+0000", 2024-04-19T01:04:44.819 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5660000000000003e-06 2024-04-19T01:04:44.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.819 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:44.819 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513767+0000", 2024-04-19T01:04:44.820 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0990000000000001e-05 2024-04-19T01:04:44.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.820 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:44.820 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513779+0000", 2024-04-19T01:04:44.820 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2033e-05 2024-04-19T01:04:44.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.820 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.820 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513972+0000", 2024-04-19T01:04:44.820 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019274599999999999 2024-04-19T01:04:44.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.820 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:04:44.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513984+0000", 2024-04-19T01:04:44.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.165e-05 2024-04-19T01:04:44.821 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.821 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.519641+0000", 2024-04-19T01:04:44.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056575640000000003 2024-04-19T01:04:44.821 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.821 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for blocked object", 2024-04-19T01:04:44.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.519731+0000", 2024-04-19T01:04:44.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9462000000000005e-05 2024-04-19T01:04:44.821 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:44.822 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.590802+0000", 2024-04-19T01:04:44.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.071071239999999994 2024-04-19T01:04:44.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:04:44.822 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.590982+0000", 2024-04-19T01:04:44.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017997 2024-04-19T01:04:44.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:44.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:44.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:44.822 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.591163+0000", 2024-04-19T01:04:44.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018163599999999999 2024-04-19T01:04:44.823 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.823 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.823 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.823 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:44.823 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:44.823 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:04:44.825 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-04-19T01:04:44.825 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:04:44.825+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:04:44.926 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:44.922+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:44.951 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:04:44.952 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-04-19T01:04:44.952 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-04-19T01:04:44.952 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:04:44.965 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-04-19T01:04:45.026 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:04:45.026+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:04:45.082 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:04:45.083 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-04-19T01:04:45.083 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-04-19T01:04:45.083 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-04-19T01:04:45.083 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:04:45.097 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-04-19T01:04:45.146 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:04:45.143+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:04:45.216 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:04:45.216 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-04-19T01:04:45.216 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-04-19T01:04:45.217 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-04-19T01:04:45.217 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.217 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s2 39/31 ECSubWrite(tid=32, reqid=client.4604.0:181, at_version=39'16, trim_to=0'0, roll_forward_to=38'14))", 2024-04-19T01:04:45.217 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:09.821438+0000", 2024-04-19T01:04:45.217 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 35.395125614999998, 2024-04-19T01:04:45.217 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020304547999999999, 2024-04-19T01:04:45.217 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:45.217 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:45.217 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:45.217 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:45.217 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.217 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:45.217 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.821438+0000", 2024-04-19T01:04:45.217 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:45.218 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.218 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.218 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:45.218 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.821438+0000", 2024-04-19T01:04:45.218 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:45.218 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.218 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.218 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:45.218 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.821439+0000", 2024-04-19T01:04:45.218 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2699999999999999e-07 2024-04-19T01:04:45.218 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.218 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.219 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:45.219 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.822000+0000", 2024-04-19T01:04:45.219 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00056130600000000002 2024-04-19T01:04:45.219 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.219 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.219 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:45.219 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.822034+0000", 2024-04-19T01:04:45.219 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3785000000000003e-05 2024-04-19T01:04:45.219 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.219 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.219 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:45.219 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.822057+0000", 2024-04-19T01:04:45.219 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3289e-05 2024-04-19T01:04:45.220 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.220 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.220 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:45.220 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.822234+0000", 2024-04-19T01:04:45.220 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017660000000000001 2024-04-19T01:04:45.220 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.220 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.220 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:45.220 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.822249+0000", 2024-04-19T01:04:45.220 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5415999999999999e-05 2024-04-19T01:04:45.220 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.220 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.220 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:45.220 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.841628+0000", 2024-04-19T01:04:45.221 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019379159999999999 2024-04-19T01:04:45.221 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.221 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.221 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:45.221 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.841743+0000", 2024-04-19T01:04:45.221 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000114265 2024-04-19T01:04:45.221 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:45.222 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:45.222 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:45.222 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.222 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.222 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s2 39/31 ECSubWrite(tid=33, reqid=client.4604.0:182, at_version=39'17, trim_to=0'0, roll_forward_to=38'14))", 2024-04-19T01:04:45.222 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:09.829850+0000", 2024-04-19T01:04:45.222 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 35.386714091999998, 2024-04-19T01:04:45.222 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012982403999999999, 2024-04-19T01:04:45.222 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:45.222 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:45.223 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:45.223 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:45.223 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.223 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:45.223 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.829850+0000", 2024-04-19T01:04:45.223 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:45.223 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.223 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.223 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:45.223 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.829850+0000", 2024-04-19T01:04:45.224 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:45.224 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.224 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.224 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:45.224 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.829851+0000", 2024-04-19T01:04:45.225 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7269999999999999e-06 2024-04-19T01:04:45.225 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.225 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.225 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:45.225 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.830044+0000", 2024-04-19T01:04:45.225 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000192449 2024-04-19T01:04:45.225 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.225 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.225 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:45.225 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.830072+0000", 2024-04-19T01:04:45.225 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8492000000000001e-05 2024-04-19T01:04:45.225 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.226 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.226 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:45.226 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.830088+0000", 2024-04-19T01:04:45.226 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5423000000000001e-05 2024-04-19T01:04:45.226 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.226 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.226 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:45.226 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.834720+0000", 2024-04-19T01:04:45.226 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0046326320000000002 2024-04-19T01:04:45.226 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.227 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.227 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:45.227 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.834735+0000", 2024-04-19T01:04:45.227 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.468e-05 2024-04-19T01:04:45.227 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.227 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.227 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:45.227 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.842759+0000", 2024-04-19T01:04:45.227 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0080238400000000008 2024-04-19T01:04:45.227 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.227 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.227 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:45.227 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.842832+0000", 2024-04-19T01:04:45.227 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3161e-05 2024-04-19T01:04:45.227 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:45.228 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:45.228 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:45.228 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.228 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.228 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s2 39/31 ECSubWrite(tid=38, reqid=client.4604.0:187, at_version=39'20, trim_to=0'0, roll_forward_to=39'18))", 2024-04-19T01:04:45.228 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:09.890157+0000", 2024-04-19T01:04:45.228 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 35.326407127000003, 2024-04-19T01:04:45.229 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025377373000000002, 2024-04-19T01:04:45.229 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:45.229 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:45.229 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:45.229 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:45.229 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.229 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:45.229 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.890157+0000", 2024-04-19T01:04:45.230 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:45.230 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.230 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.230 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:45.230 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.890157+0000", 2024-04-19T01:04:45.230 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:45.230 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.230 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.230 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:45.230 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.890159+0000", 2024-04-19T01:04:45.230 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3140000000000002e-06 2024-04-19T01:04:45.231 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.231 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.231 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:45.231 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.890514+0000", 2024-04-19T01:04:45.231 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035502899999999998 2024-04-19T01:04:45.231 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.231 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.231 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:45.231 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.890556+0000", 2024-04-19T01:04:45.231 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1819000000000001e-05 2024-04-19T01:04:45.231 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.231 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.231 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:45.232 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.890581+0000", 2024-04-19T01:04:45.232 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5548999999999999e-05 2024-04-19T01:04:45.232 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.232 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.232 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:45.232 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.890752+0000", 2024-04-19T01:04:45.232 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000170147 2024-04-19T01:04:45.232 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.232 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.233 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:45.233 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.890781+0000", 2024-04-19T01:04:45.233 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9275e-05 2024-04-19T01:04:45.233 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.233 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.233 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:45.233 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.915444+0000", 2024-04-19T01:04:45.238 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024662726999999999 2024-04-19T01:04:45.238 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.239 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.239 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:45.239 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.915534+0000", 2024-04-19T01:04:45.239 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0513000000000006e-05 2024-04-19T01:04:45.239 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:45.239 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:45.239 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:45.239 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.239 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.239 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s2 39/31 ECSubWrite(tid=39, reqid=client.4604.0:188, at_version=39'21, trim_to=0'0, roll_forward_to=39'18))", 2024-04-19T01:04:45.239 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:09.897182+0000", 2024-04-19T01:04:45.239 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 35.319381532000001, 2024-04-19T01:04:45.239 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018721683999999999, 2024-04-19T01:04:45.239 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:45.240 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:45.240 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:45.240 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:45.240 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.240 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:45.240 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.897182+0000", 2024-04-19T01:04:45.240 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:45.240 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.240 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.240 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:45.240 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.897182+0000", 2024-04-19T01:04:45.240 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:45.240 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.240 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.241 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:45.241 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.897185+0000", 2024-04-19T01:04:45.241 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4099999999999998e-06 2024-04-19T01:04:45.241 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.241 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.241 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:45.241 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.897464+0000", 2024-04-19T01:04:45.241 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00027915900000000002 2024-04-19T01:04:45.241 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.241 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.241 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:45.241 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.897515+0000", 2024-04-19T01:04:45.241 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0778000000000001e-05 2024-04-19T01:04:45.241 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.241 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.242 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:45.242 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.897536+0000", 2024-04-19T01:04:45.242 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1685999999999999e-05 2024-04-19T01:04:45.242 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.242 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.242 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:45.242 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.905546+0000", 2024-04-19T01:04:45.242 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0080096409999999996 2024-04-19T01:04:45.242 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.242 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.243 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:45.243 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.905571+0000", 2024-04-19T01:04:45.243 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.482e-05 2024-04-19T01:04:45.243 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.243 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.243 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:45.243 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.915777+0000", 2024-04-19T01:04:45.243 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010205881999999999 2024-04-19T01:04:45.243 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.243 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.243 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:45.244 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.915904+0000", 2024-04-19T01:04:45.244 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012730799999999999 2024-04-19T01:04:45.244 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:45.244 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:45.244 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:45.244 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.244 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.244 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s2 39/31 ECSubWrite(tid=40, reqid=client.4604.0:189, at_version=39'22, trim_to=0'0, roll_forward_to=39'18))", 2024-04-19T01:04:45.244 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:09.900556+0000", 2024-04-19T01:04:45.244 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 35.316007984999999, 2024-04-19T01:04:45.244 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016123281, 2024-04-19T01:04:45.244 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:45.244 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:45.244 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:45.244 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:45.245 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.245 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:45.245 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.900556+0000", 2024-04-19T01:04:45.245 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:45.245 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.245 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.245 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:45.245 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.900556+0000", 2024-04-19T01:04:45.245 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:45.246 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.246 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.246 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:45.246 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.900558+0000", 2024-04-19T01:04:45.246 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0159999999999998e-06 2024-04-19T01:04:45.246 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.246 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.246 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:45.246 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.900599+0000", 2024-04-19T01:04:45.246 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1424999999999997e-05 2024-04-19T01:04:45.246 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.246 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.246 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:45.246 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.900639+0000", 2024-04-19T01:04:45.247 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9916000000000002e-05 2024-04-19T01:04:45.247 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.247 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.247 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:45.247 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.900662+0000", 2024-04-19T01:04:45.247 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3295e-05 2024-04-19T01:04:45.247 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.247 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.247 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:45.247 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.915152+0000", 2024-04-19T01:04:45.247 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014489837 2024-04-19T01:04:45.247 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.247 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.247 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:45.247 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.915168+0000", 2024-04-19T01:04:45.248 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5233e-05 2024-04-19T01:04:45.248 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.248 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.248 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:45.248 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.916606+0000", 2024-04-19T01:04:45.248 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001437993 2024-04-19T01:04:45.248 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.248 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.249 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:45.249 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:09.916679+0000", 2024-04-19T01:04:45.249 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3565999999999993e-05 2024-04-19T01:04:45.249 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:45.249 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:45.249 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:45.249 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.249 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.249 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s2 43/31 ECSubWrite(tid=70, reqid=client.4604.0:235, at_version=43'43, trim_to=0'0, roll_forward_to=41'37))", 2024-04-19T01:04:45.249 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.874738+0000", 2024-04-19T01:04:45.249 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 31.34182556, 2024-04-19T01:04:45.249 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024645897999999999, 2024-04-19T01:04:45.250 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:45.250 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:45.250 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:45.250 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:45.250 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.250 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:45.250 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.874738+0000", 2024-04-19T01:04:45.250 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:45.250 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.250 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.250 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:45.250 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.874738+0000", 2024-04-19T01:04:45.250 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:45.250 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.251 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.251 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:45.251 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.874739+0000", 2024-04-19T01:04:45.251 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.203e-06 2024-04-19T01:04:45.251 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.251 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.251 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:45.251 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.875466+0000", 2024-04-19T01:04:45.251 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00072702499999999996 2024-04-19T01:04:45.251 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.251 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.251 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:45.251 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.875489+0000", 2024-04-19T01:04:45.251 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2932e-05 2024-04-19T01:04:45.252 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.252 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.252 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:45.252 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.875509+0000", 2024-04-19T01:04:45.252 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9996999999999998e-05 2024-04-19T01:04:45.252 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.252 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.252 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:45.252 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.875656+0000", 2024-04-19T01:04:45.252 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014699099999999999 2024-04-19T01:04:45.252 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.252 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.252 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:45.252 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.875683+0000", 2024-04-19T01:04:45.252 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6435999999999999e-05 2024-04-19T01:04:45.253 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.253 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.253 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:45.253 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.899339+0000", 2024-04-19T01:04:45.253 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023656074999999999 2024-04-19T01:04:45.253 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.253 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.253 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:45.254 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.899384+0000", 2024-04-19T01:04:45.254 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5238999999999998e-05 2024-04-19T01:04:45.254 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:45.254 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:45.254 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:45.254 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.254 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.254 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 43/31 ECSubWrite(tid=43, reqid=client.4604.0:251, at_version=43'29, trim_to=0'0, roll_forward_to=43'27))", 2024-04-19T01:04:45.254 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.985922+0000", 2024-04-19T01:04:45.254 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 31.230641993999999, 2024-04-19T01:04:45.254 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.021650376999999998, 2024-04-19T01:04:45.254 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:45.254 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:45.255 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:45.255 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:45.255 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.255 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:45.255 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.985922+0000", 2024-04-19T01:04:45.255 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:45.255 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.255 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.255 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:45.256 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.985922+0000", 2024-04-19T01:04:45.256 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:45.256 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.256 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.256 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:45.256 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.985924+0000", 2024-04-19T01:04:45.256 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1169999999999998e-06 2024-04-19T01:04:45.256 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.256 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.256 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:45.256 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986891+0000", 2024-04-19T01:04:45.256 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00096720599999999997 2024-04-19T01:04:45.256 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.257 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.257 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:45.257 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986935+0000", 2024-04-19T01:04:45.257 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3776000000000002e-05 2024-04-19T01:04:45.257 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.257 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.257 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:45.257 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986960+0000", 2024-04-19T01:04:45.257 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5418999999999998e-05 2024-04-19T01:04:45.257 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.257 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.257 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:45.257 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987142+0000", 2024-04-19T01:04:45.257 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018212 2024-04-19T01:04:45.258 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.258 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.258 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:45.258 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987174+0000", 2024-04-19T01:04:45.258 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1782000000000002e-05 2024-04-19T01:04:45.258 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.258 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.258 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:45.258 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.007479+0000", 2024-04-19T01:04:45.258 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020304543000000001 2024-04-19T01:04:45.258 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.258 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.258 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:45.259 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.007572+0000", 2024-04-19T01:04:45.259 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3413999999999997e-05 2024-04-19T01:04:45.259 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:45.259 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:45.259 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:45.259 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.259 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.259 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 44/31 ECSubWrite(tid=49, reqid=osd.8.0:49, at_version=44'33, trim_to=0'0, roll_forward_to=43'29))", 2024-04-19T01:04:45.259 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.876887+0000", 2024-04-19T01:04:45.259 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 30.339676471000001, 2024-04-19T01:04:45.259 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017055688999999999, 2024-04-19T01:04:45.260 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:45.260 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:45.260 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:45.260 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:45.260 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.260 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:45.260 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876887+0000", 2024-04-19T01:04:45.260 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:45.260 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.260 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.260 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:45.260 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876887+0000", 2024-04-19T01:04:45.260 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:45.260 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.261 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.261 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:45.261 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876889+0000", 2024-04-19T01:04:45.261 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.827e-06 2024-04-19T01:04:45.261 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.261 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.261 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:45.261 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876942+0000", 2024-04-19T01:04:45.261 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3248000000000002e-05 2024-04-19T01:04:45.261 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.261 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.261 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:45.261 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.876984+0000", 2024-04-19T01:04:45.261 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1598999999999999e-05 2024-04-19T01:04:45.262 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.262 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.262 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:45.262 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.877007+0000", 2024-04-19T01:04:45.262 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3209000000000001e-05 2024-04-19T01:04:45.262 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.262 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.262 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:45.262 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.878441+0000", 2024-04-19T01:04:45.262 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0014341790000000001 2024-04-19T01:04:45.262 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.262 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.262 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:45.262 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.878457+0000", 2024-04-19T01:04:45.262 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5787e-05 2024-04-19T01:04:45.263 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.263 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.263 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:45.263 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.893844+0000", 2024-04-19T01:04:45.263 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015387283 2024-04-19T01:04:45.263 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.263 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.263 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:45.263 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.893943+0000", 2024-04-19T01:04:45.264 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8556999999999998e-05 2024-04-19T01:04:45.264 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:45.264 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:45.264 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:45.264 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.264 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.264 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s2 44/31 ECSubWrite(tid=82, reqid=client.4604.0:280, at_version=44'50, trim_to=0'0, roll_forward_to=43'48))", 2024-04-19T01:04:45.264 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.882222+0000", 2024-04-19T01:04:45.264 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 30.334342186000001, 2024-04-19T01:04:45.264 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.035827673999999997, 2024-04-19T01:04:45.264 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:45.264 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:45.264 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:45.265 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:45.265 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.265 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:45.265 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882222+0000", 2024-04-19T01:04:45.265 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:45.265 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.265 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.265 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:45.265 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882222+0000", 2024-04-19T01:04:45.265 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:45.265 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.265 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.265 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:45.266 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882222+0000", 2024-04-19T01:04:45.266 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7999999999999995e-07 2024-04-19T01:04:45.266 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.266 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.266 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:45.266 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882709+0000", 2024-04-19T01:04:45.266 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00048730799999999999 2024-04-19T01:04:45.266 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.266 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.267 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:45.267 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882742+0000", 2024-04-19T01:04:45.267 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2428000000000002e-05 2024-04-19T01:04:45.267 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.267 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.267 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:45.267 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882765+0000", 2024-04-19T01:04:45.267 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2693000000000001e-05 2024-04-19T01:04:45.267 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.267 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.267 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:45.267 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.883062+0000", 2024-04-19T01:04:45.267 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029777100000000001 2024-04-19T01:04:45.267 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.267 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.268 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:45.268 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.883088+0000", 2024-04-19T01:04:45.268 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6018999999999999e-05 2024-04-19T01:04:45.268 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.268 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.268 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:45.268 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.917995+0000", 2024-04-19T01:04:45.268 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.034906988999999999 2024-04-19T01:04:45.268 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.268 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.268 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:45.268 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.918049+0000", 2024-04-19T01:04:45.268 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3885999999999999e-05 2024-04-19T01:04:45.268 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:45.268 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:45.269 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:45.269 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.269 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.269 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s4 49/31 ECSubWrite(tid=202, reqid=osd.11.0:201, at_version=0'0, trim_to=0'0, roll_forward_to=49'24))", 2024-04-19T01:04:45.269 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:19.537002+0000", 2024-04-19T01:04:45.269 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 25.679561490000001, 2024-04-19T01:04:45.269 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023787906000000001, 2024-04-19T01:04:45.269 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:45.269 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:45.269 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:45.270 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:45.270 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.270 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:45.270 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.537002+0000", 2024-04-19T01:04:45.270 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:45.270 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.270 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.270 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:45.270 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.537002+0000", 2024-04-19T01:04:45.270 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:45.270 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.270 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.270 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:45.270 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.537004+0000", 2024-04-19T01:04:45.271 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.553e-06 2024-04-19T01:04:45.271 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.271 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.271 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:45.271 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.537031+0000", 2024-04-19T01:04:45.271 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7039999999999999e-05 2024-04-19T01:04:45.271 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.271 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.271 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:45.271 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.537063+0000", 2024-04-19T01:04:45.271 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1705000000000002e-05 2024-04-19T01:04:45.271 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.271 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.271 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:45.272 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.537084+0000", 2024-04-19T01:04:45.272 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1852999999999999e-05 2024-04-19T01:04:45.272 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.272 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.272 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:45.272 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.537559+0000", 2024-04-19T01:04:45.272 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00047440999999999998 2024-04-19T01:04:45.272 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.272 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.272 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:45.273 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.537585+0000", 2024-04-19T01:04:45.273 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6319e-05 2024-04-19T01:04:45.273 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.273 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.273 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:45.273 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.560733+0000", 2024-04-19T01:04:45.274 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023147581 2024-04-19T01:04:45.274 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.274 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.274 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:45.274 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.560790+0000", 2024-04-19T01:04:45.274 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7445000000000003e-05 2024-04-19T01:04:45.274 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:45.275 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:45.275 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:45.275 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.275 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.275 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s4 49/31 ECSubWrite(tid=203, reqid=client.4604.0:360, at_version=49'25, trim_to=0'0, roll_forward_to=49'24))", 2024-04-19T01:04:45.275 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:19.539163+0000", 2024-04-19T01:04:45.275 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 25.677400425999998, 2024-04-19T01:04:45.275 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022464522000000001, 2024-04-19T01:04:45.275 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:45.275 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:45.275 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:45.275 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:45.275 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.275 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:45.275 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.539163+0000", 2024-04-19T01:04:45.276 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:45.276 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.276 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.276 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:45.276 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.539163+0000", 2024-04-19T01:04:45.276 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:45.276 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.276 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.277 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:45.277 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.539165+0000", 2024-04-19T01:04:45.277 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.95e-06 2024-04-19T01:04:45.277 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.277 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.277 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:45.277 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.539991+0000", 2024-04-19T01:04:45.277 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00082553499999999996 2024-04-19T01:04:45.277 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.277 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.277 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:45.277 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.540003+0000", 2024-04-19T01:04:45.277 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1838000000000001e-05 2024-04-19T01:04:45.277 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.278 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.278 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:45.278 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.540011+0000", 2024-04-19T01:04:45.278 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3829999999999992e-06 2024-04-19T01:04:45.278 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.278 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.278 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:45.278 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.540704+0000", 2024-04-19T01:04:45.278 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00069282000000000002 2024-04-19T01:04:45.278 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.278 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.278 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:45.278 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.540730+0000", 2024-04-19T01:04:45.278 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5649000000000001e-05 2024-04-19T01:04:45.279 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.279 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.279 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:45.279 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.561559+0000", 2024-04-19T01:04:45.279 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020829555 2024-04-19T01:04:45.279 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.279 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.279 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:45.279 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:19.561628+0000", 2024-04-19T01:04:45.279 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8792000000000001e-05 2024-04-19T01:04:45.280 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:45.280 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:45.280 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:45.280 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.280 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.280 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 59/31 ECSubWrite(tid=66, reqid=client.4604.0:423, at_version=59'51, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:04:45.280 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.541618+0000", 2024-04-19T01:04:45.280 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.674946104, 2024-04-19T01:04:45.280 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.040796200999999997, 2024-04-19T01:04:45.280 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:45.280 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:45.280 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:45.280 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:45.281 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.281 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:45.281 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.541618+0000", 2024-04-19T01:04:45.281 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:45.281 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.281 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.281 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:45.281 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.541618+0000", 2024-04-19T01:04:45.281 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:45.281 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.281 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.281 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:45.281 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.541618+0000", 2024-04-19T01:04:45.281 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9699999999999995e-07 2024-04-19T01:04:45.281 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.282 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.282 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:45.282 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.542215+0000", 2024-04-19T01:04:45.282 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00059629300000000002 2024-04-19T01:04:45.282 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.282 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.282 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:45.282 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.542247+0000", 2024-04-19T01:04:45.282 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2339000000000003e-05 2024-04-19T01:04:45.282 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.282 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.282 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:45.282 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.542272+0000", 2024-04-19T01:04:45.282 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4973e-05 2024-04-19T01:04:45.283 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.283 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.283 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:45.283 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.542445+0000", 2024-04-19T01:04:45.283 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017274599999999999 2024-04-19T01:04:45.283 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.283 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.283 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:45.283 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.542468+0000", 2024-04-19T01:04:45.283 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3689e-05 2024-04-19T01:04:45.283 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.283 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.283 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:45.283 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.582330+0000", 2024-04-19T01:04:45.283 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.039861946000000002 2024-04-19T01:04:45.284 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.284 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.284 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:45.284 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.582414+0000", 2024-04-19T01:04:45.284 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3517999999999996e-05 2024-04-19T01:04:45.284 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:45.284 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:45.284 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:45.284 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.285 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.285 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 59/31 ECSubWrite(tid=67, reqid=client.4604.0:424, at_version=59'52, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:04:45.285 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.549371+0000", 2024-04-19T01:04:45.285 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.667192281, 2024-04-19T01:04:45.285 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.033134352999999998, 2024-04-19T01:04:45.285 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:45.285 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:45.285 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:45.285 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:45.285 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.285 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:45.285 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549371+0000", 2024-04-19T01:04:45.285 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:45.285 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.286 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.286 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:45.286 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549371+0000", 2024-04-19T01:04:45.286 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:45.286 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.286 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.286 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:45.286 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549375+0000", 2024-04-19T01:04:45.286 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0869999999999998e-06 2024-04-19T01:04:45.287 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.287 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.287 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:45.287 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549729+0000", 2024-04-19T01:04:45.287 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000354225 2024-04-19T01:04:45.287 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.287 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.287 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:45.287 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549769+0000", 2024-04-19T01:04:45.287 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0488999999999998e-05 2024-04-19T01:04:45.287 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.287 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.288 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:45.288 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549793+0000", 2024-04-19T01:04:45.288 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3702000000000001e-05 2024-04-19T01:04:45.288 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.288 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.288 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:45.288 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.561556+0000", 2024-04-19T01:04:45.288 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011763018 2024-04-19T01:04:45.288 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.288 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.288 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:45.288 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.561583+0000", 2024-04-19T01:04:45.288 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7243e-05 2024-04-19T01:04:45.288 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.289 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.289 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:45.289 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.582445+0000", 2024-04-19T01:04:45.289 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020862084999999999 2024-04-19T01:04:45.289 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.289 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.289 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:45.289 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.582506+0000", 2024-04-19T01:04:45.289 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0504000000000002e-05 2024-04-19T01:04:45.289 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:45.289 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:45.289 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:45.289 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.289 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.290 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 59/31 ECSubWrite(tid=68, reqid=client.4604.0:425, at_version=59'53, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:04:45.290 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.556915+0000", 2024-04-19T01:04:45.290 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.65964915, 2024-04-19T01:04:45.290 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.027043356000000001, 2024-04-19T01:04:45.290 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:45.290 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:45.290 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:45.290 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:45.290 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.291 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:45.291 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556915+0000", 2024-04-19T01:04:45.291 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:45.291 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.291 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.291 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:45.291 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556915+0000", 2024-04-19T01:04:45.291 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:45.291 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.291 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.291 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:45.291 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556917+0000", 2024-04-19T01:04:45.291 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9769999999999999e-06 2024-04-19T01:04:45.291 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.291 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.292 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:45.292 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.557048+0000", 2024-04-19T01:04:45.292 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013188699999999999 2024-04-19T01:04:45.292 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.292 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.292 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:45.292 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.557079+0000", 2024-04-19T01:04:45.292 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0318999999999999e-05 2024-04-19T01:04:45.292 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.292 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.292 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:45.292 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.557139+0000", 2024-04-19T01:04:45.292 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0469e-05 2024-04-19T01:04:45.292 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.292 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.293 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:45.293 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.579036+0000", 2024-04-19T01:04:45.293 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.021896863999999999 2024-04-19T01:04:45.293 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.293 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.293 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:45.293 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.579065+0000", 2024-04-19T01:04:45.293 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8846000000000001e-05 2024-04-19T01:04:45.293 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.293 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.293 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:45.293 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.583896+0000", 2024-04-19T01:04:45.293 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0048308159999999999 2024-04-19T01:04:45.293 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.293 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.294 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:45.294 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.583958+0000", 2024-04-19T01:04:45.294 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2178e-05 2024-04-19T01:04:45.294 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:45.294 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:45.294 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:45.294 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.294 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.294 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s2 59/31 ECSubWrite(tid=149, reqid=osd.13.0:148, at_version=0'0, trim_to=0'0, roll_forward_to=59'97))", 2024-04-19T01:04:45.294 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.557722+0000", 2024-04-19T01:04:45.294 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.658841496000001, 2024-04-19T01:04:45.294 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018782455, 2024-04-19T01:04:45.294 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:45.294 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:45.295 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:45.295 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:45.295 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.295 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:45.295 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.557722+0000", 2024-04-19T01:04:45.295 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:45.295 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.295 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.296 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:45.296 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.557722+0000", 2024-04-19T01:04:45.296 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:45.296 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.296 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.296 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:45.296 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.557724+0000", 2024-04-19T01:04:45.296 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.04e-06 2024-04-19T01:04:45.296 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.296 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.296 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:45.296 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.557755+0000", 2024-04-19T01:04:45.296 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1046000000000001e-05 2024-04-19T01:04:45.296 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.296 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.297 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:45.297 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.557844+0000", 2024-04-19T01:04:45.297 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8220999999999994e-05 2024-04-19T01:04:45.297 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.297 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.297 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:45.297 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.557869+0000", 2024-04-19T01:04:45.297 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5626000000000002e-05 2024-04-19T01:04:45.298 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.298 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.298 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:45.298 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558090+0000", 2024-04-19T01:04:45.298 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000220702 2024-04-19T01:04:45.298 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.298 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.298 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:45.298 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558123+0000", 2024-04-19T01:04:45.298 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2821999999999998e-05 2024-04-19T01:04:45.298 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.298 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.298 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:45.298 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.576409+0000", 2024-04-19T01:04:45.299 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018285954 2024-04-19T01:04:45.299 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.299 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.299 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:45.299 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.576505+0000", 2024-04-19T01:04:45.299 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6044000000000004e-05 2024-04-19T01:04:45.299 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:45.299 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:45.299 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:45.299 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.299 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.299 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 59/31 ECSubWrite(tid=69, reqid=client.4604.0:426, at_version=59'54, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:04:45.299 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.558693+0000", 2024-04-19T01:04:45.299 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.657871074999999, 2024-04-19T01:04:45.299 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.02532322, 2024-04-19T01:04:45.300 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:45.300 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:45.300 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:45.300 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:45.300 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.300 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:45.300 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558693+0000", 2024-04-19T01:04:45.300 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:45.300 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.300 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.301 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:45.301 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558693+0000", 2024-04-19T01:04:45.301 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:45.301 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.301 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.301 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:45.301 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558695+0000", 2024-04-19T01:04:45.301 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.863e-06 2024-04-19T01:04:45.301 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.301 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.301 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:45.301 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558741+0000", 2024-04-19T01:04:45.301 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6801999999999997e-05 2024-04-19T01:04:45.301 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.302 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.302 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:45.302 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558772+0000", 2024-04-19T01:04:45.302 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0666000000000001e-05 2024-04-19T01:04:45.302 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.302 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.304 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:45.304 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558795+0000", 2024-04-19T01:04:45.304 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3292999999999999e-05 2024-04-19T01:04:45.304 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.304 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.304 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:45.304 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.582649+0000", 2024-04-19T01:04:45.304 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023853210999999999 2024-04-19T01:04:45.304 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.304 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.305 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:45.305 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.582674+0000", 2024-04-19T01:04:45.305 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5619999999999999e-05 2024-04-19T01:04:45.305 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.305 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.305 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:45.307 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.583967+0000", 2024-04-19T01:04:45.307 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0012931570000000001 2024-04-19T01:04:45.307 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.307 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.307 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:45.307 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.584016+0000", 2024-04-19T01:04:45.307 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8607999999999999e-05 2024-04-19T01:04:45.308 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:45.308 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:45.308 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:45.308 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.308 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.308 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s4 60/31 ECSubWrite(tid=225, reqid=client.4604.0:433, at_version=60'43, trim_to=0'0, roll_forward_to=59'41))", 2024-04-19T01:04:45.308 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:30.571023+0000", 2024-04-19T01:04:45.308 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 14.645540278, 2024-04-19T01:04:45.308 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025941571, 2024-04-19T01:04:45.308 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:45.309 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:45.309 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:45.309 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:45.309 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.309 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:45.309 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:30.571023+0000", 2024-04-19T01:04:45.309 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:45.309 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.309 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.310 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:45.310 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:30.571023+0000", 2024-04-19T01:04:45.310 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:45.310 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.310 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.310 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:45.310 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:30.571024+0000", 2024-04-19T01:04:45.310 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9999999999999997e-07 2024-04-19T01:04:45.310 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.310 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.310 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:45.310 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:30.571665+0000", 2024-04-19T01:04:45.310 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00064050200000000004 2024-04-19T01:04:45.310 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.311 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.311 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:45.311 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:30.571694+0000", 2024-04-19T01:04:45.311 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9643000000000001e-05 2024-04-19T01:04:45.311 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.311 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.311 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:45.311 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:30.571712+0000", 2024-04-19T01:04:45.311 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7419e-05 2024-04-19T01:04:45.311 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.311 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.311 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:45.311 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:30.571806+0000", 2024-04-19T01:04:45.312 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4717999999999997e-05 2024-04-19T01:04:45.312 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.312 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.312 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:45.312 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:30.571822+0000", 2024-04-19T01:04:45.312 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5290000000000001e-05 2024-04-19T01:04:45.312 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.312 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.312 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:45.312 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:30.596834+0000", 2024-04-19T01:04:45.312 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025012487999999999 2024-04-19T01:04:45.312 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.312 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.312 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:45.312 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:30.596965+0000", 2024-04-19T01:04:45.313 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013091099999999999 2024-04-19T01:04:45.313 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:45.313 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:45.313 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:45.313 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.313 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.313 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s4 60/31 ECSubWrite(tid=226, reqid=client.4604.0:434, at_version=60'44, trim_to=0'0, roll_forward_to=59'41))", 2024-04-19T01:04:45.313 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:30.583117+0000", 2024-04-19T01:04:45.313 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 14.633447136000001, 2024-04-19T01:04:45.313 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014751060999999999, 2024-04-19T01:04:45.313 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:45.314 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:45.314 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:45.314 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:45.314 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.314 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:45.314 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:30.583117+0000", 2024-04-19T01:04:45.314 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:45.314 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.321 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.321 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:45.321 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:30.583117+0000", 2024-04-19T01:04:45.336 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:45.337 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.337 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.337 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:45.337 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:30.583119+0000", 2024-04-19T01:04:45.345 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.967e-06 2024-04-19T01:04:45.345 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.345 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.345 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:45.345 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:30.583589+0000", 2024-04-19T01:04:45.345 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00047055600000000001 2024-04-19T01:04:45.345 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.345 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.346 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:45.346 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:30.583619+0000", 2024-04-19T01:04:45.346 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9488999999999998e-05 2024-04-19T01:04:45.346 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.346 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.346 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:45.346 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:30.583634+0000", 2024-04-19T01:04:45.346 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5353e-05 2024-04-19T01:04:45.346 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.346 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.348 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:45.348 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:30.583802+0000", 2024-04-19T01:04:45.348 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016764899999999999 2024-04-19T01:04:45.349 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.349 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.350 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:45.350 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:30.583831+0000", 2024-04-19T01:04:45.350 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9733000000000001e-05 2024-04-19T01:04:45.350 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.350 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.350 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:45.350 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:30.597764+0000", 2024-04-19T01:04:45.350 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013932917 2024-04-19T01:04:45.350 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.350 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.350 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:45.363 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:30.597868+0000", 2024-04-19T01:04:45.363 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000103397 2024-04-19T01:04:45.363 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:45.363 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:45.363 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:45.363 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.363 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.363 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:477 3.7s0 3:e2c4c59f:::smithi15528762-35:head [read 0~3080192,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-04-19T01:04:45.363 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:35.610538+0000", 2024-04-19T01:04:45.363 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 9.6060252839999993, 2024-04-19T01:04:45.363 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012999851999999999, 2024-04-19T01:04:45.364 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:45.364 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:45.366 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-04-19T01:04:45.366 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:04:45.366 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:04:45.367 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:45.367 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 477 2024-04-19T01:04:45.367 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.368 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:45.368 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.368 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:45.368 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:35.610538+0000", 2024-04-19T01:04:45.368 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:45.368 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.368 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.368 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:45.368 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:35.610538+0000", 2024-04-19T01:04:45.369 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:45.369 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.369 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.369 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:45.369 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:35.610539+0000", 2024-04-19T01:04:45.369 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.09e-07 2024-04-19T01:04:45.369 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.371 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.371 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:45.371 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:35.610544+0000", 2024-04-19T01:04:45.371 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7520000000000004e-06 2024-04-19T01:04:45.371 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.371 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.372 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:45.372 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:35.610552+0000", 2024-04-19T01:04:45.372 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3459999999999994e-06 2024-04-19T01:04:45.372 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.372 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.372 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:45.372 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:35.610559+0000", 2024-04-19T01:04:45.373 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5100000000000004e-06 2024-04-19T01:04:45.373 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.373 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.373 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:45.373 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:35.611712+0000", 2024-04-19T01:04:45.374 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011527830000000001 2024-04-19T01:04:45.374 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.374 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.374 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:04:45.374 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:35.611841+0000", 2024-04-19T01:04:45.374 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012910600000000001 2024-04-19T01:04:45.375 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.375 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.375 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:45.375 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:35.623538+0000", 2024-04-19T01:04:45.375 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011697546 2024-04-19T01:04:45.375 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:45.376 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:45.376 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:45.376 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.376 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.376 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.4s2 71/31 ECSubWrite(tid=89, reqid=client.4604.0:518, at_version=71'22, trim_to=0'0, roll_forward_to=52'20))", 2024-04-19T01:04:45.377 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:41.545110+0000", 2024-04-19T01:04:45.377 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 3.6714533299999998, 2024-04-19T01:04:45.377 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.049869387000000001, 2024-04-19T01:04:45.377 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:04:45.378 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:04:45.378 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:04:45.378 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:04:45.378 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.378 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:04:45.378 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.545110+0000", 2024-04-19T01:04:45.379 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:45.379 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.379 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.379 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:04:45.379 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.545110+0000", 2024-04-19T01:04:45.379 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:04:45.379 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.380 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.380 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:04:45.385 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.545111+0000", 2024-04-19T01:04:45.386 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3399999999999999e-07 2024-04-19T01:04:45.386 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.386 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.386 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:04:45.386 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.546538+0000", 2024-04-19T01:04:45.386 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0014274330000000001 2024-04-19T01:04:45.386 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.386 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.386 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:04:45.386 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.546582+0000", 2024-04-19T01:04:45.386 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3155000000000003e-05 2024-04-19T01:04:45.386 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.386 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.387 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:04:45.387 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.546610+0000", 2024-04-19T01:04:45.387 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8807e-05 2024-04-19T01:04:45.387 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.387 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.387 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:04:45.387 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.546756+0000", 2024-04-19T01:04:45.387 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014540399999999999 2024-04-19T01:04:45.387 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.387 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.387 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:04:45.387 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.546778+0000", 2024-04-19T01:04:45.387 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1722999999999999e-05 2024-04-19T01:04:45.387 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.388 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.388 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:04:45.388 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.594901+0000", 2024-04-19T01:04:45.388 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.048123725999999999 2024-04-19T01:04:45.388 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:04:45.388 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:04:45.388 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:04:45.388 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.594980+0000", 2024-04-19T01:04:45.388 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8604999999999997e-05 2024-04-19T01:04:45.388 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:45.388 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:45.388 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:45.389 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:04:45.389 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:04:45.389 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:04:45.391 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-19T01:04:45.391 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:45.390+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:45.491 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:04:45.491+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:04:45.522 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:04:45.522 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:04:45.522 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-04-19T01:04:45.523 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:04:45.535 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-19T01:04:45.585 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.8 to 1 (10000) 2024-04-19T01:04:45.593 INFO:tasks.rados.rados.0.smithi155.stdout:229: done (4 left) 2024-04-19T01:04:45.593 INFO:tasks.rados.rados.0.smithi155.stdout:230: done (3 left) 2024-04-19T01:04:45.593 INFO:tasks.rados.rados.0.smithi155.stdout:231: done (2 left) 2024-04-19T01:04:45.593 INFO:tasks.rados.rados.0.smithi155.stdout:232: done (1 left) 2024-04-19T01:04:45.593 INFO:tasks.rados.rados.0.smithi155.stdout:233: done (0 left) 2024-04-19T01:04:45.594 INFO:tasks.rados.rados.0.smithi155.stdout:234: snap_create 2024-04-19T01:04:45.594 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:04:45.591+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:04:45.612 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-04-19T01:04:45.653 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:04:45.653 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:04:45.653 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-04-19T01:04:45.653 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-04-19T01:04:45.653 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:04:45.667 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-19T01:04:45.694 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:04:45.694+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:04:45.787 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:04:45.787 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-04-19T01:04:45.787 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-04-19T01:04:45.788 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-04-19T01:04:45.788 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.788 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.2s5 43/31 ECSubWrite(tid=180, reqid=osd.11.0:180, at_version=43'10, trim_to=0'0, roll_forward_to=41'8))", 2024-04-19T01:04:45.788 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:13.844393+0000", 2024-04-19T01:04:45.788 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 31.942401039, 2024-04-19T01:04:45.788 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0044436120000000004, 2024-04-19T01:04:45.788 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:45.788 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:45.788 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:45.788 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:45.788 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.788 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:45.788 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.844393+0000", 2024-04-19T01:04:45.789 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:45.789 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.789 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.789 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:45.789 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.844393+0000", 2024-04-19T01:04:45.789 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:45.789 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.789 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.789 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:45.789 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.844394+0000", 2024-04-19T01:04:45.789 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9400000000000001e-07 2024-04-19T01:04:45.789 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.789 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.790 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:45.790 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.844445+0000", 2024-04-19T01:04:45.790 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.1523999999999997e-05 2024-04-19T01:04:45.790 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.790 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.790 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:45.790 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.844472+0000", 2024-04-19T01:04:45.790 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7161e-05 2024-04-19T01:04:45.790 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.790 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.790 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:45.790 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.844488+0000", 2024-04-19T01:04:45.790 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6127e-05 2024-04-19T01:04:45.790 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.791 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.791 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:45.791 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.844592+0000", 2024-04-19T01:04:45.791 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000104034 2024-04-19T01:04:45.791 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.791 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.791 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:45.792 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.844608+0000", 2024-04-19T01:04:45.792 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5931999999999999e-05 2024-04-19T01:04:45.792 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.792 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.792 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:45.792 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.848782+0000", 2024-04-19T01:04:45.792 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.004173398 2024-04-19T01:04:45.792 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.792 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.792 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:45.792 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.848837+0000", 2024-04-19T01:04:45.793 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.5142000000000002e-05 2024-04-19T01:04:45.793 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:45.793 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:45.793 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:45.793 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.793 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.793 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.0s3 44/31 ECSubWrite(tid=23, reqid=osd.1.0:23, at_version=44'20, trim_to=0'0, roll_forward_to=43'18))", 2024-04-19T01:04:45.793 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:14.891502+0000", 2024-04-19T01:04:45.793 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 30.895292354999999, 2024-04-19T01:04:45.793 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0073445860000000002, 2024-04-19T01:04:45.793 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:45.793 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:45.793 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:45.793 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:45.794 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.794 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:45.794 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.891502+0000", 2024-04-19T01:04:45.794 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:45.794 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.794 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.794 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:45.794 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.891502+0000", 2024-04-19T01:04:45.794 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:45.794 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.794 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.794 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:45.794 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.891502+0000", 2024-04-19T01:04:45.794 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6100000000000002e-07 2024-04-19T01:04:45.795 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.795 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.795 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:45.795 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.891547+0000", 2024-04-19T01:04:45.795 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5077999999999998e-05 2024-04-19T01:04:45.795 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.795 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.795 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:45.795 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.891565+0000", 2024-04-19T01:04:45.795 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7574000000000001e-05 2024-04-19T01:04:45.795 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.795 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.795 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:45.795 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.891578+0000", 2024-04-19T01:04:45.796 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2700999999999999e-05 2024-04-19T01:04:45.796 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.796 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.796 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:45.796 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.891729+0000", 2024-04-19T01:04:45.796 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015078300000000001 2024-04-19T01:04:45.796 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.796 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.796 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:45.796 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.891759+0000", 2024-04-19T01:04:45.796 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0722999999999997e-05 2024-04-19T01:04:45.796 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.796 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.796 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:45.796 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.898759+0000", 2024-04-19T01:04:45.797 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0069993640000000001 2024-04-19T01:04:45.797 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.797 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.797 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:45.797 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.898847+0000", 2024-04-19T01:04:45.797 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.8002000000000002e-05 2024-04-19T01:04:45.797 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:45.797 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:45.797 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:45.797 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.797 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.797 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.4s3 47/31 ECSubWrite(tid=47, reqid=client.4604.0:299, at_version=47'12, trim_to=0'0, roll_forward_to=35'10))", 2024-04-19T01:04:45.797 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:17.523701+0000", 2024-04-19T01:04:45.797 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 28.263093094999999, 2024-04-19T01:04:45.798 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0044232689999999996, 2024-04-19T01:04:45.798 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:45.798 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:45.798 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:45.798 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:45.798 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.798 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:45.798 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.523701+0000", 2024-04-19T01:04:45.798 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:45.798 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.798 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.798 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:45.798 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.523701+0000", 2024-04-19T01:04:45.799 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:45.799 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.799 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.799 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:45.799 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.523702+0000", 2024-04-19T01:04:45.799 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.08e-06 2024-04-19T01:04:45.799 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.800 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.800 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:45.800 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.523758+0000", 2024-04-19T01:04:45.800 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.5488999999999997e-05 2024-04-19T01:04:45.800 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.800 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.800 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:45.800 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.523794+0000", 2024-04-19T01:04:45.800 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6502999999999999e-05 2024-04-19T01:04:45.800 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.800 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.800 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:45.801 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.523840+0000", 2024-04-19T01:04:45.801 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5899000000000002e-05 2024-04-19T01:04:45.801 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.801 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.801 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:45.801 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.524232+0000", 2024-04-19T01:04:45.801 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00039216499999999998 2024-04-19T01:04:45.801 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.801 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.802 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:45.802 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.524270+0000", 2024-04-19T01:04:45.802 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7725999999999998e-05 2024-04-19T01:04:45.802 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.802 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.802 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:45.802 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.528011+0000", 2024-04-19T01:04:45.802 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0037405590000000001 2024-04-19T01:04:45.802 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.802 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.802 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:45.802 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.528125+0000", 2024-04-19T01:04:45.802 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000113848 2024-04-19T01:04:45.802 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:45.803 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:45.803 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:45.803 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.803 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.803 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.0s3 47/31 ECSubWrite(tid=30, reqid=client.4604.0:307, at_version=47'28, trim_to=0'0, roll_forward_to=45'26))", 2024-04-19T01:04:45.803 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:17.550051+0000", 2024-04-19T01:04:45.803 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 28.236743251, 2024-04-19T01:04:45.803 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0077366320000000002, 2024-04-19T01:04:45.803 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:45.803 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:45.810 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:45.810 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:45.810 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.810 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:45.811 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.550051+0000", 2024-04-19T01:04:45.811 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:45.811 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.811 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.811 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:45.811 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.550051+0000", 2024-04-19T01:04:45.811 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:45.811 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.811 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.811 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:45.811 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.550052+0000", 2024-04-19T01:04:45.811 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7700000000000004e-07 2024-04-19T01:04:45.811 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.811 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.812 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:45.812 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.550470+0000", 2024-04-19T01:04:45.812 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00041806800000000001 2024-04-19T01:04:45.812 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.812 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.813 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:45.813 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.550509+0000", 2024-04-19T01:04:45.813 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8773000000000003e-05 2024-04-19T01:04:45.813 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.813 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.813 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:45.813 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.550531+0000", 2024-04-19T01:04:45.813 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2841999999999999e-05 2024-04-19T01:04:45.813 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.813 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.813 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:45.813 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.550645+0000", 2024-04-19T01:04:45.813 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011352499999999999 2024-04-19T01:04:45.813 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.813 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.814 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:45.814 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.550657+0000", 2024-04-19T01:04:45.814 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2252000000000001e-05 2024-04-19T01:04:45.814 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.814 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.814 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:45.814 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.557738+0000", 2024-04-19T01:04:45.814 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0070807329999999996 2024-04-19T01:04:45.814 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.814 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.814 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:45.814 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.557788+0000", 2024-04-19T01:04:45.814 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9762000000000002e-05 2024-04-19T01:04:45.814 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:45.815 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:45.815 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:45.815 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.815 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.815 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.0s3 47/31 ECSubWrite(tid=32, reqid=client.4604.0:308, at_version=47'29, trim_to=0'0, roll_forward_to=47'28))", 2024-04-19T01:04:45.815 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:17.565539+0000", 2024-04-19T01:04:45.815 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 28.221254925, 2024-04-19T01:04:45.815 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.018109618000000001, 2024-04-19T01:04:45.815 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:45.815 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:45.815 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:45.815 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:45.815 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.815 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:45.815 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.565539+0000", 2024-04-19T01:04:45.816 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:45.816 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.816 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.816 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:45.817 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.565539+0000", 2024-04-19T01:04:45.817 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:45.817 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.817 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.817 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:45.817 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.565541+0000", 2024-04-19T01:04:45.817 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.04e-06 2024-04-19T01:04:45.817 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.817 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.817 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:45.817 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.566554+0000", 2024-04-19T01:04:45.817 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0010129029999999999 2024-04-19T01:04:45.817 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.817 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.818 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:45.818 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.566591+0000", 2024-04-19T01:04:45.818 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7061999999999999e-05 2024-04-19T01:04:45.818 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.818 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.818 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:45.818 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.566613+0000", 2024-04-19T01:04:45.818 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.145e-05 2024-04-19T01:04:45.818 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.818 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.818 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:45.818 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.566752+0000", 2024-04-19T01:04:45.818 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000138868 2024-04-19T01:04:45.818 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.819 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.819 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:45.819 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.566783+0000", 2024-04-19T01:04:45.819 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0772000000000003e-05 2024-04-19T01:04:45.819 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.819 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.819 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:45.819 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.583581+0000", 2024-04-19T01:04:45.819 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.016798124000000001 2024-04-19T01:04:45.819 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.819 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.819 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:45.819 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.583649+0000", 2024-04-19T01:04:45.819 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.8398999999999993e-05 2024-04-19T01:04:45.819 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:45.820 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:45.820 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:45.820 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.820 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.820 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.0s3 47/31 ECSubWrite(tid=33, reqid=client.4604.0:309, at_version=47'30, trim_to=0'0, roll_forward_to=47'28))", 2024-04-19T01:04:45.820 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:17.573407+0000", 2024-04-19T01:04:45.820 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 28.213387317999999, 2024-04-19T01:04:45.820 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.010864425, 2024-04-19T01:04:45.820 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:45.820 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:45.820 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:45.820 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:45.820 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.820 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:45.820 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.573407+0000", 2024-04-19T01:04:45.821 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:45.821 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.821 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.821 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:45.821 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.573407+0000", 2024-04-19T01:04:45.821 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:45.821 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.821 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.821 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:45.821 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.573408+0000", 2024-04-19T01:04:45.821 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4300000000000001e-06 2024-04-19T01:04:45.821 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.821 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.821 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:45.821 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.573474+0000", 2024-04-19T01:04:45.822 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5711999999999997e-05 2024-04-19T01:04:45.822 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.822 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.822 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:45.822 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.573506+0000", 2024-04-19T01:04:45.822 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1445999999999997e-05 2024-04-19T01:04:45.822 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.823 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.823 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:45.823 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.573526+0000", 2024-04-19T01:04:45.823 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9919999999999999e-05 2024-04-19T01:04:45.823 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.823 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.823 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:45.823 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.582034+0000", 2024-04-19T01:04:45.823 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.008508837 2024-04-19T01:04:45.823 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.823 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.823 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:45.823 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.582053+0000", 2024-04-19T01:04:45.823 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8816000000000001e-05 2024-04-19T01:04:45.823 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.824 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.825 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:45.825 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.584163+0000", 2024-04-19T01:04:45.825 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0021099439999999999 2024-04-19T01:04:45.825 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.825 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.825 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:45.825 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.584271+0000", 2024-04-19T01:04:45.825 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010832000000000001 2024-04-19T01:04:45.825 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:45.825 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:45.825 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:45.825 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.825 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.825 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.0s3 47/31 ECSubWrite(tid=34, reqid=client.4604.0:310, at_version=47'31, trim_to=0'0, roll_forward_to=47'28))", 2024-04-19T01:04:45.826 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:17.574889+0000", 2024-04-19T01:04:45.826 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 28.211905805000001, 2024-04-19T01:04:45.826 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0098864950000000004, 2024-04-19T01:04:45.826 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:45.826 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:45.826 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:45.826 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:45.826 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.826 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:45.827 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.574889+0000", 2024-04-19T01:04:45.827 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:45.827 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.827 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.827 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:45.827 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.574889+0000", 2024-04-19T01:04:45.827 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:45.827 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.827 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.827 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:45.827 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.574890+0000", 2024-04-19T01:04:45.827 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.457e-06 2024-04-19T01:04:45.828 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.828 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.828 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:45.828 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.574924+0000", 2024-04-19T01:04:45.828 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4181999999999999e-05 2024-04-19T01:04:45.828 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.829 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.829 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:45.830 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.574953+0000", 2024-04-19T01:04:45.830 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9226000000000001e-05 2024-04-19T01:04:45.830 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.830 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.830 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:45.830 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.574974+0000", 2024-04-19T01:04:45.830 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0273000000000001e-05 2024-04-19T01:04:45.830 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.830 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.830 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:45.830 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.583276+0000", 2024-04-19T01:04:45.830 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.008302495 2024-04-19T01:04:45.830 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.831 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.831 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:45.839 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.583290+0000", 2024-04-19T01:04:45.840 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3696999999999999e-05 2024-04-19T01:04:45.840 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.840 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.841 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:45.841 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.584690+0000", 2024-04-19T01:04:45.841 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001400047 2024-04-19T01:04:45.841 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.841 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.841 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:45.842 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.584775+0000", 2024-04-19T01:04:45.842 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5117999999999994e-05 2024-04-19T01:04:45.842 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:45.842 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:45.842 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:45.843 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.843 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.843 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as3 48/31 ECSubWrite(tid=71, reqid=client.4604.0:354, at_version=48'10, trim_to=0'0, roll_forward_to=34'8))", 2024-04-19T01:04:45.843 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.556344+0000", 2024-04-19T01:04:45.844 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 27.23045016, 2024-04-19T01:04:45.844 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.030027702, 2024-04-19T01:04:45.844 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:45.844 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:45.844 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:45.845 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:45.845 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.845 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:45.845 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.556344+0000", 2024-04-19T01:04:45.845 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:45.846 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.846 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.846 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:45.846 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.556344+0000", 2024-04-19T01:04:45.846 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:45.846 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.847 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.847 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:45.851 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.556345+0000", 2024-04-19T01:04:45.851 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1199999999999999e-07 2024-04-19T01:04:45.851 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.851 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.852 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:45.852 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.557285+0000", 2024-04-19T01:04:45.852 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00094096299999999996 2024-04-19T01:04:45.852 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.852 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.852 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:45.852 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.557335+0000", 2024-04-19T01:04:45.853 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9669000000000002e-05 2024-04-19T01:04:45.853 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.853 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.853 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:45.853 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.557358+0000", 2024-04-19T01:04:45.854 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3099999999999999e-05 2024-04-19T01:04:45.854 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.854 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.854 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:45.854 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.557490+0000", 2024-04-19T01:04:45.854 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013213400000000001 2024-04-19T01:04:45.854 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.855 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.855 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:45.855 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.557512+0000", 2024-04-19T01:04:45.856 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1603999999999998e-05 2024-04-19T01:04:45.856 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.856 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.856 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:45.856 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.586316+0000", 2024-04-19T01:04:45.857 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.028803714000000001 2024-04-19T01:04:45.857 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.857 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.857 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:45.857 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.586372+0000", 2024-04-19T01:04:45.857 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6205999999999997e-05 2024-04-19T01:04:45.858 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:45.858 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:45.858 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:45.858 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.860 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.860 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as3 49/31 ECSubWrite(tid=73, reqid=osd.12.0:73, at_version=49'12, trim_to=0'0, roll_forward_to=48'10))", 2024-04-19T01:04:45.861 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:19.530679+0000", 2024-04-19T01:04:45.862 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 26.256115336000001, 2024-04-19T01:04:45.862 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0088712419999999997, 2024-04-19T01:04:45.863 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:45.864 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:45.864 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:45.864 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:45.864 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.864 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:45.864 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.530679+0000", 2024-04-19T01:04:45.865 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:45.865 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.865 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.865 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:45.865 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.530679+0000", 2024-04-19T01:04:45.865 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:45.865 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.866 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.866 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:45.866 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.530680+0000", 2024-04-19T01:04:45.866 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7999999999999996e-07 2024-04-19T01:04:45.866 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.866 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.866 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:45.866 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.530735+0000", 2024-04-19T01:04:45.867 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.5235000000000002e-05 2024-04-19T01:04:45.867 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.867 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.867 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:45.867 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.530759+0000", 2024-04-19T01:04:45.867 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4479999999999999e-05 2024-04-19T01:04:45.867 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.867 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.867 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:45.867 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.530781+0000", 2024-04-19T01:04:45.867 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2259000000000002e-05 2024-04-19T01:04:45.867 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.867 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.867 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:45.868 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.530943+0000", 2024-04-19T01:04:45.868 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000161268 2024-04-19T01:04:45.868 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.868 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.868 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:45.868 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.530967+0000", 2024-04-19T01:04:45.868 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4499e-05 2024-04-19T01:04:45.868 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.868 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.868 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:45.868 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.539434+0000", 2024-04-19T01:04:45.868 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0084667590000000008 2024-04-19T01:04:45.868 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.868 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.869 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:45.869 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.539550+0000", 2024-04-19T01:04:45.869 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011626200000000001 2024-04-19T01:04:45.869 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:45.869 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:45.869 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:45.869 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.869 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.869 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.0s3 49/31 ECSubWrite(tid=39, reqid=osd.1.0:39, at_version=49'35, trim_to=0'0, roll_forward_to=48'33))", 2024-04-19T01:04:45.869 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:19.540342+0000", 2024-04-19T01:04:45.869 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 26.246451919999998, 2024-04-19T01:04:45.869 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0060086879999999999, 2024-04-19T01:04:45.869 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:45.869 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:45.869 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:45.870 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:45.870 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.870 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:45.870 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540342+0000", 2024-04-19T01:04:45.870 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:45.870 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.870 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.870 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:45.873 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540342+0000", 2024-04-19T01:04:45.873 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:45.873 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.874 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.874 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:45.874 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540343+0000", 2024-04-19T01:04:45.874 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4300000000000002e-07 2024-04-19T01:04:45.874 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.874 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.874 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:45.874 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540394+0000", 2024-04-19T01:04:45.874 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.1258999999999998e-05 2024-04-19T01:04:45.874 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.874 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.874 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:45.875 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540427+0000", 2024-04-19T01:04:45.875 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2898999999999998e-05 2024-04-19T01:04:45.875 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.875 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.875 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:45.875 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540450+0000", 2024-04-19T01:04:45.875 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2690000000000001e-05 2024-04-19T01:04:45.875 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.878 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.878 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:45.878 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540584+0000", 2024-04-19T01:04:45.878 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000133714 2024-04-19T01:04:45.878 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.878 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.879 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:45.879 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540604+0000", 2024-04-19T01:04:45.879 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0061e-05 2024-04-19T01:04:45.879 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.879 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.879 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:45.879 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.546254+0000", 2024-04-19T01:04:45.879 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0056502369999999998 2024-04-19T01:04:45.879 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.879 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.879 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:45.879 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.546351+0000", 2024-04-19T01:04:45.879 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7084999999999996e-05 2024-04-19T01:04:45.879 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:45.879 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:45.880 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:45.880 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.880 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.880 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as3 58/31 ECSubWrite(tid=90, reqid=osd.12.0:89, at_version=0'0, trim_to=0'0, roll_forward_to=58'18))", 2024-04-19T01:04:45.880 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:28.666810+0000", 2024-04-19T01:04:45.880 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 17.119984157000001, 2024-04-19T01:04:45.880 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0062188399999999998, 2024-04-19T01:04:45.880 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:45.880 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:45.881 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:45.881 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:45.881 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.881 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:45.881 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:28.666810+0000", 2024-04-19T01:04:45.881 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:45.881 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.881 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.881 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:45.881 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:28.666810+0000", 2024-04-19T01:04:45.881 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:45.881 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.881 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.881 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:45.882 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:28.666811+0000", 2024-04-19T01:04:45.882 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.9999999999999996e-07 2024-04-19T01:04:45.882 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.882 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.882 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:45.882 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:28.666824+0000", 2024-04-19T01:04:45.882 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.276e-05 2024-04-19T01:04:45.882 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.882 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.882 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:45.882 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:28.666841+0000", 2024-04-19T01:04:45.882 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7234000000000001e-05 2024-04-19T01:04:45.882 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.882 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.883 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:45.883 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:28.666852+0000", 2024-04-19T01:04:45.883 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0625000000000001e-05 2024-04-19T01:04:45.883 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.883 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.883 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:45.883 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:28.666919+0000", 2024-04-19T01:04:45.883 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7657000000000006e-05 2024-04-19T01:04:45.883 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.883 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.883 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:45.884 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:28.666931+0000", 2024-04-19T01:04:45.884 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1608e-05 2024-04-19T01:04:45.884 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.884 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.884 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:45.884 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:28.672986+0000", 2024-04-19T01:04:45.884 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0060553100000000004 2024-04-19T01:04:45.884 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.884 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.884 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:45.884 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:28.673029+0000", 2024-04-19T01:04:45.884 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2746e-05 2024-04-19T01:04:45.884 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:45.884 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:45.885 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:45.885 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.885 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.885 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as3 59/31 ECSubWrite(tid=91, reqid=client.4604.0:421, at_version=59'19, trim_to=0'0, roll_forward_to=58'18))", 2024-04-19T01:04:45.885 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:29.530580+0000", 2024-04-19T01:04:45.885 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 16.256214818, 2024-04-19T01:04:45.885 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0055975629999999998, 2024-04-19T01:04:45.885 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:45.885 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:45.885 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:45.885 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:45.885 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.885 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:45.885 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.530580+0000", 2024-04-19T01:04:45.886 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:45.886 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.886 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.886 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:45.886 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.530580+0000", 2024-04-19T01:04:45.886 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:45.886 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.886 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.886 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:45.886 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.530580+0000", 2024-04-19T01:04:45.886 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2399999999999997e-07 2024-04-19T01:04:45.887 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.887 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.887 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:45.887 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.530632+0000", 2024-04-19T01:04:45.887 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2111999999999998e-05 2024-04-19T01:04:45.887 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.887 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.887 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:45.888 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.530664+0000", 2024-04-19T01:04:45.888 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1155999999999998e-05 2024-04-19T01:04:45.888 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.888 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.888 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:45.888 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.530686+0000", 2024-04-19T01:04:45.888 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2093e-05 2024-04-19T01:04:45.888 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.888 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.888 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:45.888 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.530853+0000", 2024-04-19T01:04:45.888 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016714400000000001 2024-04-19T01:04:45.889 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.889 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.889 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:45.889 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.530881+0000", 2024-04-19T01:04:45.889 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7846000000000001e-05 2024-04-19T01:04:45.889 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.889 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.889 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:45.889 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.536098+0000", 2024-04-19T01:04:45.889 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0052177430000000004 2024-04-19T01:04:45.889 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.889 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.889 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:45.889 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.536177+0000", 2024-04-19T01:04:45.890 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8745000000000006e-05 2024-04-19T01:04:45.890 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:45.890 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:45.890 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:45.890 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.890 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.890 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as3 59/31 ECSubWrite(tid=92, reqid=client.4604.0:422, at_version=59'20, trim_to=0'0, roll_forward_to=58'18))", 2024-04-19T01:04:45.890 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:29.531885+0000", 2024-04-19T01:04:45.890 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 16.254909525999999, 2024-04-19T01:04:45.890 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0043512029999999997, 2024-04-19T01:04:45.890 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:45.890 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:45.890 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:45.890 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:45.891 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.891 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:45.891 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.531885+0000", 2024-04-19T01:04:45.891 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:45.891 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.891 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.891 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:45.891 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.531885+0000", 2024-04-19T01:04:45.891 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:45.891 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.891 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.891 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:45.891 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.531887+0000", 2024-04-19T01:04:45.892 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9759999999999998e-06 2024-04-19T01:04:45.892 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.892 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.892 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:45.892 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.531929+0000", 2024-04-19T01:04:45.892 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2472999999999998e-05 2024-04-19T01:04:45.892 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.892 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.892 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:45.892 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.531962+0000", 2024-04-19T01:04:45.892 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2966000000000003e-05 2024-04-19T01:04:45.892 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.892 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.892 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:45.893 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.531984+0000", 2024-04-19T01:04:45.893 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1776999999999998e-05 2024-04-19T01:04:45.893 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.893 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.893 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:45.893 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.532203+0000", 2024-04-19T01:04:45.893 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021878300000000001 2024-04-19T01:04:45.893 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.893 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.893 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:45.893 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.532230+0000", 2024-04-19T01:04:45.893 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7359e-05 2024-04-19T01:04:45.893 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.893 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.894 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:45.894 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.536184+0000", 2024-04-19T01:04:45.894 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0039542140000000002 2024-04-19T01:04:45.894 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.894 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.894 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:45.894 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.536236+0000", 2024-04-19T01:04:45.894 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.1654999999999999e-05 2024-04-19T01:04:45.894 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:45.894 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:45.894 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:45.894 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.895 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.895 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as3 59/31 ECSubWrite(tid=93, reqid=osd.12.0:93, at_version=59'22, trim_to=0'0, roll_forward_to=58'18))", 2024-04-19T01:04:45.895 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:29.533708+0000", 2024-04-19T01:04:45.895 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 16.253086830000001, 2024-04-19T01:04:45.895 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0074038869999999996, 2024-04-19T01:04:45.896 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:45.896 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:45.896 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:45.896 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:45.896 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.896 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:45.896 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.533708+0000", 2024-04-19T01:04:45.896 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:45.896 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.896 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.896 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:45.896 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.533708+0000", 2024-04-19T01:04:45.896 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:45.896 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.897 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.897 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:45.897 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.533709+0000", 2024-04-19T01:04:45.897 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.623e-06 2024-04-19T01:04:45.897 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.897 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.897 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:45.898 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.533758+0000", 2024-04-19T01:04:45.898 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9249999999999998e-05 2024-04-19T01:04:45.898 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.898 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.898 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:45.898 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.533790+0000", 2024-04-19T01:04:45.898 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1905e-05 2024-04-19T01:04:45.898 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.898 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.898 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:45.898 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.533812+0000", 2024-04-19T01:04:45.898 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.177e-05 2024-04-19T01:04:45.898 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.899 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.899 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:45.899 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.534071+0000", 2024-04-19T01:04:45.899 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00025881299999999999 2024-04-19T01:04:45.899 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.899 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.899 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:45.899 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.534097+0000", 2024-04-19T01:04:45.899 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5885999999999999e-05 2024-04-19T01:04:45.899 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.899 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.899 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:45.899 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.541069+0000", 2024-04-19T01:04:45.899 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.006971728 2024-04-19T01:04:45.899 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.900 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.900 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:45.900 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.541111+0000", 2024-04-19T01:04:45.900 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2911999999999998e-05 2024-04-19T01:04:45.900 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:45.900 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:45.900 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:45.900 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.900 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.900 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.1s2 64/31 ECSubWrite(tid=28, reqid=osd.6.0:28, at_version=64'20, trim_to=0'0, roll_forward_to=62'18))", 2024-04-19T01:04:45.900 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:34.571701+0000", 2024-04-19T01:04:45.900 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 11.215093545, 2024-04-19T01:04:45.900 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0055527429999999997, 2024-04-19T01:04:45.900 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:45.901 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:45.901 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:45.901 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:45.901 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.901 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:45.901 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:34.571701+0000", 2024-04-19T01:04:45.901 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:45.902 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.902 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.902 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:45.902 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:34.571701+0000", 2024-04-19T01:04:45.902 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:45.902 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.902 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.902 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:45.902 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:34.571702+0000", 2024-04-19T01:04:45.902 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.9999999999999997e-07 2024-04-19T01:04:45.902 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.903 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.903 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:45.903 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:34.571756+0000", 2024-04-19T01:04:45.903 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4951999999999998e-05 2024-04-19T01:04:45.903 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.903 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.903 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:45.903 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:34.571799+0000", 2024-04-19T01:04:45.903 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2823e-05 2024-04-19T01:04:45.903 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.903 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.903 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:45.903 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:34.571827+0000", 2024-04-19T01:04:45.904 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7906e-05 2024-04-19T01:04:45.904 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.904 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.904 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:45.904 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:34.572010+0000", 2024-04-19T01:04:45.904 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00018268000000000001 2024-04-19T01:04:45.904 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.904 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.904 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:45.904 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:34.572038+0000", 2024-04-19T01:04:45.904 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7880000000000001e-05 2024-04-19T01:04:45.904 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.904 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.905 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:45.905 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:34.577163+0000", 2024-04-19T01:04:45.905 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0051248090000000001 2024-04-19T01:04:45.905 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.905 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.905 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:45.905 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:34.577254+0000", 2024-04-19T01:04:45.905 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.0993000000000001e-05 2024-04-19T01:04:45.905 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:45.905 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:45.905 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:45.905 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.905 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.905 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as3 65/31 ECSubWrite(tid=107, reqid=client.4604.0:478, at_version=65'26, trim_to=0'0, roll_forward_to=64'24))", 2024-04-19T01:04:45.906 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:35.638966+0000", 2024-04-19T01:04:45.906 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.147828663, 2024-04-19T01:04:45.906 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.01558386, 2024-04-19T01:04:45.906 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:45.906 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:45.906 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:45.906 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:45.906 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.906 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:45.906 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.638966+0000", 2024-04-19T01:04:45.906 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:45.906 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.906 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.906 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:45.907 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.638966+0000", 2024-04-19T01:04:45.907 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:45.907 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.907 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.907 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:45.907 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.638967+0000", 2024-04-19T01:04:45.907 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2500000000000001e-06 2024-04-19T01:04:45.907 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.907 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.907 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:45.907 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.639799+0000", 2024-04-19T01:04:45.908 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00083235399999999999 2024-04-19T01:04:45.908 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.908 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.908 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:45.908 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.639828+0000", 2024-04-19T01:04:45.908 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8513e-05 2024-04-19T01:04:45.908 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.908 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.908 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:45.908 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.639854+0000", 2024-04-19T01:04:45.908 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5769999999999999e-05 2024-04-19T01:04:45.909 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.909 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.909 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:45.909 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.640035+0000", 2024-04-19T01:04:45.909 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000181714 2024-04-19T01:04:45.909 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.909 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.909 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:45.909 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.640070+0000", 2024-04-19T01:04:45.909 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5179000000000003e-05 2024-04-19T01:04:45.909 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.909 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.909 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:45.910 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.654477+0000", 2024-04-19T01:04:45.910 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.014406402 2024-04-19T01:04:45.910 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.910 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.910 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:45.910 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.654550+0000", 2024-04-19T01:04:45.910 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2677999999999998e-05 2024-04-19T01:04:45.910 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:45.910 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:45.910 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:45.910 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.910 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.910 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as3 65/31 ECSubWrite(tid=108, reqid=client.4604.0:479, at_version=65'27, trim_to=0'0, roll_forward_to=64'24))", 2024-04-19T01:04:45.910 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:35.655152+0000", 2024-04-19T01:04:45.911 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.131642368, 2024-04-19T01:04:45.911 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.010020193, 2024-04-19T01:04:45.911 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:45.911 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:45.911 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:45.911 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:45.911 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.911 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:45.911 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.655152+0000", 2024-04-19T01:04:45.911 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:45.911 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.911 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.911 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:45.911 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.655152+0000", 2024-04-19T01:04:45.912 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:45.912 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.912 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.912 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:45.912 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.655154+0000", 2024-04-19T01:04:45.912 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.883e-06 2024-04-19T01:04:45.912 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.912 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.912 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:45.912 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.655409+0000", 2024-04-19T01:04:45.912 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000254827 2024-04-19T01:04:45.912 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.912 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.912 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:45.912 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.655437+0000", 2024-04-19T01:04:45.913 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8085999999999998e-05 2024-04-19T01:04:45.913 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.913 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.913 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:45.913 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.655455+0000", 2024-04-19T01:04:45.913 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8413000000000001e-05 2024-04-19T01:04:45.913 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.913 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.913 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:45.913 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.655610+0000", 2024-04-19T01:04:45.913 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000155005 2024-04-19T01:04:45.913 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.913 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.913 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:45.914 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.655636+0000", 2024-04-19T01:04:45.914 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6118999999999998e-05 2024-04-19T01:04:45.914 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.914 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.914 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:45.914 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.665068+0000", 2024-04-19T01:04:45.914 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0094318170000000003 2024-04-19T01:04:45.914 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.914 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.914 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:45.914 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.665172+0000", 2024-04-19T01:04:45.914 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010404300000000001 2024-04-19T01:04:45.914 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:45.918 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:45.918 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:45.918 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.918 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.919 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as3 65/31 ECSubWrite(tid=109, reqid=client.4604.0:480, at_version=65'28, trim_to=0'0, roll_forward_to=65'26))", 2024-04-19T01:04:45.919 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:35.666249+0000", 2024-04-19T01:04:45.919 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.120545464999999, 2024-04-19T01:04:45.919 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0091825489999999999, 2024-04-19T01:04:45.919 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:45.919 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:45.920 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:45.920 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:45.920 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.920 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:45.920 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.666249+0000", 2024-04-19T01:04:45.920 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:45.921 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.921 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.921 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:45.921 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.666249+0000", 2024-04-19T01:04:45.921 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:45.923 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.923 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.926 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:45.927 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.666250+0000", 2024-04-19T01:04:45.927 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5e-06 2024-04-19T01:04:45.927 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.927 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.927 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:45.927 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.666372+0000", 2024-04-19T01:04:45.927 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000121932 2024-04-19T01:04:45.927 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.927 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.927 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:45.928 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.666406+0000", 2024-04-19T01:04:45.928 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3810000000000003e-05 2024-04-19T01:04:45.928 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.928 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.928 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:45.928 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.666422+0000", 2024-04-19T01:04:45.928 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6175999999999999e-05 2024-04-19T01:04:45.928 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.928 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.928 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:45.928 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.666575+0000", 2024-04-19T01:04:45.929 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000152671 2024-04-19T01:04:45.929 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.929 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.929 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:45.929 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.666593+0000", 2024-04-19T01:04:45.929 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7586000000000001e-05 2024-04-19T01:04:45.929 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.929 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.929 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:45.929 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.675355+0000", 2024-04-19T01:04:45.929 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0087623149999999997 2024-04-19T01:04:45.929 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.929 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.929 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:45.930 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.675431+0000", 2024-04-19T01:04:45.930 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6558999999999997e-05 2024-04-19T01:04:45.930 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:45.930 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:45.930 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:45.930 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.930 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.930 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.2s5 74/31 ECSubWrite(tid=246, reqid=osd.11.0:246, at_version=74'18, trim_to=0'0, roll_forward_to=72'16))", 2024-04-19T01:04:45.930 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:44.560594+0000", 2024-04-19T01:04:45.930 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 1.2261998709999999, 2024-04-19T01:04:45.930 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0044402909999999999, 2024-04-19T01:04:45.930 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:45.930 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:45.931 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:45.931 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:45.931 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.931 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:45.931 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.560594+0000", 2024-04-19T01:04:45.931 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:45.931 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.931 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.931 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:45.931 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.560594+0000", 2024-04-19T01:04:45.931 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:45.931 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.932 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.932 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:45.932 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.560595+0000", 2024-04-19T01:04:45.932 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5700000000000002e-07 2024-04-19T01:04:45.932 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.932 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.932 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:45.932 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.560654+0000", 2024-04-19T01:04:45.932 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8842999999999999e-05 2024-04-19T01:04:45.932 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.932 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.932 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:45.932 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.560684+0000", 2024-04-19T01:04:45.932 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0386e-05 2024-04-19T01:04:45.933 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.933 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.933 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:45.933 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.560757+0000", 2024-04-19T01:04:45.933 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2238999999999997e-05 2024-04-19T01:04:45.933 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.933 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.933 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:45.933 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.561032+0000", 2024-04-19T01:04:45.933 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00027519300000000002 2024-04-19T01:04:45.933 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.933 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.933 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:45.933 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.561066+0000", 2024-04-19T01:04:45.933 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3769999999999997e-05 2024-04-19T01:04:45.934 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.934 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.934 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:45.934 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.564952+0000", 2024-04-19T01:04:45.934 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0038861709999999999 2024-04-19T01:04:45.934 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.934 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.934 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:45.934 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.565035+0000", 2024-04-19T01:04:45.934 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.3032000000000001e-05 2024-04-19T01:04:45.934 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:45.934 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:45.934 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:45.935 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.935 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.935 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.2s5 74/31 ECSubWrite(tid=247, reqid=osd.11.0:246, at_version=0'0, trim_to=0'0, roll_forward_to=74'18))", 2024-04-19T01:04:45.935 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:44.572861+0000", 2024-04-19T01:04:45.935 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 1.2139332999999999, 2024-04-19T01:04:45.935 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0090086430000000002, 2024-04-19T01:04:45.935 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:45.935 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:45.935 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:45.935 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:45.935 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.935 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:45.935 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.572861+0000", 2024-04-19T01:04:45.935 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:45.935 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.936 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.936 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:45.936 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.572861+0000", 2024-04-19T01:04:45.936 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:45.936 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.936 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.936 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:45.936 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.572863+0000", 2024-04-19T01:04:45.936 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0169999999999999e-06 2024-04-19T01:04:45.936 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.936 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.936 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:45.936 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.572892+0000", 2024-04-19T01:04:45.936 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9062999999999999e-05 2024-04-19T01:04:45.937 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.937 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.937 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:45.937 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.572923+0000", 2024-04-19T01:04:45.937 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0969000000000001e-05 2024-04-19T01:04:45.937 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.937 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.937 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:45.937 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.572947+0000", 2024-04-19T01:04:45.937 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4223000000000002e-05 2024-04-19T01:04:45.937 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.937 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.937 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:45.937 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.573120+0000", 2024-04-19T01:04:45.937 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017283500000000001 2024-04-19T01:04:45.938 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.938 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.938 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:45.938 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.573146+0000", 2024-04-19T01:04:45.938 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6106e-05 2024-04-19T01:04:45.938 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.938 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.939 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:45.939 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.581785+0000", 2024-04-19T01:04:45.939 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0086389110000000009 2024-04-19T01:04:45.939 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:45.939 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:45.939 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:45.939 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.581870+0000", 2024-04-19T01:04:45.939 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4518999999999995e-05 2024-04-19T01:04:45.939 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:45.940 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:45.940 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:45.940 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:45.940 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:45.940 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:04:45.942 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-19T01:04:45.943 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:04:45.940+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:04:46.043 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:46.043+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:46.113 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:04:46.114 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:04:46.114 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-04-19T01:04:46.114 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:04:46.127 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-19T01:04:46.145 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:04:46.142+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:04:46.231 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-19T01:04:46.245 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:04:46.245 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:04:46.245 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-04-19T01:04:46.245 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-04-19T01:04:46.245 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:04:46.246 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:04:46.244+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:04:46.259 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-19T01:04:46.346 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:04:46.346+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:04:46.390 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-04-19T01:04:46.390 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T01:04:46.410 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-04-19T01:04:46.411 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T01:04:46.429 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:04:46.430 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-04-19T01:04:46.430 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-04-19T01:04:46.430 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-04-19T01:04:46.430 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.430 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 43/31 ECSubWrite(tid=47, reqid=client.4604.0:263, at_version=43'8, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:04:46.430 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:13.999456+0000", 2024-04-19T01:04:46.430 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 32.429410687999997, 2024-04-19T01:04:46.430 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0090503839999999999, 2024-04-19T01:04:46.430 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:46.430 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:46.430 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:46.430 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:46.431 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.431 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:46.431 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.999456+0000", 2024-04-19T01:04:46.431 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:46.431 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.431 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.431 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:46.431 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.999456+0000", 2024-04-19T01:04:46.431 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:46.431 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.431 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.431 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:46.431 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.999458+0000", 2024-04-19T01:04:46.431 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.874e-06 2024-04-19T01:04:46.431 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.432 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.432 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:46.432 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.999813+0000", 2024-04-19T01:04:46.432 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00035491100000000002 2024-04-19T01:04:46.432 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.432 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.432 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:46.432 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.999832+0000", 2024-04-19T01:04:46.432 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9425000000000001e-05 2024-04-19T01:04:46.432 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.432 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.432 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:46.432 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.999844+0000", 2024-04-19T01:04:46.432 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1838000000000001e-05 2024-04-19T01:04:46.433 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.433 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.433 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:46.433 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.999950+0000", 2024-04-19T01:04:46.433 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000106125 2024-04-19T01:04:46.433 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.433 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.433 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:46.433 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.999965+0000", 2024-04-19T01:04:46.433 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4966e-05 2024-04-19T01:04:46.433 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.433 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.433 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:46.433 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.008459+0000", 2024-04-19T01:04:46.434 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0084938979999999997 2024-04-19T01:04:46.434 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.434 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.434 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:46.434 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.008506+0000", 2024-04-19T01:04:46.434 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7346999999999999e-05 2024-04-19T01:04:46.434 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:46.434 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:46.434 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:46.434 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.434 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.434 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 43/31 ECSubWrite(tid=48, reqid=client.4604.0:264, at_version=43'9, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:04:46.434 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:14.001305+0000", 2024-04-19T01:04:46.434 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 32.427561734000001, 2024-04-19T01:04:46.435 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0076904640000000002, 2024-04-19T01:04:46.435 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:46.435 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:46.435 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:46.435 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:46.435 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.435 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:46.435 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.001305+0000", 2024-04-19T01:04:46.435 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:46.435 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.435 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.435 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:46.435 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.001305+0000", 2024-04-19T01:04:46.435 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:46.435 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.436 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.436 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:46.436 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.001306+0000", 2024-04-19T01:04:46.436 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5700000000000001e-07 2024-04-19T01:04:46.438 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.438 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.438 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:46.438 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.001417+0000", 2024-04-19T01:04:46.438 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011103900000000001 2024-04-19T01:04:46.438 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.439 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.439 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:46.439 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.001434+0000", 2024-04-19T01:04:46.439 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7075000000000002e-05 2024-04-19T01:04:46.439 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.439 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.439 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:46.439 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.001442+0000", 2024-04-19T01:04:46.439 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6930000000000007e-06 2024-04-19T01:04:46.439 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.439 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.439 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:46.439 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.005465+0000", 2024-04-19T01:04:46.439 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0040232990000000001 2024-04-19T01:04:46.440 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.440 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.440 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:46.440 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.005476+0000", 2024-04-19T01:04:46.440 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0781999999999999e-05 2024-04-19T01:04:46.440 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.440 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.440 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:46.440 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.008961+0000", 2024-04-19T01:04:46.440 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0034857299999999998 2024-04-19T01:04:46.440 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.440 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.440 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:46.441 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.008995+0000", 2024-04-19T01:04:46.441 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3989000000000003e-05 2024-04-19T01:04:46.441 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:46.441 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:46.441 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:46.441 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.441 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.441 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 43/31 ECSubWrite(tid=49, reqid=client.4604.0:265, at_version=43'10, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:04:46.441 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:14.004792+0000", 2024-04-19T01:04:46.441 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 32.424075086999999, 2024-04-19T01:04:46.442 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0064878169999999999, 2024-04-19T01:04:46.442 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:46.442 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:46.442 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:46.442 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:46.442 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.442 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:46.442 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.004792+0000", 2024-04-19T01:04:46.442 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:46.442 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.442 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.442 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:46.442 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.004792+0000", 2024-04-19T01:04:46.442 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:46.443 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.443 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.443 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:46.443 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.004792+0000", 2024-04-19T01:04:46.443 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2299999999999999e-07 2024-04-19T01:04:46.443 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.443 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.443 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:46.443 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.004821+0000", 2024-04-19T01:04:46.443 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8365e-05 2024-04-19T01:04:46.443 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.443 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.443 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:46.443 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.004836+0000", 2024-04-19T01:04:46.444 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4894e-05 2024-04-19T01:04:46.444 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.444 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.444 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:46.444 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.004846+0000", 2024-04-19T01:04:46.444 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0579e-05 2024-04-19T01:04:46.444 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.444 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.444 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:46.444 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.008569+0000", 2024-04-19T01:04:46.444 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0037230079999999999 2024-04-19T01:04:46.444 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.444 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.444 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:46.444 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.008582+0000", 2024-04-19T01:04:46.445 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2330000000000001e-05 2024-04-19T01:04:46.445 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.445 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.445 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:46.445 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.011248+0000", 2024-04-19T01:04:46.445 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0026668220000000001 2024-04-19T01:04:46.445 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.445 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.445 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:46.445 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.011279+0000", 2024-04-19T01:04:46.445 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0895999999999997e-05 2024-04-19T01:04:46.445 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:46.445 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:46.445 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:46.446 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.446 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.446 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 44/31 ECSubWrite(tid=53, reqid=osd.0.0:53, at_version=44'12, trim_to=0'0, roll_forward_to=43'10))", 2024-04-19T01:04:46.446 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:14.887273+0000", 2024-04-19T01:04:46.446 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 31.541593980999998, 2024-04-19T01:04:46.446 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0081171100000000003, 2024-04-19T01:04:46.446 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:46.447 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:46.447 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:46.447 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:46.447 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.447 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:46.447 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.887273+0000", 2024-04-19T01:04:46.447 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:46.447 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.447 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.447 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:46.447 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.887273+0000", 2024-04-19T01:04:46.447 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:46.447 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.448 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.448 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:46.448 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.887273+0000", 2024-04-19T01:04:46.448 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8999999999999995e-07 2024-04-19T01:04:46.448 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.448 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.448 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:46.448 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.887331+0000", 2024-04-19T01:04:46.448 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7905000000000002e-05 2024-04-19T01:04:46.448 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.448 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.448 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:46.448 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.887362+0000", 2024-04-19T01:04:46.448 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1086e-05 2024-04-19T01:04:46.449 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.449 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.449 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:46.449 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.887385+0000", 2024-04-19T01:04:46.449 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2843000000000001e-05 2024-04-19T01:04:46.449 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.449 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.449 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:46.449 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.887605+0000", 2024-04-19T01:04:46.449 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021989899999999999 2024-04-19T01:04:46.449 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.449 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.449 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:46.449 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.887631+0000", 2024-04-19T01:04:46.450 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6225999999999999e-05 2024-04-19T01:04:46.450 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.450 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.450 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:46.450 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.895300+0000", 2024-04-19T01:04:46.450 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.007668467 2024-04-19T01:04:46.450 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.450 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.450 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:46.450 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.895390+0000", 2024-04-19T01:04:46.450 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.9894000000000004e-05 2024-04-19T01:04:46.450 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:46.450 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:46.450 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:46.451 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.451 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.451 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 47/31 ECSubWrite(tid=61, reqid=client.4604.0:317, at_version=47'16, trim_to=0'0, roll_forward_to=45'14))", 2024-04-19T01:04:46.451 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:17.604486+0000", 2024-04-19T01:04:46.451 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 28.824381138, 2024-04-19T01:04:46.451 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.025312589, 2024-04-19T01:04:46.451 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:46.451 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:46.451 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:46.451 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:46.452 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.452 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:46.452 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.604486+0000", 2024-04-19T01:04:46.452 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:46.452 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.452 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.452 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:46.452 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.604486+0000", 2024-04-19T01:04:46.452 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:46.452 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.452 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.452 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:46.452 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.604486+0000", 2024-04-19T01:04:46.452 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.9999999999999997e-07 2024-04-19T01:04:46.453 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.453 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.453 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:46.453 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.604960+0000", 2024-04-19T01:04:46.453 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00047350300000000001 2024-04-19T01:04:46.453 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.453 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.453 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:46.453 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.605002+0000", 2024-04-19T01:04:46.453 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1949000000000001e-05 2024-04-19T01:04:46.453 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.453 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.453 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:46.453 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.605032+0000", 2024-04-19T01:04:46.454 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9865999999999998e-05 2024-04-19T01:04:46.454 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.454 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.454 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:46.454 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.605289+0000", 2024-04-19T01:04:46.454 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00025766500000000002 2024-04-19T01:04:46.454 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.454 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.454 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:46.454 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.605322+0000", 2024-04-19T01:04:46.454 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2845999999999997e-05 2024-04-19T01:04:46.454 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.454 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.455 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:46.455 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.629741+0000", 2024-04-19T01:04:46.455 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.024419335 2024-04-19T01:04:46.455 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.455 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.455 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:46.455 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.629798+0000", 2024-04-19T01:04:46.455 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6725000000000003e-05 2024-04-19T01:04:46.455 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:46.455 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:46.455 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:46.455 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.455 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.455 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 47/31 ECSubWrite(tid=62, reqid=client.4604.0:318, at_version=47'17, trim_to=0'0, roll_forward_to=45'14))", 2024-04-19T01:04:46.456 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:17.611532+0000", 2024-04-19T01:04:46.456 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 28.817334224, 2024-04-19T01:04:46.456 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.018327896, 2024-04-19T01:04:46.456 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:46.456 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:46.456 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:46.456 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:46.456 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.456 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:46.456 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.611532+0000", 2024-04-19T01:04:46.456 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:46.456 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.456 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.456 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:46.457 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.611532+0000", 2024-04-19T01:04:46.457 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:46.457 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.457 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.457 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:46.457 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.611534+0000", 2024-04-19T01:04:46.457 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.979e-06 2024-04-19T01:04:46.457 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.457 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.457 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:46.457 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.611863+0000", 2024-04-19T01:04:46.457 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00032853100000000002 2024-04-19T01:04:46.457 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.457 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.458 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:46.458 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.611907+0000", 2024-04-19T01:04:46.458 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4425000000000002e-05 2024-04-19T01:04:46.458 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.458 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.458 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:46.458 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.611935+0000", 2024-04-19T01:04:46.458 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7993000000000001e-05 2024-04-19T01:04:46.458 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.458 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.458 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:46.458 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.617071+0000", 2024-04-19T01:04:46.458 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0051354069999999998 2024-04-19T01:04:46.458 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.459 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.459 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:46.459 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.617091+0000", 2024-04-19T01:04:46.459 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0061999999999999e-05 2024-04-19T01:04:46.459 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.459 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.459 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:46.459 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.629827+0000", 2024-04-19T01:04:46.459 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.012735943 2024-04-19T01:04:46.459 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.459 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.459 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:46.459 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.629860+0000", 2024-04-19T01:04:46.459 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3556000000000002e-05 2024-04-19T01:04:46.460 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:46.460 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:46.460 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:46.460 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.460 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.460 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 47/31 ECSubWrite(tid=63, reqid=client.4604.0:319, at_version=47'18, trim_to=0'0, roll_forward_to=45'14))", 2024-04-19T01:04:46.460 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:17.621137+0000", 2024-04-19T01:04:46.460 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 28.807729976000001, 2024-04-19T01:04:46.460 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0092948300000000005, 2024-04-19T01:04:46.460 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:46.460 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:46.460 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:46.460 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:46.460 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.460 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:46.461 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.621137+0000", 2024-04-19T01:04:46.461 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:46.461 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.461 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.461 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:46.461 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.621137+0000", 2024-04-19T01:04:46.461 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:46.461 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.461 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.461 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:46.461 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.621139+0000", 2024-04-19T01:04:46.461 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3130000000000001e-06 2024-04-19T01:04:46.461 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.462 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.462 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:46.462 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.621276+0000", 2024-04-19T01:04:46.462 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013658400000000001 2024-04-19T01:04:46.462 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.462 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.462 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:46.462 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.621317+0000", 2024-04-19T01:04:46.462 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1152000000000002e-05 2024-04-19T01:04:46.462 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.462 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.462 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:46.462 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.621341+0000", 2024-04-19T01:04:46.462 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4600000000000002e-05 2024-04-19T01:04:46.463 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.464 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.464 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:46.464 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.627256+0000", 2024-04-19T01:04:46.464 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0059141890000000003 2024-04-19T01:04:46.464 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.464 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.464 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:46.464 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.627268+0000", 2024-04-19T01:04:46.464 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2903e-05 2024-04-19T01:04:46.464 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.464 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.464 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:46.466 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.630366+0000", 2024-04-19T01:04:46.466 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0030979879999999999 2024-04-19T01:04:46.466 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.466 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.466 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:46.466 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.630432+0000", 2024-04-19T01:04:46.467 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5100999999999999e-05 2024-04-19T01:04:46.467 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:46.467 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:46.467 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:46.467 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.467 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.467 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 47/31 ECSubWrite(tid=64, reqid=client.4604.0:320, at_version=47'19, trim_to=0'0, roll_forward_to=45'14))", 2024-04-19T01:04:46.467 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:17.623031+0000", 2024-04-19T01:04:46.467 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 28.805835242000001, 2024-04-19T01:04:46.468 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0081197209999999999, 2024-04-19T01:04:46.468 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:46.468 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:46.469 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:46.469 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:46.469 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.469 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:46.469 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.623031+0000", 2024-04-19T01:04:46.469 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:46.469 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.469 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.469 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:46.470 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.623031+0000", 2024-04-19T01:04:46.470 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:46.470 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.470 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.470 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:46.470 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.623033+0000", 2024-04-19T01:04:46.470 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.694e-06 2024-04-19T01:04:46.470 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.471 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.471 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:46.471 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.623067+0000", 2024-04-19T01:04:46.471 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3436999999999999e-05 2024-04-19T01:04:46.471 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.471 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.471 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:46.474 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.623093+0000", 2024-04-19T01:04:46.474 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6469999999999999e-05 2024-04-19T01:04:46.474 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.474 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.474 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:46.474 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.623114+0000", 2024-04-19T01:04:46.475 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1423000000000001e-05 2024-04-19T01:04:46.475 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.475 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.475 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:46.475 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.629946+0000", 2024-04-19T01:04:46.475 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0068313330000000002 2024-04-19T01:04:46.475 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.475 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.475 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:46.475 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.629963+0000", 2024-04-19T01:04:46.475 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7289e-05 2024-04-19T01:04:46.475 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.476 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.476 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:46.476 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.631074+0000", 2024-04-19T01:04:46.476 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011109570000000001 2024-04-19T01:04:46.476 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.476 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.476 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:46.476 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.631151+0000", 2024-04-19T01:04:46.476 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7118000000000003e-05 2024-04-19T01:04:46.476 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:46.476 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:46.476 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:46.476 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.477 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.477 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 48/31 ECSubWrite(tid=75, reqid=osd.0.0:75, at_version=48'21, trim_to=0'0, roll_forward_to=47'19))", 2024-04-19T01:04:46.477 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.528567+0000", 2024-04-19T01:04:46.477 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 27.900299405999998, 2024-04-19T01:04:46.477 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0061716480000000001, 2024-04-19T01:04:46.477 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:46.477 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:46.477 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:46.477 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:46.477 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.477 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:46.477 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.528567+0000", 2024-04-19T01:04:46.477 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:46.477 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.478 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.478 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:46.478 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.528567+0000", 2024-04-19T01:04:46.478 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:46.478 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.478 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.478 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:46.478 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.528568+0000", 2024-04-19T01:04:46.478 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0999999999999998e-07 2024-04-19T01:04:46.478 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.478 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.478 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:46.478 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.528627+0000", 2024-04-19T01:04:46.478 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8745000000000001e-05 2024-04-19T01:04:46.479 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.479 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.479 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:46.479 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.528665+0000", 2024-04-19T01:04:46.479 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8479000000000002e-05 2024-04-19T01:04:46.479 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.479 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.479 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:46.479 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.528694+0000", 2024-04-19T01:04:46.479 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8770000000000001e-05 2024-04-19T01:04:46.479 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.479 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.479 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:46.480 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.528825+0000", 2024-04-19T01:04:46.480 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013142500000000001 2024-04-19T01:04:46.480 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.480 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.480 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:46.480 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.528840+0000", 2024-04-19T01:04:46.480 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4436999999999999e-05 2024-04-19T01:04:46.481 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.481 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.481 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:46.481 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.534649+0000", 2024-04-19T01:04:46.481 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0058094840000000002 2024-04-19T01:04:46.481 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.481 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.481 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:46.481 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.534739+0000", 2024-04-19T01:04:46.482 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.9598000000000005e-05 2024-04-19T01:04:46.482 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:46.482 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:46.482 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:46.482 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.482 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.482 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 48/31 ECSubWrite(tid=76, reqid=client.4604.0:345, at_version=48'22, trim_to=0'0, roll_forward_to=47'19))", 2024-04-19T01:04:46.482 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.537945+0000", 2024-04-19T01:04:46.482 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 27.890921248000001, 2024-04-19T01:04:46.482 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.034986066000000003, 2024-04-19T01:04:46.482 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:46.482 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:46.483 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:46.483 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:46.483 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.483 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:46.483 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.537945+0000", 2024-04-19T01:04:46.483 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:46.483 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.483 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.483 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:46.484 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.537945+0000", 2024-04-19T01:04:46.484 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:46.484 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.484 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.484 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:46.484 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.537948+0000", 2024-04-19T01:04:46.484 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6929999999999999e-06 2024-04-19T01:04:46.484 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.484 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.484 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:46.484 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.538485+0000", 2024-04-19T01:04:46.484 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00053661900000000001 2024-04-19T01:04:46.484 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.485 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.485 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:46.485 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.538530+0000", 2024-04-19T01:04:46.485 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5423000000000002e-05 2024-04-19T01:04:46.485 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.485 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.485 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:46.485 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.538558+0000", 2024-04-19T01:04:46.485 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7569999999999999e-05 2024-04-19T01:04:46.485 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.485 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.485 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:46.485 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.538778+0000", 2024-04-19T01:04:46.485 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000220478 2024-04-19T01:04:46.485 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.486 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.486 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:46.486 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.538806+0000", 2024-04-19T01:04:46.486 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8197000000000001e-05 2024-04-19T01:04:46.486 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.486 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.486 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:46.486 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.572854+0000", 2024-04-19T01:04:46.486 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.034047315000000002 2024-04-19T01:04:46.486 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.486 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.486 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:46.486 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.572931+0000", 2024-04-19T01:04:46.486 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7770999999999998e-05 2024-04-19T01:04:46.487 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:46.487 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:46.487 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:46.487 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.487 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.487 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 48/31 ECSubWrite(tid=78, reqid=client.4604.0:346, at_version=48'23, trim_to=0'0, roll_forward_to=48'21))", 2024-04-19T01:04:46.487 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.548797+0000", 2024-04-19T01:04:46.487 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 27.880069183, 2024-04-19T01:04:46.487 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.024207922, 2024-04-19T01:04:46.487 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:46.487 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:46.487 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:46.487 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:46.487 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.488 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:46.488 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.548797+0000", 2024-04-19T01:04:46.488 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:46.488 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.488 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.488 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:46.488 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.548797+0000", 2024-04-19T01:04:46.488 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:46.488 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.488 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.488 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:46.488 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.548799+0000", 2024-04-19T01:04:46.488 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.604e-06 2024-04-19T01:04:46.488 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.489 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.489 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:46.489 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.548991+0000", 2024-04-19T01:04:46.489 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019153400000000001 2024-04-19T01:04:46.489 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.489 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.489 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:46.489 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.549011+0000", 2024-04-19T01:04:46.489 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0332000000000001e-05 2024-04-19T01:04:46.489 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.489 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.489 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:46.489 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.549025+0000", 2024-04-19T01:04:46.490 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4260000000000001e-05 2024-04-19T01:04:46.490 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.490 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.490 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:46.490 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.552893+0000", 2024-04-19T01:04:46.490 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0038674460000000001 2024-04-19T01:04:46.490 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.490 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.490 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:46.490 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.552906+0000", 2024-04-19T01:04:46.490 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3105000000000001e-05 2024-04-19T01:04:46.490 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.490 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.490 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:46.491 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.572950+0000", 2024-04-19T01:04:46.491 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.020043787 2024-04-19T01:04:46.491 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.491 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.491 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:46.491 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.573005+0000", 2024-04-19T01:04:46.491 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.5853999999999998e-05 2024-04-19T01:04:46.491 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:46.491 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:46.491 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:46.491 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.491 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.491 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 48/31 ECSubWrite(tid=79, reqid=client.4604.0:347, at_version=48'24, trim_to=0'0, roll_forward_to=48'21))", 2024-04-19T01:04:46.491 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.555132+0000", 2024-04-19T01:04:46.492 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 27.873734751000001, 2024-04-19T01:04:46.492 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.018366424999999999, 2024-04-19T01:04:46.492 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:46.492 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:46.492 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:46.492 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:46.492 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.492 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:46.492 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.555132+0000", 2024-04-19T01:04:46.492 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:46.492 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.492 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.493 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:46.496 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.555132+0000", 2024-04-19T01:04:46.498 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:46.498 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.498 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.498 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:46.498 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.555133+0000", 2024-04-19T01:04:46.498 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.42e-06 2024-04-19T01:04:46.498 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.498 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.498 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:46.499 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.555208+0000", 2024-04-19T01:04:46.499 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4532999999999999e-05 2024-04-19T01:04:46.499 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.499 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.499 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:46.500 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.555254+0000", 2024-04-19T01:04:46.500 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6031e-05 2024-04-19T01:04:46.500 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.500 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.500 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:46.500 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.555268+0000", 2024-04-19T01:04:46.500 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4068e-05 2024-04-19T01:04:46.500 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.500 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.500 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:46.500 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.571711+0000", 2024-04-19T01:04:46.500 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.016442656999999999 2024-04-19T01:04:46.500 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.500 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.501 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:46.501 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.571720+0000", 2024-04-19T01:04:46.501 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1370000000000008e-06 2024-04-19T01:04:46.501 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.501 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.501 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:46.501 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.573461+0000", 2024-04-19T01:04:46.501 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0017408180000000001 2024-04-19T01:04:46.501 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.501 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.501 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:46.501 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.573498+0000", 2024-04-19T01:04:46.501 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7761e-05 2024-04-19T01:04:46.502 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:46.502 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:46.502 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:46.502 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.502 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.502 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as2 48/31 ECSubWrite(tid=71, reqid=client.4604.0:354, at_version=48'10, trim_to=0'0, roll_forward_to=34'8))", 2024-04-19T01:04:46.502 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.556963+0000", 2024-04-19T01:04:46.502 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 27.871904033, 2024-04-19T01:04:46.502 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.029734679, 2024-04-19T01:04:46.502 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:46.502 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:46.502 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:46.502 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:46.502 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.503 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:46.503 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.556963+0000", 2024-04-19T01:04:46.503 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:46.503 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.503 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.503 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:46.503 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.556963+0000", 2024-04-19T01:04:46.503 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:46.503 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.503 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.503 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:46.503 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.556963+0000", 2024-04-19T01:04:46.503 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5499999999999998e-07 2024-04-19T01:04:46.503 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.514 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.514 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:46.514 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.558128+0000", 2024-04-19T01:04:46.522 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011650180000000001 2024-04-19T01:04:46.522 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.522 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.522 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:46.522 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.558199+0000", 2024-04-19T01:04:46.522 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0630999999999995e-05 2024-04-19T01:04:46.523 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.523 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.523 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:46.523 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.558221+0000", 2024-04-19T01:04:46.523 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2546999999999999e-05 2024-04-19T01:04:46.523 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.523 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.523 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:46.523 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.558350+0000", 2024-04-19T01:04:46.523 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012838699999999999 2024-04-19T01:04:46.523 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.523 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.523 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:46.523 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.558370+0000", 2024-04-19T01:04:46.524 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0106999999999999e-05 2024-04-19T01:04:46.524 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.524 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.524 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:46.524 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.586643+0000", 2024-04-19T01:04:46.524 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.028273007999999999 2024-04-19T01:04:46.524 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.524 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.524 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:46.524 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.586697+0000", 2024-04-19T01:04:46.524 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4326000000000001e-05 2024-04-19T01:04:46.524 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:46.524 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:46.524 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:46.525 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.525 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.525 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 48/31 ECSubWrite(tid=80, reqid=client.4604.0:348, at_version=48'25, trim_to=0'0, roll_forward_to=48'21))", 2024-04-19T01:04:46.525 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.559574+0000", 2024-04-19T01:04:46.525 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 27.869292761000001, 2024-04-19T01:04:46.525 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.016989658000000001, 2024-04-19T01:04:46.525 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:46.525 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:46.525 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:46.525 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:46.525 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.525 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:46.525 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.559574+0000", 2024-04-19T01:04:46.525 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:46.525 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.526 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.529 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:46.529 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.559574+0000", 2024-04-19T01:04:46.529 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:46.529 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.529 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.529 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:46.529 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.559574+0000", 2024-04-19T01:04:46.530 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9400000000000001e-07 2024-04-19T01:04:46.530 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.530 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.530 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:46.530 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.559589+0000", 2024-04-19T01:04:46.530 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4873e-05 2024-04-19T01:04:46.530 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.530 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.530 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:46.530 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.559597+0000", 2024-04-19T01:04:46.530 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.1089999999999998e-06 2024-04-19T01:04:46.530 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.530 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.531 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:46.531 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.559605+0000", 2024-04-19T01:04:46.531 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.267e-06 2024-04-19T01:04:46.531 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.531 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.531 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:46.531 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.573085+0000", 2024-04-19T01:04:46.531 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.013480373 2024-04-19T01:04:46.531 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.531 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.531 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:46.531 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.573107+0000", 2024-04-19T01:04:46.531 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2031999999999998e-05 2024-04-19T01:04:46.531 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.532 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.532 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:46.532 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.576518+0000", 2024-04-19T01:04:46.532 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.003411458 2024-04-19T01:04:46.532 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.532 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.532 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:46.532 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.576564+0000", 2024-04-19T01:04:46.532 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5151999999999997e-05 2024-04-19T01:04:46.532 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:46.532 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:46.533 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:46.533 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.533 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.533 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 64/31 ECSubWrite(tid=100, reqid=client.4604.0:468, at_version=64'39, trim_to=0'0, roll_forward_to=62'35))", 2024-04-19T01:04:46.533 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:34.557491+0000", 2024-04-19T01:04:46.533 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 11.871375239000001, 2024-04-19T01:04:46.533 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.006834283, 2024-04-19T01:04:46.533 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:46.533 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:46.533 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:46.533 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:46.533 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.533 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:46.533 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:34.557491+0000", 2024-04-19T01:04:46.534 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:46.534 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.534 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.534 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:46.534 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:34.557491+0000", 2024-04-19T01:04:46.534 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:46.534 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.534 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.534 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:46.534 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:34.557493+0000", 2024-04-19T01:04:46.534 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6500000000000001e-06 2024-04-19T01:04:46.534 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.534 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.534 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:46.535 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:34.557735+0000", 2024-04-19T01:04:46.535 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00024215900000000001 2024-04-19T01:04:46.535 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.535 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.535 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:46.535 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:34.557770+0000", 2024-04-19T01:04:46.535 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4943000000000001e-05 2024-04-19T01:04:46.535 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.536 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.536 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:46.536 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:34.557792+0000", 2024-04-19T01:04:46.537 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2283e-05 2024-04-19T01:04:46.537 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.537 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.537 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:46.537 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:34.557965+0000", 2024-04-19T01:04:46.537 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017270100000000001 2024-04-19T01:04:46.537 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.537 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.537 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:46.537 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:34.557994+0000", 2024-04-19T01:04:46.537 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8515999999999999e-05 2024-04-19T01:04:46.537 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.537 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.538 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:46.538 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:34.564128+0000", 2024-04-19T01:04:46.538 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0061344399999999997 2024-04-19T01:04:46.538 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.538 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.538 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:46.538 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:34.564326+0000", 2024-04-19T01:04:46.538 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000197591 2024-04-19T01:04:46.538 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:46.538 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:46.538 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:46.538 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.538 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.538 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as2 65/31 ECSubWrite(tid=107, reqid=client.4604.0:478, at_version=65'26, trim_to=0'0, roll_forward_to=64'24))", 2024-04-19T01:04:46.539 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:35.640013+0000", 2024-04-19T01:04:46.539 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.788854079, 2024-04-19T01:04:46.539 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.015040102, 2024-04-19T01:04:46.539 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:46.539 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:46.539 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:46.539 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:46.539 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.539 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:46.539 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.640013+0000", 2024-04-19T01:04:46.539 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:46.539 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.540 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.540 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:46.540 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.640013+0000", 2024-04-19T01:04:46.540 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:46.540 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.540 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.540 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:46.540 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.640014+0000", 2024-04-19T01:04:46.540 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0330000000000001e-06 2024-04-19T01:04:46.540 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.540 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.540 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:46.540 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.640436+0000", 2024-04-19T01:04:46.540 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00042256800000000001 2024-04-19T01:04:46.541 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.541 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.541 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:46.541 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.640475+0000", 2024-04-19T01:04:46.541 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9096e-05 2024-04-19T01:04:46.541 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.541 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.541 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:46.541 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.640499+0000", 2024-04-19T01:04:46.541 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4099000000000001e-05 2024-04-19T01:04:46.541 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.541 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.541 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:46.541 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.640651+0000", 2024-04-19T01:04:46.541 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015175799999999999 2024-04-19T01:04:46.542 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.542 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.542 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:46.542 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.640670+0000", 2024-04-19T01:04:46.542 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8519999999999999e-05 2024-04-19T01:04:46.542 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.542 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.542 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:46.542 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.654985+0000", 2024-04-19T01:04:46.542 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.014315682999999999 2024-04-19T01:04:46.542 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.543 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.543 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:46.543 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.655053+0000", 2024-04-19T01:04:46.543 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7345e-05 2024-04-19T01:04:46.543 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:46.544 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:46.544 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:46.544 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.544 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.544 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as2 65/31 ECSubWrite(tid=108, reqid=client.4604.0:479, at_version=65'27, trim_to=0'0, roll_forward_to=64'24))", 2024-04-19T01:04:46.544 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:35.655491+0000", 2024-04-19T01:04:46.544 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.773375286, 2024-04-19T01:04:46.544 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.010179575999999999, 2024-04-19T01:04:46.544 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:46.544 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:46.544 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:46.544 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:46.544 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.544 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:46.545 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.655491+0000", 2024-04-19T01:04:46.545 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:46.545 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.545 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.545 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:46.545 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.655491+0000", 2024-04-19T01:04:46.545 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:46.545 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.545 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.545 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:46.545 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.655493+0000", 2024-04-19T01:04:46.545 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.59e-06 2024-04-19T01:04:46.545 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.545 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.546 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:46.546 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.655680+0000", 2024-04-19T01:04:46.546 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000186937 2024-04-19T01:04:46.546 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.546 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.546 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:46.546 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.655708+0000", 2024-04-19T01:04:46.546 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8195999999999999e-05 2024-04-19T01:04:46.547 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.547 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.547 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:46.547 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.655724+0000", 2024-04-19T01:04:46.547 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5423000000000001e-05 2024-04-19T01:04:46.547 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.547 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.547 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:46.547 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.655856+0000", 2024-04-19T01:04:46.547 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000132528 2024-04-19T01:04:46.547 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.547 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.547 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:46.548 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.655882+0000", 2024-04-19T01:04:46.548 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5565999999999999e-05 2024-04-19T01:04:46.548 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.548 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.548 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:46.548 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.665599+0000", 2024-04-19T01:04:46.548 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0097172769999999999 2024-04-19T01:04:46.548 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.548 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.548 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:46.548 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.665671+0000", 2024-04-19T01:04:46.548 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2058999999999996e-05 2024-04-19T01:04:46.548 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:46.548 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:46.549 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:46.549 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.549 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.549 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as2 65/31 ECSubWrite(tid=109, reqid=client.4604.0:480, at_version=65'28, trim_to=0'0, roll_forward_to=65'26))", 2024-04-19T01:04:46.549 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:35.666777+0000", 2024-04-19T01:04:46.549 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.762089806000001, 2024-04-19T01:04:46.549 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0080505449999999992, 2024-04-19T01:04:46.549 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:46.549 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:46.549 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:46.549 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:46.549 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.549 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:46.550 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.666777+0000", 2024-04-19T01:04:46.550 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:46.550 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.550 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.550 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:46.550 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.666777+0000", 2024-04-19T01:04:46.550 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:46.550 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.550 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.550 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:46.551 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.666779+0000", 2024-04-19T01:04:46.551 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.776e-06 2024-04-19T01:04:46.551 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.551 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.551 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:46.551 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.666875+0000", 2024-04-19T01:04:46.551 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6246e-05 2024-04-19T01:04:46.551 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.551 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.551 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:46.551 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.666903+0000", 2024-04-19T01:04:46.551 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8005999999999999e-05 2024-04-19T01:04:46.552 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.552 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.552 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:46.552 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.666925+0000", 2024-04-19T01:04:46.552 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1696e-05 2024-04-19T01:04:46.552 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.552 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.552 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:46.552 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.667055+0000", 2024-04-19T01:04:46.552 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013004800000000001 2024-04-19T01:04:46.552 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.552 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.552 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:46.552 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.667073+0000", 2024-04-19T01:04:46.553 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8445999999999999e-05 2024-04-19T01:04:46.553 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.553 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.553 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:46.553 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.674730+0000", 2024-04-19T01:04:46.553 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0076571019999999998 2024-04-19T01:04:46.553 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.553 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.553 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:46.553 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.674827+0000", 2024-04-19T01:04:46.553 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7225000000000005e-05 2024-04-19T01:04:46.553 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:46.553 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:46.553 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:46.554 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.554 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.554 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as2 71/31 ECSubWrite(tid=134, reqid=client.4604.0:522, at_version=71'49, trim_to=0'0, roll_forward_to=70'47))", 2024-04-19T01:04:46.554 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:41.526442+0000", 2024-04-19T01:04:46.554 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 4.9024250259999995, 2024-04-19T01:04:46.554 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0087350069999999995, 2024-04-19T01:04:46.554 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:46.554 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:46.554 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:46.554 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:46.554 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.554 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:46.555 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.526442+0000", 2024-04-19T01:04:46.555 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:46.555 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.555 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.555 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:46.555 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.526442+0000", 2024-04-19T01:04:46.555 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:46.555 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.555 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.555 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:46.555 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.526442+0000", 2024-04-19T01:04:46.555 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5600000000000005e-07 2024-04-19T01:04:46.556 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.556 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.556 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:46.556 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.526688+0000", 2024-04-19T01:04:46.556 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00024532699999999999 2024-04-19T01:04:46.556 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.556 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.556 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:46.556 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.526721+0000", 2024-04-19T01:04:46.556 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3853000000000003e-05 2024-04-19T01:04:46.556 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.556 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.556 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:46.557 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.526746+0000", 2024-04-19T01:04:46.557 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4439000000000001e-05 2024-04-19T01:04:46.557 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.557 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.557 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:46.557 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.526912+0000", 2024-04-19T01:04:46.557 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016591500000000001 2024-04-19T01:04:46.557 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.558 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.558 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:46.558 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.526938+0000", 2024-04-19T01:04:46.558 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5749999999999999e-05 2024-04-19T01:04:46.558 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.558 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.558 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:46.558 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.535082+0000", 2024-04-19T01:04:46.558 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0081442379999999998 2024-04-19T01:04:46.558 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.558 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.558 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:46.558 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.535177+0000", 2024-04-19T01:04:46.559 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4828999999999996e-05 2024-04-19T01:04:46.559 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:46.559 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:46.559 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:46.559 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.559 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.559 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.2s6 74/31 ECSubWrite(tid=246, reqid=osd.11.0:246, at_version=74'18, trim_to=0'0, roll_forward_to=72'16))", 2024-04-19T01:04:46.559 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:44.561627+0000", 2024-04-19T01:04:46.559 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 1.8672391940000002, 2024-04-19T01:04:46.559 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0087837249999999992, 2024-04-19T01:04:46.559 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:46.560 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:46.560 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:46.560 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:46.560 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.560 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:46.560 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.561627+0000", 2024-04-19T01:04:46.560 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:46.560 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.560 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.560 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:46.560 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.561627+0000", 2024-04-19T01:04:46.560 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:46.561 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.561 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.561 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:46.561 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.561628+0000", 2024-04-19T01:04:46.561 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2e-07 2024-04-19T01:04:46.561 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.562 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.562 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:46.562 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.561685+0000", 2024-04-19T01:04:46.562 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6675999999999997e-05 2024-04-19T01:04:46.562 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.562 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.562 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:46.562 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.561730+0000", 2024-04-19T01:04:46.562 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5355999999999997e-05 2024-04-19T01:04:46.562 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.562 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.563 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:46.563 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.561754+0000", 2024-04-19T01:04:46.563 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4403e-05 2024-04-19T01:04:46.563 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.563 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.563 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:46.563 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.561910+0000", 2024-04-19T01:04:46.563 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015511100000000001 2024-04-19T01:04:46.563 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.563 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.563 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:46.563 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.561926+0000", 2024-04-19T01:04:46.563 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6733000000000001e-05 2024-04-19T01:04:46.563 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.564 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.564 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:46.564 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.570303+0000", 2024-04-19T01:04:46.564 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0083770240000000003 2024-04-19T01:04:46.564 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:46.564 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:46.564 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:46.564 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.570411+0000", 2024-04-19T01:04:46.564 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000107902 2024-04-19T01:04:46.564 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:46.564 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:46.564 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:46.564 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:46.564 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:46.565 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:04:46.565 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-19T01:04:46.566 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-04-19T01:04:46.567 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T01:04:46.568 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-04-19T01:04:46.568 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T01:04:46.568 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-19T01:04:46.569 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-04-19T01:04:46.570 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T01:04:46.570 INFO:tasks.rados.rados.0.smithi155.stdout:234: done (0 left) 2024-04-19T01:04:46.570 INFO:tasks.rados.rados.0.smithi155.stdout:235: delete oid 11 current snap is 24 2024-04-19T01:04:46.570 INFO:tasks.rados.rados.0.smithi155.stdout:235: done (0 left) 2024-04-19T01:04:46.570 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 30 current snap is 24 2024-04-19T01:04:46.570 INFO:tasks.rados.rados.0.smithi155.stdout:236: seq_num 103 ranges {1966080=573440,2539520=786432,3325952=376832} 2024-04-19T01:04:46.570 INFO:tasks.rados.rados.0.smithi155.stdout:236: writing smithi15528762-30 from 1966080 to 2539520 tid 1 2024-04-19T01:04:46.570 INFO:tasks.rados.rados.0.smithi155.stdout:236: writing smithi15528762-30 from 2539520 to 3325952 tid 2 2024-04-19T01:04:46.571 INFO:tasks.rados.rados.0.smithi155.stdout:236: writing smithi15528762-30 from 3325952 to 3702784 tid 3 2024-04-19T01:04:46.571 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 38 current snap is 24 2024-04-19T01:04:46.571 INFO:tasks.rados.rados.0.smithi155.stdout:237: seq_num 104 ranges {622592=589824,1212416=540672,1753088=622592} 2024-04-19T01:04:46.571 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:04:46.564+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:04:46.571 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-04-19T01:04:46.571 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T01:04:46.572 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.9 to 1 (10000) 2024-04-19T01:04:46.572 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-04-19T01:04:46.572 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T01:04:46.573 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-04-19T01:04:46.574 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-04-19T01:04:46.574 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T01:04:46.574 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2024-04-19T01:04:46.575 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T01:04:46.575 INFO:tasks.rados.rados.0.smithi155.stdout:237: writing smithi15528762-38 from 622592 to 1212416 tid 1 2024-04-19T01:04:46.579 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2024-04-19T01:04:46.579 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T01:04:46.580 INFO:tasks.rados.rados.0.smithi155.stdout:237: writing smithi15528762-38 from 1212416 to 1753088 tid 2 2024-04-19T01:04:46.585 INFO:tasks.rados.rados.0.smithi155.stdout:237: writing smithi15528762-38 from 1753088 to 2375680 tid 3 2024-04-19T01:04:46.585 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 22 current snap is 24 2024-04-19T01:04:46.585 INFO:tasks.rados.rados.0.smithi155.stdout:238: seq_num 105 ranges {2457600=606208,3063808=688128,3751936=475136} 2024-04-19T01:04:46.606 INFO:tasks.rados.rados.0.smithi155.stdout:238: writing smithi15528762-22 from 2457600 to 3063808 tid 1 2024-04-19T01:04:46.610 INFO:tasks.rados.rados.0.smithi155.stdout:238: writing smithi15528762-22 from 3063808 to 3751936 tid 2 2024-04-19T01:04:46.611 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2024-04-19T01:04:46.612 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T01:04:46.614 INFO:tasks.rados.rados.0.smithi155.stdout:238: writing smithi15528762-22 from 3751936 to 4227072 tid 3 2024-04-19T01:04:46.614 INFO:tasks.rados.rados.0.smithi155.stdout:239: snap_create 2024-04-19T01:04:46.614 INFO:tasks.rados.rados.0.smithi155.stdout:236: finishing write tid 1 to smithi15528762-30 2024-04-19T01:04:46.614 INFO:tasks.rados.rados.0.smithi155.stdout:236: finishing write tid 2 to smithi15528762-30 2024-04-19T01:04:46.614 INFO:tasks.rados.rados.0.smithi155.stdout:236: finishing write tid 3 to smithi15528762-30 2024-04-19T01:04:46.614 INFO:tasks.rados.rados.0.smithi155.stdout:236: finishing write tid 4 to smithi15528762-30 2024-04-19T01:04:46.614 INFO:tasks.rados.rados.0.smithi155.stdout:236: finishing write tid 5 to smithi15528762-30 2024-04-19T01:04:46.614 INFO:tasks.rados.rados.0.smithi155.stdout:236: oid 30 updating version 0 to 58 2024-04-19T01:04:46.614 INFO:tasks.rados.rados.0.smithi155.stdout:236: oid 30 updating version 58 to 59 2024-04-19T01:04:46.614 INFO:tasks.rados.rados.0.smithi155.stdout:236: oid 30 version 59 is already newer than 56 2024-04-19T01:04:46.614 INFO:tasks.rados.rados.0.smithi155.stdout:236: oid 30 version 59 is already newer than 57 2024-04-19T01:04:46.614 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 30 v 59 (ObjNum 103 snap 24 seq_num 103) dirty exists 2024-04-19T01:04:46.614 INFO:tasks.rados.rados.0.smithi155.stdout:236: left oid 30 (ObjNum 103 snap 24 seq_num 103) 2024-04-19T01:04:46.633 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2024-04-19T01:04:46.633 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T01:04:46.648 INFO:tasks.rados.rados.0.smithi155.stdout:237: finishing write tid 1 to smithi15528762-38 2024-04-19T01:04:46.651 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2024-04-19T01:04:46.652 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-19T01:04:46.662 INFO:tasks.rados.rados.0.smithi155.stdout:237: finishing write tid 2 to smithi15528762-38 2024-04-19T01:04:46.662 INFO:tasks.rados.rados.0.smithi155.stdout:237: finishing write tid 3 to smithi15528762-38 2024-04-19T01:04:46.662 INFO:tasks.rados.rados.0.smithi155.stdout:237: finishing write tid 4 to smithi15528762-38 2024-04-19T01:04:46.663 INFO:tasks.rados.rados.0.smithi155.stdout:238: finishing write tid 1 to smithi15528762-22 2024-04-19T01:04:46.663 INFO:tasks.rados.rados.0.smithi155.stdout:238: finishing write tid 2 to smithi15528762-22 2024-04-19T01:04:46.663 INFO:tasks.rados.rados.0.smithi155.stdout:238: finishing write tid 3 to smithi15528762-22 2024-04-19T01:04:46.663 INFO:tasks.rados.rados.0.smithi155.stdout:238: finishing write tid 4 to smithi15528762-22 2024-04-19T01:04:46.666 INFO:tasks.rados.rados.0.smithi155.stdout:237: finishing write tid 5 to smithi15528762-38 2024-04-19T01:04:46.666 INFO:tasks.rados.rados.0.smithi155.stdout:237: oid 38 updating version 0 to 76 2024-04-19T01:04:46.666 INFO:tasks.rados.rados.0.smithi155.stdout:237: oid 38 updating version 76 to 77 2024-04-19T01:04:46.666 INFO:tasks.rados.rados.0.smithi155.stdout:237: oid 38 version 77 is already newer than 75 2024-04-19T01:04:46.666 INFO:tasks.rados.rados.0.smithi155.stdout:237: oid 38 version 77 is already newer than 74 2024-04-19T01:04:46.666 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 38 v 77 (ObjNum 104 snap 24 seq_num 104) dirty exists 2024-04-19T01:04:46.666 INFO:tasks.rados.rados.0.smithi155.stdout:237: left oid 38 (ObjNum 104 snap 24 seq_num 104) 2024-04-19T01:04:46.667 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:04:46.667+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:04:46.668 INFO:tasks.rados.rados.0.smithi155.stdout:238: finishing write tid 5 to smithi15528762-22 2024-04-19T01:04:46.668 INFO:tasks.rados.rados.0.smithi155.stdout:238: oid 22 updating version 0 to 80 2024-04-19T01:04:46.668 INFO:tasks.rados.rados.0.smithi155.stdout:238: oid 22 updating version 80 to 81 2024-04-19T01:04:46.668 INFO:tasks.rados.rados.0.smithi155.stdout:238: oid 22 updating version 81 to 82 2024-04-19T01:04:46.668 INFO:tasks.rados.rados.0.smithi155.stdout:238: oid 22 version 82 is already newer than 79 2024-04-19T01:04:46.668 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 22 v 82 (ObjNum 105 snap 24 seq_num 105) dirty exists 2024-04-19T01:04:46.668 INFO:tasks.rados.rados.0.smithi155.stdout:238: left oid 22 (ObjNum 105 snap 24 seq_num 105) 2024-04-19T01:04:46.673 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2024-04-19T01:04:46.673 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-19T01:04:46.694 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2024-04-19T01:04:46.694 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-19T01:04:46.703 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:46.703 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:46.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:04:46.714+0000 7f8a56d9d640 0 -- 172.21.15.33:0/3787080458 >> [v2:172.21.15.155:6807/741132602,v1:172.21.15.155:6810/741132602] conn(0x7f8a58224830 msgr2=0x7f8a58226c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-19T01:04:46.717 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-19T01:04:46.720 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2024-04-19T01:04:46.767 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:04:46.766+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:04:46.868 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:04:46.866+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:04:46.886 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:46.886 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:46.900 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-19T01:04:46.969 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:04:46.966+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:04:47.023 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:47.023 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:47.037 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-04-19T01:04:47.071 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:04:47.070+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:04:47.163 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:47.164 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:47.172 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:04:47.170+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:04:47.177 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-04-19T01:04:47.273 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:04:47.270+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:04:47.301 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:47.301 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:47.315 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-04-19T01:04:47.374 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:04:47.375+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:04:47.443 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:47.443 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:47.458 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-19T01:04:47.506 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:04:47.506+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:04:47.577 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:04:47.576+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:04:47.594 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:47.595 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:47.601 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2024-04-19T01:04:47.601 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.10 to 1 (10000) 2024-04-19T01:04:47.609 INFO:tasks.rados.rados.0.smithi155.stdout:236: done (3 left) 2024-04-19T01:04:47.609 INFO:tasks.rados.rados.0.smithi155.stdout:237: done (2 left) 2024-04-19T01:04:47.609 INFO:tasks.rados.rados.0.smithi155.stdout:238: done (1 left) 2024-04-19T01:04:47.609 INFO:tasks.rados.rados.0.smithi155.stdout:239: done (0 left) 2024-04-19T01:04:47.609 INFO:tasks.rados.rados.0.smithi155.stdout:240: read oid 38 snap -1 2024-04-19T01:04:47.609 INFO:tasks.rados.rados.0.smithi155.stdout:240: expect (ObjNum 104 snap 24 seq_num 104) 2024-04-19T01:04:47.609 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 1 current snap is 25 2024-04-19T01:04:47.610 INFO:tasks.rados.rados.0.smithi155.stdout:241: seq_num 106 ranges {32768=622592,655360=442368,1097728=524288,1622016=196608} 2024-04-19T01:04:47.611 INFO:tasks.rados.rados.0.smithi155.stdout:241: writing smithi15528762-1 from 32768 to 655360 tid 1 2024-04-19T01:04:47.614 INFO:tasks.rados.rados.0.smithi155.stdout:241: writing smithi15528762-1 from 655360 to 1097728 tid 2 2024-04-19T01:04:47.618 INFO:tasks.rados.rados.0.smithi155.stdout:241: writing smithi15528762-1 from 1097728 to 1622016 tid 3 2024-04-19T01:04:47.619 INFO:tasks.rados.rados.0.smithi155.stdout:241: writing smithi15528762-1 from 1622016 to 1818624 tid 4 2024-04-19T01:04:47.619 INFO:tasks.rados.rados.0.smithi155.stdout:242: snap_remove snap 24 2024-04-19T01:04:47.628 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-19T01:04:47.638 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-04-19T01:04:47.678 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:04:47.676+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:04:47.760 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:47.760 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:47.776 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-19T01:04:47.779 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:04:47.778+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:04:47.880 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:04:47.880+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:04:47.894 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:47.894 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:47.909 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-19T01:04:47.980 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:04:47.978+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:04:48.026 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:48.027 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:48.041 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-19T01:04:48.081 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:04:48.079+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:04:48.163 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:48.163 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:48.177 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-19T01:04:48.182 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:04:48.180+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:04:48.283 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:04:48.284+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:04:48.295 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:48.296 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:48.310 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-19T01:04:48.384 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:04:48.386+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:04:48.429 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:48.429 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:48.447 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-19T01:04:48.485 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:04:48.482+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:04:48.574 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:48.574 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:48.587 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:04:48.586+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:04:48.588 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-19T01:04:48.609 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.11 to 1 (10000) 2024-04-19T01:04:48.614 INFO:tasks.rados.rados.0.smithi155.stdout:242: done (2 left) 2024-04-19T01:04:48.614 INFO:tasks.rados.rados.0.smithi155.stdout:243: snap_remove snap 19 2024-04-19T01:04:48.614 INFO:tasks.rados.rados.0.smithi155.stdout:240: expect (ObjNum 104 snap 24 seq_num 104) 2024-04-19T01:04:48.635 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 12 1 2024-04-19T01:04:48.700 INFO:tasks.rados.rados.0.smithi155.stdout:241: finishing write tid 1 to smithi15528762-1 2024-04-19T01:04:48.700 INFO:tasks.rados.rados.0.smithi155.stdout:241: finishing write tid 2 to smithi15528762-1 2024-04-19T01:04:48.700 INFO:tasks.rados.rados.0.smithi155.stdout:241: finishing write tid 3 to smithi15528762-1 2024-04-19T01:04:48.700 INFO:tasks.rados.rados.0.smithi155.stdout:241: finishing write tid 4 to smithi15528762-1 2024-04-19T01:04:48.700 INFO:tasks.rados.rados.0.smithi155.stdout:241: finishing write tid 5 to smithi15528762-1 2024-04-19T01:04:48.700 INFO:tasks.rados.rados.0.smithi155.stdout:241: finishing write tid 6 to smithi15528762-1 2024-04-19T01:04:48.700 INFO:tasks.rados.rados.0.smithi155.stdout:241: oid 1 updating version 0 to 59 2024-04-19T01:04:48.700 INFO:tasks.rados.rados.0.smithi155.stdout:241: oid 1 version 59 is already newer than 58 2024-04-19T01:04:48.700 INFO:tasks.rados.rados.0.smithi155.stdout:241: oid 1 updating version 59 to 60 2024-04-19T01:04:48.700 INFO:tasks.rados.rados.0.smithi155.stdout:241: oid 1 updating version 60 to 62 2024-04-19T01:04:48.701 INFO:tasks.rados.rados.0.smithi155.stdout:241: oid 1 version 62 is already newer than 61 2024-04-19T01:04:48.701 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 1 v 62 (ObjNum 106 snap 25 seq_num 106) dirty exists 2024-04-19T01:04:48.701 INFO:tasks.rados.rados.0.smithi155.stdout:241: left oid 1 (ObjNum 106 snap 25 seq_num 106) 2024-04-19T01:04:48.725 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:04:48.726+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:04:48.731 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:48.732 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:48.741 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-19T01:04:48.747 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-04-19T01:04:48.789 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:48.786+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:48.890 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-04-19T01:04:48.891 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T01:04:48.891 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:04:48.890+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:04:48.926 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-04-19T01:04:48.927 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T01:04:48.927 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:48.927 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:48.941 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-04-19T01:04:48.947 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-04-19T01:04:48.947 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T01:04:48.968 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-04-19T01:04:48.968 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T01:04:48.984 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-04-19T01:04:48.985 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T01:04:48.991 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:04:48.992+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:04:49.005 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-04-19T01:04:49.005 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T01:04:49.025 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-04-19T01:04:49.025 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T01:04:49.046 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-04-19T01:04:49.046 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T01:04:49.058 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:49.058 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:49.062 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2024-04-19T01:04:49.063 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T01:04:49.072 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-04-19T01:04:49.085 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2024-04-19T01:04:49.085 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T01:04:49.105 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2024-04-19T01:04:49.106 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T01:04:49.106 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:04:49.106+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:04:49.126 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2024-04-19T01:04:49.126 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T01:04:49.143 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2024-04-19T01:04:49.143 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-19T01:04:49.164 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2024-04-19T01:04:49.164 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-19T01:04:49.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:04:49.166+0000 7fe87b7fe640 0 -- 172.21.15.33:0/24728814 >> [v2:172.21.15.118:6800/506416463,v1:172.21.15.118:6801/506416463] conn(0x7fe87c223df0 msgr2=0x7fe87c2261e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-19T01:04:49.187 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2024-04-19T01:04:49.187 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-19T01:04:49.193 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:49.190+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:49.205 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:49.206 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2024-04-19T01:04:49.206 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2024-04-19T01:04:49.206 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-04-19T01:04:49.206 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.206 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:253 3.ds0 3:b9317ffa:::smithi15528762-1:head [append 0~704512] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:49.206 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.983022+0000", 2024-04-19T01:04:49.206 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 35.220424809000001, 2024-04-19T01:04:49.206 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.034825673000000001, 2024-04-19T01:04:49.206 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:49.207 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:49.207 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:49.207 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:49.207 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:49.207 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:49.207 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 253 2024-04-19T01:04:49.207 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.207 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:49.207 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.207 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:49.207 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.983022+0000", 2024-04-19T01:04:49.207 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:49.207 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.207 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.208 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:49.208 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.983022+0000", 2024-04-19T01:04:49.208 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:49.208 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.208 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.208 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:49.208 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.983023+0000", 2024-04-19T01:04:49.208 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.6500000000000008e-07 2024-04-19T01:04:49.208 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.208 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.208 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:49.208 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.983925+0000", 2024-04-19T01:04:49.208 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000902439 2024-04-19T01:04:49.209 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.209 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.209 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:49.209 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.983943+0000", 2024-04-19T01:04:49.209 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7975e-05 2024-04-19T01:04:49.209 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.209 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.209 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:49.209 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.983952+0000", 2024-04-19T01:04:49.209 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.4270000000000008e-06 2024-04-19T01:04:49.210 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.210 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.210 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:49.210 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.984029+0000", 2024-04-19T01:04:49.210 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7631000000000003e-05 2024-04-19T01:04:49.211 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.211 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.211 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:49.211 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.984150+0000", 2024-04-19T01:04:49.211 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000120548 2024-04-19T01:04:49.211 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.211 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.211 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:49.211 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.986166+0000", 2024-04-19T01:04:49.211 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0020163410000000001 2024-04-19T01:04:49.211 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.211 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.211 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:49.211 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.001084+0000", 2024-04-19T01:04:49.212 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014917872 2024-04-19T01:04:49.212 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.212 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.212 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:49.212 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.017779+0000", 2024-04-19T01:04:49.212 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.016694817000000001 2024-04-19T01:04:49.212 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.212 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.212 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:49.212 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.017848+0000", 2024-04-19T01:04:49.212 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.8658000000000006e-05 2024-04-19T01:04:49.212 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:49.212 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:49.212 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:49.213 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.213 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.213 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:254 3.ds0 3:b9317ffa:::smithi15528762-1:head [append 0~376832] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:49.213 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.985096+0000", 2024-04-19T01:04:49.213 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 35.218351194, 2024-04-19T01:04:49.213 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.043912868000000001, 2024-04-19T01:04:49.213 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:49.213 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:49.213 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:49.213 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:49.213 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:49.213 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:49.213 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 254 2024-04-19T01:04:49.213 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.214 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:49.214 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.214 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:49.214 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985096+0000", 2024-04-19T01:04:49.214 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:49.214 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.214 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.214 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:49.214 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985096+0000", 2024-04-19T01:04:49.214 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:49.214 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.214 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.214 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:49.214 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985098+0000", 2024-04-19T01:04:49.215 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0540000000000002e-06 2024-04-19T01:04:49.215 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.215 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.215 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:49.215 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985632+0000", 2024-04-19T01:04:49.215 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00053471200000000001 2024-04-19T01:04:49.215 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.215 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.215 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:49.215 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985650+0000", 2024-04-19T01:04:49.215 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7217000000000001e-05 2024-04-19T01:04:49.215 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.215 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.216 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:49.216 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985658+0000", 2024-04-19T01:04:49.216 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.8689999999999993e-06 2024-04-19T01:04:49.216 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.216 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.216 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:49.216 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.995027+0000", 2024-04-19T01:04:49.216 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0093689159999999997 2024-04-19T01:04:49.216 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.216 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.216 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:49.216 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.995141+0000", 2024-04-19T01:04:49.216 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011387000000000001 2024-04-19T01:04:49.216 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.217 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.217 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:49.217 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.996005+0000", 2024-04-19T01:04:49.217 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00086397999999999998 2024-04-19T01:04:49.217 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.217 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.217 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:49.217 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.002138+0000", 2024-04-19T01:04:49.217 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.006133182 2024-04-19T01:04:49.217 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.217 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.217 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:49.218 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.028913+0000", 2024-04-19T01:04:49.218 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.026774686999999998 2024-04-19T01:04:49.218 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.218 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.218 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:49.218 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.029008+0000", 2024-04-19T01:04:49.218 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.5381e-05 2024-04-19T01:04:49.218 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:49.218 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:49.218 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:49.218 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.219 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.219 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:255 3.ds0 3:b9317ffa:::smithi15528762-1:head [setxattr _header (55) in=7b] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:49.220 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.985699+0000", 2024-04-19T01:04:49.220 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 35.217747656, 2024-04-19T01:04:49.220 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.043535886000000003, 2024-04-19T01:04:49.220 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:49.220 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:49.220 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:49.220 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:49.220 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:49.220 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:49.220 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 255 2024-04-19T01:04:49.220 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.220 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:49.220 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.221 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:49.221 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985699+0000", 2024-04-19T01:04:49.221 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:49.221 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.221 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.221 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:49.221 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985699+0000", 2024-04-19T01:04:49.221 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:49.221 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.221 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.221 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:49.221 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985700+0000", 2024-04-19T01:04:49.221 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4199999999999995e-07 2024-04-19T01:04:49.222 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.222 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.222 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:49.222 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985705+0000", 2024-04-19T01:04:49.222 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7960000000000003e-06 2024-04-19T01:04:49.222 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.222 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.222 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:49.222 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985715+0000", 2024-04-19T01:04:49.222 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0285999999999999e-05 2024-04-19T01:04:49.222 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.222 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.223 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:49.223 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985735+0000", 2024-04-19T01:04:49.223 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9666000000000001e-05 2024-04-19T01:04:49.223 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.223 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.223 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:49.223 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.001214+0000", 2024-04-19T01:04:49.224 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015479694 2024-04-19T01:04:49.224 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.224 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.224 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:49.224 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.001417+0000", 2024-04-19T01:04:49.224 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00020274099999999999 2024-04-19T01:04:49.224 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.224 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.224 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:49.224 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.001931+0000", 2024-04-19T01:04:49.224 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000513599 2024-04-19T01:04:49.224 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.224 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.225 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:49.225 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.002446+0000", 2024-04-19T01:04:49.225 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00051521100000000001 2024-04-19T01:04:49.225 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.225 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.225 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:49.225 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.029136+0000", 2024-04-19T01:04:49.225 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.026690523000000001 2024-04-19T01:04:49.225 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.225 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.225 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:49.225 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.029235+0000", 2024-04-19T01:04:49.225 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.8727999999999994e-05 2024-04-19T01:04:49.225 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:49.226 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:49.226 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:49.226 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.226 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.226 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:256 3.ds0 3:b9317ffa:::smithi15528762-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:49.226 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.985744+0000", 2024-04-19T01:04:49.226 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 35.217702869999997, 2024-04-19T01:04:49.226 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.046574693, 2024-04-19T01:04:49.226 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:49.227 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:49.227 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:49.227 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:49.227 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:49.227 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:49.227 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 256 2024-04-19T01:04:49.227 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.228 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:49.228 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.228 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:49.228 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985744+0000", 2024-04-19T01:04:49.228 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:49.228 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.228 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.228 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:49.228 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985744+0000", 2024-04-19T01:04:49.228 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:49.228 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.228 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.228 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:49.228 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985744+0000", 2024-04-19T01:04:49.229 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1699999999999999e-07 2024-04-19T01:04:49.229 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.229 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.229 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:49.229 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985747+0000", 2024-04-19T01:04:49.229 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3429999999999998e-06 2024-04-19T01:04:49.229 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.229 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.229 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:49.229 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985753+0000", 2024-04-19T01:04:49.229 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8619999999999999e-06 2024-04-19T01:04:49.229 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.229 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.229 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:49.230 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985756+0000", 2024-04-19T01:04:49.230 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8820000000000003e-06 2024-04-19T01:04:49.230 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.230 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.230 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:49.230 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.002179+0000", 2024-04-19T01:04:49.230 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.01642302 2024-04-19T01:04:49.230 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.230 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.230 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:04:49.230 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.002278+0000", 2024-04-19T01:04:49.231 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.8548000000000006e-05 2024-04-19T01:04:49.231 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.231 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.231 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:49.231 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.029285+0000", 2024-04-19T01:04:49.231 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.027006935999999999 2024-04-19T01:04:49.231 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.231 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.231 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:49.231 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.029400+0000", 2024-04-19T01:04:49.231 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000115438 2024-04-19T01:04:49.231 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.232 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.232 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:49.232 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.032319+0000", 2024-04-19T01:04:49.232 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0029182470000000001 2024-04-19T01:04:49.232 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:49.232 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:49.232 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:49.232 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.232 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.232 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:330 3.ds0 3:bf91dcb5:::smithi15528762-2:head [append 0~507904] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:04:49.232 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.674394+0000", 2024-04-19T01:04:49.232 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 31.529052718999999, 2024-04-19T01:04:49.232 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.036711424999999999, 2024-04-19T01:04:49.233 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:49.233 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:49.233 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:49.233 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:49.233 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:49.233 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:49.233 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 330 2024-04-19T01:04:49.234 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.234 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:49.234 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.234 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:49.234 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.674394+0000", 2024-04-19T01:04:49.235 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:49.235 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.235 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.235 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:49.235 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.674394+0000", 2024-04-19T01:04:49.235 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:49.235 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.235 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.235 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:49.235 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.674396+0000", 2024-04-19T01:04:49.235 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3870000000000002e-06 2024-04-19T01:04:49.235 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.236 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.236 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:49.236 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.675539+0000", 2024-04-19T01:04:49.236 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001142321 2024-04-19T01:04:49.236 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.236 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.236 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:49.236 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.675564+0000", 2024-04-19T01:04:49.236 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5126e-05 2024-04-19T01:04:49.236 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.236 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.236 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:49.236 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.675575+0000", 2024-04-19T01:04:49.236 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0951999999999999e-05 2024-04-19T01:04:49.237 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.237 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.237 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:49.237 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.675641+0000", 2024-04-19T01:04:49.237 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5914999999999995e-05 2024-04-19T01:04:49.237 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.237 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.237 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:49.237 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.675713+0000", 2024-04-19T01:04:49.237 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2169000000000006e-05 2024-04-19T01:04:49.237 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.237 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.237 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:49.237 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.676428+0000", 2024-04-19T01:04:49.238 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00071535000000000001 2024-04-19T01:04:49.238 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.238 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.238 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:49.238 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680290+0000", 2024-04-19T01:04:49.238 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003861532 2024-04-19T01:04:49.238 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.238 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.238 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:49.239 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.711050+0000", 2024-04-19T01:04:49.239 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.030759916000000002 2024-04-19T01:04:49.239 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.239 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.239 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:49.239 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.711106+0000", 2024-04-19T01:04:49.239 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.5757000000000002e-05 2024-04-19T01:04:49.239 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:49.239 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:49.239 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:49.239 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.239 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.239 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:331 3.ds0 3:bf91dcb5:::smithi15528762-2:head [append 0~507904] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:04:49.240 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.680101+0000", 2024-04-19T01:04:49.240 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 31.523345939999999, 2024-04-19T01:04:49.240 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.031204012, 2024-04-19T01:04:49.240 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:49.240 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:49.240 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:49.240 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:49.240 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:49.240 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:49.241 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 331 2024-04-19T01:04:49.241 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.241 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:49.241 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.241 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:49.241 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680101+0000", 2024-04-19T01:04:49.241 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:49.241 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.242 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.242 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:49.242 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680101+0000", 2024-04-19T01:04:49.242 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:49.242 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.242 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.242 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:49.242 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680103+0000", 2024-04-19T01:04:49.242 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.956e-06 2024-04-19T01:04:49.242 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.243 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.243 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:49.243 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680444+0000", 2024-04-19T01:04:49.243 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00034149199999999999 2024-04-19T01:04:49.243 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.243 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.243 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:49.243 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680460+0000", 2024-04-19T01:04:49.243 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5554e-05 2024-04-19T01:04:49.243 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.243 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.243 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:49.243 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680471+0000", 2024-04-19T01:04:49.244 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0852e-05 2024-04-19T01:04:49.244 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.244 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.244 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:49.244 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680539+0000", 2024-04-19T01:04:49.244 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.8619000000000001e-05 2024-04-19T01:04:49.244 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.244 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.244 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:49.245 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680652+0000", 2024-04-19T01:04:49.245 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011258499999999999 2024-04-19T01:04:49.245 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.245 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.245 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:49.245 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.681392+0000", 2024-04-19T01:04:49.246 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00074039899999999998 2024-04-19T01:04:49.246 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.246 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.246 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:49.246 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.691230+0000", 2024-04-19T01:04:49.246 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0098373260000000004 2024-04-19T01:04:49.246 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.246 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.246 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:49.246 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.711264+0000", 2024-04-19T01:04:49.246 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.020034501999999999 2024-04-19T01:04:49.246 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.247 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.247 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:49.247 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.711305+0000", 2024-04-19T01:04:49.247 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0726999999999998e-05 2024-04-19T01:04:49.247 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:49.247 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:49.247 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:49.247 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.247 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.247 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:332 3.ds0 3:bf91dcb5:::smithi15528762-2:head [append 0~262144] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:04:49.247 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.682638+0000", 2024-04-19T01:04:49.247 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 31.520808836, 2024-04-19T01:04:49.247 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.029333649999999999, 2024-04-19T01:04:49.247 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:49.248 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:49.248 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:49.248 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:49.248 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:49.248 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:49.248 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 332 2024-04-19T01:04:49.248 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.248 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:49.248 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.249 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:49.249 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682638+0000", 2024-04-19T01:04:49.249 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:49.249 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.249 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.249 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:49.249 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682638+0000", 2024-04-19T01:04:49.249 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:49.249 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.249 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.249 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:49.249 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682640+0000", 2024-04-19T01:04:49.250 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5990000000000001e-06 2024-04-19T01:04:49.250 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.250 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.250 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:49.250 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682770+0000", 2024-04-19T01:04:49.250 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013076200000000001 2024-04-19T01:04:49.250 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.250 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.250 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:49.250 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682784+0000", 2024-04-19T01:04:49.250 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3786e-05 2024-04-19T01:04:49.250 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.250 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.251 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:49.251 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682792+0000", 2024-04-19T01:04:49.251 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.3000000000000002e-06 2024-04-19T01:04:49.251 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.251 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.251 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:49.251 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.686722+0000", 2024-04-19T01:04:49.252 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0039299749999999996 2024-04-19T01:04:49.252 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.252 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.252 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:49.252 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.686804+0000", 2024-04-19T01:04:49.252 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1588000000000006e-05 2024-04-19T01:04:49.252 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.252 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.252 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:49.252 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.687521+0000", 2024-04-19T01:04:49.252 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00071744300000000001 2024-04-19T01:04:49.252 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.253 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.253 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:49.253 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.691494+0000", 2024-04-19T01:04:49.253 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0039721239999999996 2024-04-19T01:04:49.253 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.253 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.253 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:49.253 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.711934+0000", 2024-04-19T01:04:49.253 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.020440649000000002 2024-04-19T01:04:49.253 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.253 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.253 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:49.253 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.711972+0000", 2024-04-19T01:04:49.253 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7423999999999999e-05 2024-04-19T01:04:49.254 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:49.254 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:49.254 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:49.254 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.254 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.254 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:333 3.ds0 3:bf91dcb5:::smithi15528762-2:head [setxattr _header (55) in=7b] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:04:49.254 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.682805+0000", 2024-04-19T01:04:49.254 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 31.520641435000002, 2024-04-19T01:04:49.254 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.029376039, 2024-04-19T01:04:49.254 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:49.254 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:49.254 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:49.254 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:49.255 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:49.255 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:49.255 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 333 2024-04-19T01:04:49.255 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.255 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:49.255 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.255 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:49.255 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682805+0000", 2024-04-19T01:04:49.256 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:49.256 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.256 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.256 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:49.256 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682805+0000", 2024-04-19T01:04:49.256 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:49.256 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.256 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.256 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:49.256 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682806+0000", 2024-04-19T01:04:49.256 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4099999999999999e-07 2024-04-19T01:04:49.256 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.256 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.256 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:49.257 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682815+0000", 2024-04-19T01:04:49.257 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.516e-06 2024-04-19T01:04:49.257 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.257 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.257 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:49.257 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682823+0000", 2024-04-19T01:04:49.257 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6459999999999993e-06 2024-04-19T01:04:49.257 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.257 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.257 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:49.257 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682832+0000", 2024-04-19T01:04:49.257 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.6260000000000001e-06 2024-04-19T01:04:49.257 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.257 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.258 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:49.258 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.690350+0000", 2024-04-19T01:04:49.258 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0075183730000000001 2024-04-19T01:04:49.258 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.258 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.258 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:49.258 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.690456+0000", 2024-04-19T01:04:49.258 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010617599999999999 2024-04-19T01:04:49.258 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.258 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.258 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:49.258 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.690951+0000", 2024-04-19T01:04:49.258 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00049519699999999995 2024-04-19T01:04:49.259 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.259 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.259 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:49.262 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.693060+0000", 2024-04-19T01:04:49.263 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0021084239999999998 2024-04-19T01:04:49.263 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.263 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.263 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:49.263 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.712088+0000", 2024-04-19T01:04:49.263 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.019028024000000001 2024-04-19T01:04:49.263 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.263 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.263 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:49.263 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.712181+0000", 2024-04-19T01:04:49.263 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.3516000000000004e-05 2024-04-19T01:04:49.263 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:49.263 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:49.264 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:49.264 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.264 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.264 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:334 3.ds0 3:bf91dcb5:::smithi15528762-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:04:49.264 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.682840+0000", 2024-04-19T01:04:49.264 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 31.520607286000001, 2024-04-19T01:04:49.264 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.031162179000000002, 2024-04-19T01:04:49.264 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:49.264 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:49.264 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:49.264 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:49.264 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:49.264 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:49.264 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 334 2024-04-19T01:04:49.265 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.265 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:49.265 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.265 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:49.265 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682840+0000", 2024-04-19T01:04:49.265 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:49.265 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.265 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.265 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:49.265 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682840+0000", 2024-04-19T01:04:49.265 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:49.266 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.266 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.266 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:49.266 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682840+0000", 2024-04-19T01:04:49.266 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4899999999999995e-07 2024-04-19T01:04:49.266 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.266 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.267 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:49.267 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682848+0000", 2024-04-19T01:04:49.267 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5270000000000001e-06 2024-04-19T01:04:49.267 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.267 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.267 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:49.267 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682855+0000", 2024-04-19T01:04:49.267 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.4889999999999998e-06 2024-04-19T01:04:49.267 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.267 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.267 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:49.267 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682860+0000", 2024-04-19T01:04:49.267 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.164e-06 2024-04-19T01:04:49.267 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.267 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.268 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:49.268 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.691525+0000", 2024-04-19T01:04:49.268 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0086643239999999993 2024-04-19T01:04:49.268 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.268 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.268 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:04:49.268 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.691618+0000", 2024-04-19T01:04:49.268 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2983000000000001e-05 2024-04-19T01:04:49.268 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.268 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.268 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:49.268 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.712661+0000", 2024-04-19T01:04:49.268 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021043777 2024-04-19T01:04:49.268 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.269 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.269 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:49.269 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.712733+0000", 2024-04-19T01:04:49.269 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2063000000000004e-05 2024-04-19T01:04:49.269 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.269 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.269 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:49.269 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.714002+0000", 2024-04-19T01:04:49.269 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0012683029999999999 2024-04-19T01:04:49.269 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:49.269 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:49.269 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:49.269 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.269 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.269 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:354 3.as0 3:56c32ee7:::smithi15528762-47:head [copy-from ver 35] snapc 9=[8,7] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:04:49.270 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.544030+0000", 2024-04-19T01:04:49.270 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 30.659417259000001, 2024-04-19T01:04:49.270 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.049063250000000003, 2024-04-19T01:04:49.270 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:49.270 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:49.270 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:49.270 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:49.270 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:49.270 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:49.270 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 354 2024-04-19T01:04:49.270 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.270 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:49.270 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.270 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:49.271 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544030+0000", 2024-04-19T01:04:49.271 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:49.271 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.271 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.271 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:49.271 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544030+0000", 2024-04-19T01:04:49.271 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:49.271 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.271 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.271 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:49.271 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544033+0000", 2024-04-19T01:04:49.271 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.506e-06 2024-04-19T01:04:49.271 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.271 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.271 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:49.272 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544046+0000", 2024-04-19T01:04:49.272 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3208e-05 2024-04-19T01:04:49.272 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.272 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.272 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:49.273 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544082+0000", 2024-04-19T01:04:49.273 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5509999999999997e-05 2024-04-19T01:04:49.273 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.273 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.273 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:49.273 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544101+0000", 2024-04-19T01:04:49.273 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9500999999999998e-05 2024-04-19T01:04:49.273 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.273 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.274 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:49.274 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544230+0000", 2024-04-19T01:04:49.274 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000129114 2024-04-19T01:04:49.274 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.274 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.274 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:49.274 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544428+0000", 2024-04-19T01:04:49.274 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00019776799999999999 2024-04-19T01:04:49.274 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.274 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.274 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:49.274 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.556395+0000", 2024-04-19T01:04:49.274 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011967261 2024-04-19T01:04:49.274 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.275 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.275 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:49.275 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.574011+0000", 2024-04-19T01:04:49.275 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017615654000000001 2024-04-19T01:04:49.275 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.275 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.275 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:49.275 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.592920+0000", 2024-04-19T01:04:49.275 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018908812000000001 2024-04-19T01:04:49.275 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.275 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.275 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:49.275 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.593093+0000", 2024-04-19T01:04:49.275 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00017291600000000001 2024-04-19T01:04:49.276 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:49.276 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:49.276 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:49.276 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.276 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.276 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:355 3.as0 3:56c32ee7:::smithi15528762-47:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:04:49.276 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.544150+0000", 2024-04-19T01:04:49.276 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 30.659296486999999, 2024-04-19T01:04:49.276 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.029722035000000001, 2024-04-19T01:04:49.277 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:49.277 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:49.277 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:49.277 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:49.277 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:49.277 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:49.277 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 355 2024-04-19T01:04:49.277 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.277 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:49.277 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.277 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:49.277 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544150+0000", 2024-04-19T01:04:49.277 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:49.278 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.278 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.278 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:49.278 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544150+0000", 2024-04-19T01:04:49.278 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:49.278 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.278 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.278 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:49.278 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544152+0000", 2024-04-19T01:04:49.278 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.28e-06 2024-04-19T01:04:49.278 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.278 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.278 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:49.278 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544158+0000", 2024-04-19T01:04:49.279 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4160000000000001e-06 2024-04-19T01:04:49.279 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.279 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.279 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:49.279 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544178+0000", 2024-04-19T01:04:49.279 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0054000000000001e-05 2024-04-19T01:04:49.279 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.279 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.279 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:49.279 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544195+0000", 2024-04-19T01:04:49.279 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6835999999999999e-05 2024-04-19T01:04:49.279 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.279 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.279 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:49.279 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544717+0000", 2024-04-19T01:04:49.280 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00052258800000000005 2024-04-19T01:04:49.280 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.280 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.280 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for blocked object", 2024-04-19T01:04:49.280 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544835+0000", 2024-04-19T01:04:49.280 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000117448 2024-04-19T01:04:49.280 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.280 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.280 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:49.280 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.573609+0000", 2024-04-19T01:04:49.280 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.028774094 2024-04-19T01:04:49.280 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.280 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.280 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:49.281 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.573721+0000", 2024-04-19T01:04:49.281 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011205900000000001 2024-04-19T01:04:49.281 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.281 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.281 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:49.281 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.573872+0000", 2024-04-19T01:04:49.281 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015126000000000001 2024-04-19T01:04:49.281 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:49.281 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:49.281 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:49.281 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.281 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.281 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:478 3.as0 3:5a488dbb:::smithi15528762-13:head [append 0~770048] snapc 12=[12,11,10] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-19T01:04:49.281 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:35.631704+0000", 2024-04-19T01:04:49.281 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 13.571743259, 2024-04-19T01:04:49.282 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.031533347000000003, 2024-04-19T01:04:49.282 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:49.282 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:49.282 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:49.282 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:49.282 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:49.282 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:49.282 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 478 2024-04-19T01:04:49.282 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.282 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:49.282 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.282 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:49.282 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.631704+0000", 2024-04-19T01:04:49.283 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:49.283 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.283 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.283 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:49.283 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.631704+0000", 2024-04-19T01:04:49.283 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:49.283 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.283 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.283 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:49.283 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.631707+0000", 2024-04-19T01:04:49.283 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0800000000000002e-06 2024-04-19T01:04:49.283 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.283 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.283 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:49.284 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.632909+0000", 2024-04-19T01:04:49.284 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0012019070000000001 2024-04-19T01:04:49.284 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.284 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.284 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:49.284 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.632934+0000", 2024-04-19T01:04:49.284 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5289000000000001e-05 2024-04-19T01:04:49.284 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.284 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.284 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:49.284 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.632945+0000", 2024-04-19T01:04:49.284 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1046999999999999e-05 2024-04-19T01:04:49.284 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.284 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.284 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:49.285 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.633050+0000", 2024-04-19T01:04:49.285 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010516 2024-04-19T01:04:49.285 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.285 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.285 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op must wait for map", 2024-04-19T01:04:49.285 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.633065+0000", 2024-04-19T01:04:49.285 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4765e-05 2024-04-19T01:04:49.285 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.285 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.285 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:49.285 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.636005+0000", 2024-04-19T01:04:49.285 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0029403530000000002 2024-04-19T01:04:49.285 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.285 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.286 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:49.286 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.636188+0000", 2024-04-19T01:04:49.286 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00018290099999999999 2024-04-19T01:04:49.286 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.286 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.286 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:49.286 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.638553+0000", 2024-04-19T01:04:49.286 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0023651800000000001 2024-04-19T01:04:49.286 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.286 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.286 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:49.286 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.662256+0000", 2024-04-19T01:04:49.287 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.023702524999999999 2024-04-19T01:04:49.287 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.287 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.287 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:49.287 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.662993+0000", 2024-04-19T01:04:49.287 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00073735000000000001 2024-04-19T01:04:49.287 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.287 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.287 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:49.287 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.663237+0000", 2024-04-19T01:04:49.288 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00024379 2024-04-19T01:04:49.288 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:49.288 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:49.288 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:49.288 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.288 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.288 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:479 3.as0 3:5a488dbb:::smithi15528762-13:head [append 0~475136] snapc 12=[12,11,10] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-19T01:04:49.288 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:35.633850+0000", 2024-04-19T01:04:49.288 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 13.569597001, 2024-04-19T01:04:49.288 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.042497337000000003, 2024-04-19T01:04:49.288 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:49.288 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:49.288 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:49.288 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:49.289 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:49.289 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:49.289 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 479 2024-04-19T01:04:49.289 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.289 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:49.289 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.289 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:49.289 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.633850+0000", 2024-04-19T01:04:49.289 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:49.289 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.289 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.289 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:49.289 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.633850+0000", 2024-04-19T01:04:49.290 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:49.290 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.290 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.290 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:49.290 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.633852+0000", 2024-04-19T01:04:49.290 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.26e-06 2024-04-19T01:04:49.290 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.290 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.290 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:49.290 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.634421+0000", 2024-04-19T01:04:49.291 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000569058 2024-04-19T01:04:49.291 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.291 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.291 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:49.291 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.634441+0000", 2024-04-19T01:04:49.291 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0169e-05 2024-04-19T01:04:49.291 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.291 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.291 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:49.291 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.634452+0000", 2024-04-19T01:04:49.291 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0963e-05 2024-04-19T01:04:49.291 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.291 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.291 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:49.292 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.634540+0000", 2024-04-19T01:04:49.292 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.7563000000000002e-05 2024-04-19T01:04:49.292 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.292 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.292 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:04:49.292 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.634550+0000", 2024-04-19T01:04:49.292 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0491e-05 2024-04-19T01:04:49.292 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.292 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.292 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:49.292 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.653619+0000", 2024-04-19T01:04:49.292 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.019068284000000001 2024-04-19T01:04:49.292 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.292 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.293 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:49.293 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.653716+0000", 2024-04-19T01:04:49.293 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7546000000000004e-05 2024-04-19T01:04:49.293 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.293 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.293 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:49.293 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.654525+0000", 2024-04-19T01:04:49.293 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00080861300000000002 2024-04-19T01:04:49.293 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.293 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.293 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:49.293 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.674163+0000", 2024-04-19T01:04:49.293 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.019638666999999999 2024-04-19T01:04:49.294 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.294 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.294 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:49.294 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.676228+0000", 2024-04-19T01:04:49.294 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0020650379999999999 2024-04-19T01:04:49.294 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.294 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.294 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:49.294 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.676347+0000", 2024-04-19T01:04:49.294 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011868500000000001 2024-04-19T01:04:49.294 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:49.294 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:49.294 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:49.295 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.295 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.295 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:480 3.as0 3:5a488dbb:::smithi15528762-13:head [append 0~294912] snapc 12=[12,11,10] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-19T01:04:49.295 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:35.639539+0000", 2024-04-19T01:04:49.295 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 13.563907367000001, 2024-04-19T01:04:49.295 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.038046241000000001, 2024-04-19T01:04:49.295 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:49.295 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:49.295 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:49.295 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:49.295 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:49.295 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:49.295 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 480 2024-04-19T01:04:49.296 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.296 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:49.296 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.296 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:49.296 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639539+0000", 2024-04-19T01:04:49.296 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:49.296 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.296 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.296 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:49.296 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639539+0000", 2024-04-19T01:04:49.296 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:49.296 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.297 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.297 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:49.297 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639541+0000", 2024-04-19T01:04:49.297 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.922e-06 2024-04-19T01:04:49.297 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.297 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.297 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:49.297 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639847+0000", 2024-04-19T01:04:49.298 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00030593599999999998 2024-04-19T01:04:49.298 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.298 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.298 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:49.298 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639886+0000", 2024-04-19T01:04:49.298 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9171000000000002e-05 2024-04-19T01:04:49.298 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.298 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.298 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:49.298 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639898+0000", 2024-04-19T01:04:49.298 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1103e-05 2024-04-19T01:04:49.298 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.299 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.299 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:49.299 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.663566+0000", 2024-04-19T01:04:49.299 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.023668392 2024-04-19T01:04:49.299 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.299 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.299 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:49.299 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.663853+0000", 2024-04-19T01:04:49.299 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00028731200000000002 2024-04-19T01:04:49.299 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.299 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.299 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:49.299 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.665713+0000", 2024-04-19T01:04:49.300 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001860109 2024-04-19T01:04:49.300 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.300 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.300 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:49.300 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.675595+0000", 2024-04-19T01:04:49.300 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0098813939999999999 2024-04-19T01:04:49.300 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.300 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.300 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:49.300 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.677466+0000", 2024-04-19T01:04:49.300 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0018710199999999999 2024-04-19T01:04:49.300 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.300 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.300 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:49.301 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.677586+0000", 2024-04-19T01:04:49.301 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000119882 2024-04-19T01:04:49.301 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:49.301 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:49.301 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:49.301 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.301 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.301 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:481 3.as0 3:5a488dbb:::smithi15528762-13:head [setxattr _header (57) in=7b] snapc 12=[12,11,10] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-19T01:04:49.301 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:35.639915+0000", 2024-04-19T01:04:49.301 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 13.563532146, 2024-04-19T01:04:49.301 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.038026319000000003, 2024-04-19T01:04:49.301 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:49.301 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:49.302 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:49.302 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:49.302 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:49.302 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:49.302 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 481 2024-04-19T01:04:49.302 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.302 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:49.302 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.302 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:49.302 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639915+0000", 2024-04-19T01:04:49.302 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:49.302 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.302 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.303 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:49.303 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639915+0000", 2024-04-19T01:04:49.303 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:49.303 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.303 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.303 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:49.303 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639915+0000", 2024-04-19T01:04:49.303 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.0800000000000004e-07 2024-04-19T01:04:49.303 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.303 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.303 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:49.303 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639919+0000", 2024-04-19T01:04:49.303 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0589999999999996e-06 2024-04-19T01:04:49.303 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.304 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.304 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:49.304 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639929+0000", 2024-04-19T01:04:49.304 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.9590000000000007e-06 2024-04-19T01:04:49.304 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.304 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.304 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:49.304 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639936+0000", 2024-04-19T01:04:49.304 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2480000000000001e-06 2024-04-19T01:04:49.304 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.304 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.304 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:49.304 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.674290+0000", 2024-04-19T01:04:49.304 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.034354279000000001 2024-04-19T01:04:49.305 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.305 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.305 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:49.305 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.674404+0000", 2024-04-19T01:04:49.305 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000114189 2024-04-19T01:04:49.305 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.305 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.305 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:49.305 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.675012+0000", 2024-04-19T01:04:49.305 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00060776199999999997 2024-04-19T01:04:49.305 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.305 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.305 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:49.305 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.676365+0000", 2024-04-19T01:04:49.306 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0013529849999999999 2024-04-19T01:04:49.306 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.306 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.306 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:49.306 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.677805+0000", 2024-04-19T01:04:49.306 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014399160000000001 2024-04-19T01:04:49.306 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.306 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.306 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:49.306 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.677941+0000", 2024-04-19T01:04:49.306 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013621400000000001 2024-04-19T01:04:49.306 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:49.306 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:49.307 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:49.307 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.307 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.307 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:482 3.as0 3:5a488dbb:::smithi15528762-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-04-19T01:04:49.307 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:35.639943+0000", 2024-04-19T01:04:49.307 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 13.563503947999999, 2024-04-19T01:04:49.307 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.040997591999999999, 2024-04-19T01:04:49.307 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:49.308 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:49.308 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:49.308 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:49.308 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:49.308 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:49.308 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 482 2024-04-19T01:04:49.308 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.308 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:49.308 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.308 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:49.308 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639943+0000", 2024-04-19T01:04:49.308 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:49.308 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.309 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.309 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:49.309 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639943+0000", 2024-04-19T01:04:49.309 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:49.309 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.309 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.309 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:49.309 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639943+0000", 2024-04-19T01:04:49.309 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.89e-07 2024-04-19T01:04:49.309 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.310 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.310 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:49.310 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639945+0000", 2024-04-19T01:04:49.310 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9870000000000002e-06 2024-04-19T01:04:49.310 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.310 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.310 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:49.310 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639951+0000", 2024-04-19T01:04:49.310 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.9479999999999996e-06 2024-04-19T01:04:49.310 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.310 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.310 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:49.310 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639956+0000", 2024-04-19T01:04:49.310 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8040000000000001e-06 2024-04-19T01:04:49.311 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.311 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.311 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:49.311 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.676400+0000", 2024-04-19T01:04:49.311 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.036444184999999997 2024-04-19T01:04:49.311 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.311 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.311 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:04:49.311 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.676473+0000", 2024-04-19T01:04:49.311 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2704000000000001e-05 2024-04-19T01:04:49.311 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.311 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.311 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:49.311 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.678615+0000", 2024-04-19T01:04:49.311 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0021422030000000001 2024-04-19T01:04:49.312 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.312 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.312 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:49.312 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.678811+0000", 2024-04-19T01:04:49.312 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000196026 2024-04-19T01:04:49.312 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.312 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.312 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:49.312 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.680940+0000", 2024-04-19T01:04:49.312 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0021293459999999998 2024-04-19T01:04:49.312 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:49.312 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:49.312 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:49.312 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.313 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.313 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:520 3.as0 3:58da707e:::smithi15528762-12:head [copy-from ver 38] snapc 16=[16,15,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-04-19T01:04:49.313 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:41.512306+0000", 2024-04-19T01:04:49.313 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 7.6911410250000003, 2024-04-19T01:04:49.313 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.089648318000000005, 2024-04-19T01:04:49.313 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:49.313 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:49.313 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:49.313 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:49.313 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:49.313 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:49.313 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 520 2024-04-19T01:04:49.313 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.313 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:49.314 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.314 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:49.314 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512306+0000", 2024-04-19T01:04:49.314 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:49.314 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.314 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.314 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:49.314 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512306+0000", 2024-04-19T01:04:49.314 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:49.314 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.314 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.314 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:49.314 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512309+0000", 2024-04-19T01:04:49.314 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2509999999999999e-06 2024-04-19T01:04:49.315 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.315 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.315 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:49.315 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512326+0000", 2024-04-19T01:04:49.315 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6881999999999999e-05 2024-04-19T01:04:49.315 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.315 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.315 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:49.315 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512356+0000", 2024-04-19T01:04:49.315 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9920000000000002e-05 2024-04-19T01:04:49.315 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.315 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.315 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:49.315 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512371+0000", 2024-04-19T01:04:49.315 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5325999999999999e-05 2024-04-19T01:04:49.316 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.316 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.316 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:49.316 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512489+0000", 2024-04-19T01:04:49.316 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011835100000000001 2024-04-19T01:04:49.316 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.316 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.316 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op must wait for map", 2024-04-19T01:04:49.316 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512500+0000", 2024-04-19T01:04:49.316 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0437e-05 2024-04-19T01:04:49.316 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.316 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.316 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:49.316 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.514800+0000", 2024-04-19T01:04:49.316 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0023000719999999998 2024-04-19T01:04:49.317 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.317 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.317 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:49.317 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.514923+0000", 2024-04-19T01:04:49.317 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012253599999999999 2024-04-19T01:04:49.317 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.317 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.317 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:49.317 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.541802+0000", 2024-04-19T01:04:49.317 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.026879473000000001 2024-04-19T01:04:49.317 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.317 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.317 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:49.317 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.564261+0000", 2024-04-19T01:04:49.317 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.022458887 2024-04-19T01:04:49.318 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.318 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.318 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:49.318 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.601886+0000", 2024-04-19T01:04:49.318 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.037625204000000002 2024-04-19T01:04:49.318 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.318 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.318 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:49.318 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.601954+0000", 2024-04-19T01:04:49.318 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.7978999999999994e-05 2024-04-19T01:04:49.318 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:49.319 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:49.319 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:49.319 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.319 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.319 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:521 3.as0 3:58da707e:::smithi15528762-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e71)", 2024-04-19T01:04:49.319 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:41.512407+0000", 2024-04-19T01:04:49.319 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 7.6910396030000001, 2024-04-19T01:04:49.319 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.051806598000000002, 2024-04-19T01:04:49.319 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:49.319 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:49.319 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:49.320 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:49.320 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:49.320 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:49.320 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 521 2024-04-19T01:04:49.320 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.320 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:49.320 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.320 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:49.320 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512407+0000", 2024-04-19T01:04:49.320 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:49.320 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.320 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.320 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:49.320 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512407+0000", 2024-04-19T01:04:49.321 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:49.321 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.321 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.321 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:49.321 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512408+0000", 2024-04-19T01:04:49.321 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.4799999999999997e-07 2024-04-19T01:04:49.321 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.321 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.321 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:49.321 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512411+0000", 2024-04-19T01:04:49.321 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5109999999999999e-06 2024-04-19T01:04:49.321 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.321 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.321 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:49.322 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512423+0000", 2024-04-19T01:04:49.322 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1188e-05 2024-04-19T01:04:49.322 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.322 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.322 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:49.322 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512430+0000", 2024-04-19T01:04:49.322 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.3540000000000002e-06 2024-04-19T01:04:49.322 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.322 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.322 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:49.322 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512598+0000", 2024-04-19T01:04:49.322 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00016804 2024-04-19T01:04:49.322 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.322 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.323 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:04:49.323 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512605+0000", 2024-04-19T01:04:49.323 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.9290000000000001e-06 2024-04-19T01:04:49.323 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.323 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.323 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:49.323 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.515130+0000", 2024-04-19T01:04:49.323 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0025246299999999999 2024-04-19T01:04:49.323 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.323 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.323 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for blocked object", 2024-04-19T01:04:49.323 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.515181+0000", 2024-04-19T01:04:49.323 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.1100000000000002e-05 2024-04-19T01:04:49.323 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.323 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.324 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:49.324 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.563980+0000", 2024-04-19T01:04:49.324 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.048799373 2024-04-19T01:04:49.324 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.324 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.324 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:49.324 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.564087+0000", 2024-04-19T01:04:49.324 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000106827 2024-04-19T01:04:49.324 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.324 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.324 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:49.324 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.564214+0000", 2024-04-19T01:04:49.324 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012689799999999999 2024-04-19T01:04:49.324 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:49.325 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:49.325 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:49.325 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.325 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.325 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:522 3.as0 3:5a488dbb:::smithi15528762-13:head [copy-from ver 2] snapc 16=[16,15,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-04-19T01:04:49.325 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:41.512460+0000", 2024-04-19T01:04:49.325 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 7.6909870539999998, 2024-04-19T01:04:49.325 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.022697987999999999, 2024-04-19T01:04:49.325 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:49.325 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:49.325 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:49.325 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:49.325 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:49.325 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:49.326 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 522 2024-04-19T01:04:49.326 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.326 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:49.326 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.326 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:49.326 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512460+0000", 2024-04-19T01:04:49.326 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:49.326 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.326 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.326 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:49.326 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512460+0000", 2024-04-19T01:04:49.326 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:49.326 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.326 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.327 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:49.327 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512460+0000", 2024-04-19T01:04:49.327 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4899999999999995e-07 2024-04-19T01:04:49.327 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.327 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.327 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:49.327 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512465+0000", 2024-04-19T01:04:49.327 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5560000000000001e-06 2024-04-19T01:04:49.327 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.327 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.327 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:49.327 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512474+0000", 2024-04-19T01:04:49.327 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9050000000000003e-06 2024-04-19T01:04:49.327 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.327 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.328 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:49.328 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512482+0000", 2024-04-19T01:04:49.328 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7819999999999994e-06 2024-04-19T01:04:49.328 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.328 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.328 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:49.328 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512683+0000", 2024-04-19T01:04:49.328 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00020168000000000001 2024-04-19T01:04:49.328 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.328 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.328 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:04:49.328 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512691+0000", 2024-04-19T01:04:49.328 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.9710000000000004e-06 2024-04-19T01:04:49.328 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.328 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.329 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:49.329 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.515212+0000", 2024-04-19T01:04:49.329 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.002520567 2024-04-19T01:04:49.329 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.329 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.329 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:49.329 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.515359+0000", 2024-04-19T01:04:49.329 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014727300000000001 2024-04-19T01:04:49.330 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.330 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.330 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:49.330 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.524092+0000", 2024-04-19T01:04:49.330 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0087328500000000003 2024-04-19T01:04:49.330 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.330 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.330 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:49.330 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.528284+0000", 2024-04-19T01:04:49.330 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0041917730000000002 2024-04-19T01:04:49.330 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.330 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.330 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:49.330 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.535055+0000", 2024-04-19T01:04:49.330 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0067715939999999997 2024-04-19T01:04:49.331 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.331 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.331 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:49.331 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.535158+0000", 2024-04-19T01:04:49.331 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000102388 2024-04-19T01:04:49.331 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:49.331 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:49.331 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:49.331 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.331 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.331 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:523 3.as0 3:5a488dbb:::smithi15528762-13:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e71)", 2024-04-19T01:04:49.331 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:41.512492+0000", 2024-04-19T01:04:49.331 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 7.6909550979999999, 2024-04-19T01:04:49.331 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015724650999999999, 2024-04-19T01:04:49.332 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:49.332 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:49.332 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:49.332 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:49.332 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:49.332 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:49.332 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 523 2024-04-19T01:04:49.332 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.332 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:49.332 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.332 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:49.332 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512492+0000", 2024-04-19T01:04:49.332 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:49.332 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.333 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.333 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:49.333 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512492+0000", 2024-04-19T01:04:49.333 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:49.333 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.333 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.333 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:49.333 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512492+0000", 2024-04-19T01:04:49.333 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2500000000000001e-07 2024-04-19T01:04:49.333 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.333 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.333 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:49.333 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512495+0000", 2024-04-19T01:04:49.333 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7669999999999999e-06 2024-04-19T01:04:49.333 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.334 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.334 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:49.334 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512503+0000", 2024-04-19T01:04:49.334 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.3599999999999996e-06 2024-04-19T01:04:49.334 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.334 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.334 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:49.334 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512509+0000", 2024-04-19T01:04:49.334 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.772e-06 2024-04-19T01:04:49.334 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.334 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.334 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:49.334 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512981+0000", 2024-04-19T01:04:49.334 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00047226000000000001 2024-04-19T01:04:49.335 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.335 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.335 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:04:49.335 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512988+0000", 2024-04-19T01:04:49.335 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.9430000000000003e-06 2024-04-19T01:04:49.335 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.335 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.335 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:49.335 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.515588+0000", 2024-04-19T01:04:49.335 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0025993380000000001 2024-04-19T01:04:49.335 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.335 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.335 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for blocked object", 2024-04-19T01:04:49.335 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.515624+0000", 2024-04-19T01:04:49.336 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.6023999999999999e-05 2024-04-19T01:04:49.336 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.336 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.336 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:49.336 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.527975+0000", 2024-04-19T01:04:49.336 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012351735000000001 2024-04-19T01:04:49.336 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.336 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.336 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:49.336 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.528083+0000", 2024-04-19T01:04:49.336 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000107822 2024-04-19T01:04:49.336 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:49.336 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:49.336 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:49.337 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.528216+0000", 2024-04-19T01:04:49.337 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013320500000000001 2024-04-19T01:04:49.337 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:49.337 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:49.337 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:49.337 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:49.337 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:49.337 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:49.338 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2024-04-19T01:04:49.338 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-19T01:04:49.340 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-19T01:04:49.341 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:04:49.338+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:04:49.439 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:04:49.438+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:04:49.472 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:49.472 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [], 2024-04-19T01:04:49.472 INFO:teuthology.orchestra.run.smithi110.stdout: "num_ops": 0 2024-04-19T01:04:49.472 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:49.488 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-19T01:04:49.540 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:04:49.538+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:04:49.612 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:49.612 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [], 2024-04-19T01:04:49.612 INFO:teuthology.orchestra.run.smithi110.stdout: "complaint_time": 30, 2024-04-19T01:04:49.612 INFO:teuthology.orchestra.run.smithi110.stdout: "num_blocked_ops": 0 2024-04-19T01:04:49.612 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:49.617 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.12 to 1 (10000) 2024-04-19T01:04:49.625 INFO:tasks.rados.rados.0.smithi155.stdout:240: done (2 left) 2024-04-19T01:04:49.625 INFO:tasks.rados.rados.0.smithi155.stdout:241: done (1 left) 2024-04-19T01:04:49.626 INFO:tasks.rados.rados.0.smithi155.stdout:243: done (0 left) 2024-04-19T01:04:49.626 INFO:tasks.rados.rados.0.smithi155.stdout:244: delete oid 50 current snap is 25 2024-04-19T01:04:49.628 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-19T01:04:49.640 INFO:tasks.rados.rados.0.smithi155.stdout:244: done (0 left) 2024-04-19T01:04:49.640 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 17 current snap is 25 2024-04-19T01:04:49.640 INFO:tasks.rados.rados.0.smithi155.stdout:245: seq_num 108 ranges {278528=655360,933888=606208,1540096=475136,2015232=81920} 2024-04-19T01:04:49.640 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-19T01:04:49.642 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:04:49.639+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:04:49.649 INFO:tasks.rados.rados.0.smithi155.stdout:245: writing smithi15528762-17 from 278528 to 933888 tid 1 2024-04-19T01:04:49.654 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 13 1 2024-04-19T01:04:49.655 INFO:tasks.rados.rados.0.smithi155.stdout:245: writing smithi15528762-17 from 933888 to 1540096 tid 2 2024-04-19T01:04:49.660 INFO:tasks.rados.rados.0.smithi155.stdout:245: writing smithi15528762-17 from 1540096 to 2015232 tid 3 2024-04-19T01:04:49.662 INFO:tasks.rados.rados.0.smithi155.stdout:245: writing smithi15528762-17 from 2015232 to 2097152 tid 4 2024-04-19T01:04:49.662 INFO:tasks.rados.rados.0.smithi155.stdout:246: rollback oid 6 current snap is 25 2024-04-19T01:04:49.662 INFO:tasks.rados.rados.0.smithi155.stdout:rollback oid 6 to 21 2024-04-19T01:04:49.662 INFO:tasks.rados.rados.0.smithi155.stdout:247: copy_from oid 10 from oid 20 current snap is 25 2024-04-19T01:04:49.662 INFO:tasks.rados.rados.0.smithi155.stdout:248: read oid 3 snap -1 2024-04-19T01:04:49.662 INFO:tasks.rados.rados.0.smithi155.stdout:248: expect (ObjNum 2 snap 0 seq_num 2) 2024-04-19T01:04:49.663 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 16 current snap is 25 2024-04-19T01:04:49.663 INFO:tasks.rados.rados.0.smithi155.stdout:249: seq_num 109 ranges {262144=671744,933888=753664,1687552=409600} 2024-04-19T01:04:49.672 INFO:tasks.rados.rados.0.smithi155.stdout:249: writing smithi15528762-16 from 262144 to 933888 tid 1 2024-04-19T01:04:49.679 INFO:tasks.rados.rados.0.smithi155.stdout:249: writing smithi15528762-16 from 933888 to 1687552 tid 2 2024-04-19T01:04:49.684 INFO:tasks.rados.rados.0.smithi155.stdout:249: writing smithi15528762-16 from 1687552 to 2097152 tid 3 2024-04-19T01:04:49.684 INFO:tasks.rados.rados.0.smithi155.stdout:250: snap_create 2024-04-19T01:04:49.685 INFO:tasks.rados.rados.0.smithi155.stdout:246: finishing rollback tid 0 to smithi15528762-6 2024-04-19T01:04:49.685 INFO:tasks.rados.rados.0.smithi155.stdout:246: finishing rollback tid 1 to smithi15528762-6 2024-04-19T01:04:49.685 INFO:tasks.rados.rados.0.smithi155.stdout:246: finishing rollback tid 2 to smithi15528762-6 2024-04-19T01:04:49.685 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 6 v 47 (ObjNum 78 snap 8 seq_num 78) dirty exists 2024-04-19T01:04:49.685 INFO:tasks.rados.rados.0.smithi155.stdout:247: finishing copy_from racing read to smithi15528762-10 2024-04-19T01:04:49.685 INFO:tasks.rados.rados.0.smithi155.stdout:247: finishing copy_from to smithi15528762-10 2024-04-19T01:04:49.685 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 10 v 2 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-04-19T01:04:49.694 INFO:tasks.rados.rados.0.smithi155.stdout:248: expect (ObjNum 2 snap 0 seq_num 2) 2024-04-19T01:04:49.702 INFO:tasks.rados.rados.0.smithi155.stdout:245: finishing write tid 1 to smithi15528762-17 2024-04-19T01:04:49.704 INFO:tasks.rados.rados.0.smithi155.stdout:245: finishing write tid 2 to smithi15528762-17 2024-04-19T01:04:49.704 INFO:tasks.rados.rados.0.smithi155.stdout:245: finishing write tid 3 to smithi15528762-17 2024-04-19T01:04:49.705 INFO:tasks.rados.rados.0.smithi155.stdout:245: finishing write tid 4 to smithi15528762-17 2024-04-19T01:04:49.705 INFO:tasks.rados.rados.0.smithi155.stdout:245: finishing write tid 5 to smithi15528762-17 2024-04-19T01:04:49.707 INFO:tasks.rados.rados.0.smithi155.stdout:245: finishing write tid 6 to smithi15528762-17 2024-04-19T01:04:49.707 INFO:tasks.rados.rados.0.smithi155.stdout:245: oid 17 updating version 0 to 94 2024-04-19T01:04:49.707 INFO:tasks.rados.rados.0.smithi155.stdout:245: oid 17 version 94 is already newer than 91 2024-04-19T01:04:49.707 INFO:tasks.rados.rados.0.smithi155.stdout:245: oid 17 version 94 is already newer than 92 2024-04-19T01:04:49.707 INFO:tasks.rados.rados.0.smithi155.stdout:245: oid 17 version 94 is already newer than 93 2024-04-19T01:04:49.708 INFO:tasks.rados.rados.0.smithi155.stdout:245: oid 17 version 94 is already newer than 90 2024-04-19T01:04:49.708 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 17 v 94 (ObjNum 108 snap 25 seq_num 108) dirty exists 2024-04-19T01:04:49.708 INFO:tasks.rados.rados.0.smithi155.stdout:245: left oid 17 (ObjNum 108 snap 25 seq_num 108) 2024-04-19T01:04:49.712 INFO:tasks.rados.rados.0.smithi155.stdout:249: finishing write tid 1 to smithi15528762-16 2024-04-19T01:04:49.717 INFO:tasks.rados.rados.0.smithi155.stdout:249: finishing write tid 2 to smithi15528762-16 2024-04-19T01:04:49.717 INFO:tasks.rados.rados.0.smithi155.stdout:249: finishing write tid 3 to smithi15528762-16 2024-04-19T01:04:49.717 INFO:tasks.rados.rados.0.smithi155.stdout:249: finishing write tid 4 to smithi15528762-16 2024-04-19T01:04:49.722 INFO:tasks.rados.rados.0.smithi155.stdout:249: finishing write tid 5 to smithi15528762-16 2024-04-19T01:04:49.722 INFO:tasks.rados.rados.0.smithi155.stdout:249: oid 16 updating version 0 to 57 2024-04-19T01:04:49.722 INFO:tasks.rados.rados.0.smithi155.stdout:249: oid 16 version 57 is already newer than 55 2024-04-19T01:04:49.722 INFO:tasks.rados.rados.0.smithi155.stdout:249: oid 16 version 57 is already newer than 56 2024-04-19T01:04:49.722 INFO:tasks.rados.rados.0.smithi155.stdout:249: oid 16 version 57 is already newer than 54 2024-04-19T01:04:49.722 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 16 v 57 (ObjNum 109 snap 25 seq_num 109) dirty exists 2024-04-19T01:04:49.722 INFO:tasks.rados.rados.0.smithi155.stdout:249: left oid 16 (ObjNum 109 snap 25 seq_num 109) 2024-04-19T01:04:49.743 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:04:49.742+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:04:49.762 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:49.762 INFO:teuthology.orchestra.run.smithi110.stdout: "size": 20, 2024-04-19T01:04:49.762 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 600, 2024-04-19T01:04:49.762 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [ 2024-04-19T01:04:49.763 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.763 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:170 3.0s0 3:0abdaea3:::smithi15528762-5:head [rollback 3] snapc 5=[5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-19T01:04:49.763 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:08.802081+0000", 2024-04-19T01:04:49.763 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 40.960324323999998, 2024-04-19T01:04:49.763 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.017167723999999999, 2024-04-19T01:04:49.763 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:49.764 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:49.764 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:49.764 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:49.764 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:49.764 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:49.764 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 170 2024-04-19T01:04:49.765 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.765 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:49.765 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.765 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:49.765 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.802081+0000", 2024-04-19T01:04:49.765 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:49.765 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.766 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.766 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:49.766 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.802081+0000", 2024-04-19T01:04:49.766 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:49.766 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.767 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.767 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:49.767 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.802083+0000", 2024-04-19T01:04:49.767 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.1230000000000001e-06 2024-04-19T01:04:49.767 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.767 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.767 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:49.768 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.802088+0000", 2024-04-19T01:04:49.768 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.0529999999999997e-06 2024-04-19T01:04:49.768 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.768 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.768 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:49.768 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.802102+0000", 2024-04-19T01:04:49.769 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.4569000000000001e-05 2024-04-19T01:04:49.769 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.769 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.769 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:49.769 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.802113+0000", 2024-04-19T01:04:49.769 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.0556e-05 2024-04-19T01:04:49.770 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.770 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.770 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:49.770 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.802425+0000", 2024-04-19T01:04:49.770 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00031243699999999999 2024-04-19T01:04:49.770 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.771 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.771 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:04:49.771 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.802438+0000", 2024-04-19T01:04:49.771 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.2493e-05 2024-04-19T01:04:49.771 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.771 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.772 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:49.772 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.810002+0000", 2024-04-19T01:04:49.772 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0075644969999999999 2024-04-19T01:04:49.772 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.772 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.772 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:49.773 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.810220+0000", 2024-04-19T01:04:49.773 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000217436 2024-04-19T01:04:49.773 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.773 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.773 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:49.773 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.811138+0000", 2024-04-19T01:04:49.774 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00091886099999999998 2024-04-19T01:04:49.774 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.774 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.774 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:49.774 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.815974+0000", 2024-04-19T01:04:49.774 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0048358519999999999 2024-04-19T01:04:49.775 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.775 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.776 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:49.776 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.819154+0000", 2024-04-19T01:04:49.776 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0031794890000000002 2024-04-19T01:04:49.776 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.776 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.776 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:49.777 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.819249+0000", 2024-04-19T01:04:49.777 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.5358000000000004e-05 2024-04-19T01:04:49.777 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:49.777 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:49.777 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:49.777 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.778 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.778 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:171 3.0s0 3:0abdaea3:::smithi15528762-5:head [append 0~0] snapc 5=[5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-19T01:04:49.778 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:08.802222+0000", 2024-04-19T01:04:49.778 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 40.960183682, 2024-04-19T01:04:49.778 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.018484289000000001, 2024-04-19T01:04:49.778 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:49.779 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:49.779 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:49.779 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:49.779 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:49.779 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:49.779 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 171 2024-04-19T01:04:49.780 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.780 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:49.780 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.780 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:49.780 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.802222+0000", 2024-04-19T01:04:49.780 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:49.780 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.781 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.781 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:49.781 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.802222+0000", 2024-04-19T01:04:49.781 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:49.781 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.781 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.781 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:49.782 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.802223+0000", 2024-04-19T01:04:49.782 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.3170000000000001e-06 2024-04-19T01:04:49.782 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.782 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.782 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:49.782 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.802232+0000", 2024-04-19T01:04:49.783 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.3960000000000006e-06 2024-04-19T01:04:49.783 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.783 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.783 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:49.783 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.802251+0000", 2024-04-19T01:04:49.783 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9609000000000001e-05 2024-04-19T01:04:49.783 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.784 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.784 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:49.784 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.802266+0000", 2024-04-19T01:04:49.784 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5051999999999999e-05 2024-04-19T01:04:49.784 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.784 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.784 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:49.785 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.802513+0000", 2024-04-19T01:04:49.785 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000246371 2024-04-19T01:04:49.785 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.785 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.785 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:04:49.785 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.802524+0000", 2024-04-19T01:04:49.785 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.1535999999999999e-05 2024-04-19T01:04:49.785 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.785 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.786 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:49.786 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.812512+0000", 2024-04-19T01:04:49.786 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0099875599999999995 2024-04-19T01:04:49.786 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.786 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.786 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:49.786 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.812675+0000", 2024-04-19T01:04:49.787 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000163525 2024-04-19T01:04:49.787 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.787 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.787 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:49.787 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.813813+0000", 2024-04-19T01:04:49.787 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001137676 2024-04-19T01:04:49.788 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.788 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.788 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:49.788 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.816015+0000", 2024-04-19T01:04:49.788 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0022017410000000001 2024-04-19T01:04:49.788 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.788 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.789 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:49.789 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.820527+0000", 2024-04-19T01:04:49.789 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0045121720000000001 2024-04-19T01:04:49.789 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.789 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.789 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:49.789 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:08.820706+0000", 2024-04-19T01:04:49.790 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00017933399999999999 2024-04-19T01:04:49.790 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:49.790 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:49.790 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:49.790 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.790 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.791 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs5 43/31 ECSubWrite(tid=47, reqid=client.4604.0:263, at_version=43'8, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:04:49.791 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.993731+0000", 2024-04-19T01:04:49.791 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 35.768675043999998, 2024-04-19T01:04:49.791 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.025167086000000002, 2024-04-19T01:04:49.791 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:49.791 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:49.792 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:49.792 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:49.792 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.792 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:49.792 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.993731+0000", 2024-04-19T01:04:49.792 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:49.792 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.793 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.793 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:49.793 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.993731+0000", 2024-04-19T01:04:49.793 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:49.793 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.793 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.793 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:49.794 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.993733+0000", 2024-04-19T01:04:49.794 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.114e-06 2024-04-19T01:04:49.794 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.794 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.794 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:49.794 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.994282+0000", 2024-04-19T01:04:49.794 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00054962800000000003 2024-04-19T01:04:49.795 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.795 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.795 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:49.795 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.994320+0000", 2024-04-19T01:04:49.795 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.7824999999999998e-05 2024-04-19T01:04:49.795 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.795 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.796 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:49.796 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.994345+0000", 2024-04-19T01:04:49.796 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4295e-05 2024-04-19T01:04:49.796 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.796 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.796 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:49.797 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.994505+0000", 2024-04-19T01:04:49.797 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00016080700000000001 2024-04-19T01:04:49.797 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.797 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.797 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:49.797 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.994525+0000", 2024-04-19T01:04:49.797 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0049000000000001e-05 2024-04-19T01:04:49.798 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.798 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.798 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:49.798 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.018713+0000", 2024-04-19T01:04:49.798 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.024187810000000001 2024-04-19T01:04:49.798 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.799 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.799 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:49.799 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.018898+0000", 2024-04-19T01:04:49.799 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000184558 2024-04-19T01:04:49.799 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:49.800 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:49.800 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:49.800 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.800 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.800 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs5 43/31 ECSubWrite(tid=48, reqid=client.4604.0:264, at_version=43'9, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:04:49.800 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.006038+0000", 2024-04-19T01:04:49.800 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 35.756367779000001, 2024-04-19T01:04:49.801 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.018149728, 2024-04-19T01:04:49.801 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:49.801 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:49.801 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:49.801 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:49.801 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.802 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:49.802 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006038+0000", 2024-04-19T01:04:49.802 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:49.802 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.802 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.802 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:49.802 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006038+0000", 2024-04-19T01:04:49.803 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:49.803 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.803 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.803 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:49.803 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006040+0000", 2024-04-19T01:04:49.803 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0829999999999998e-06 2024-04-19T01:04:49.804 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.804 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.804 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:49.804 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006230+0000", 2024-04-19T01:04:49.804 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00018986799999999999 2024-04-19T01:04:49.805 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.805 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.805 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:49.805 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006274+0000", 2024-04-19T01:04:49.805 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.3671999999999998e-05 2024-04-19T01:04:49.805 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.806 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.806 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:49.806 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006299+0000", 2024-04-19T01:04:49.806 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5062000000000002e-05 2024-04-19T01:04:49.806 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.806 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.807 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:49.807 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.008623+0000", 2024-04-19T01:04:49.807 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0023245459999999998 2024-04-19T01:04:49.807 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.807 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.807 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:49.807 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.008653+0000", 2024-04-19T01:04:49.808 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.0212000000000001e-05 2024-04-19T01:04:49.808 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.808 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.808 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:49.808 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.024106+0000", 2024-04-19T01:04:49.808 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.015452815 2024-04-19T01:04:49.809 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.809 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.809 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:49.809 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.024188+0000", 2024-04-19T01:04:49.809 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.1470000000000004e-05 2024-04-19T01:04:49.809 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:49.809 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:49.810 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:49.810 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.810 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.810 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs5 43/31 ECSubWrite(tid=49, reqid=client.4604.0:265, at_version=43'10, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:04:49.810 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.006353+0000", 2024-04-19T01:04:49.810 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 35.756052386, 2024-04-19T01:04:49.811 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.024130236999999999, 2024-04-19T01:04:49.811 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:49.811 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:49.811 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:49.811 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:49.811 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.811 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:49.812 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006353+0000", 2024-04-19T01:04:49.812 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:49.812 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.812 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.812 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:49.812 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006353+0000", 2024-04-19T01:04:49.813 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:49.813 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.813 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.813 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:49.813 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006354+0000", 2024-04-19T01:04:49.813 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.0899999999999999e-06 2024-04-19T01:04:49.814 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.814 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.814 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:49.814 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006393+0000", 2024-04-19T01:04:49.814 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.8897999999999999e-05 2024-04-19T01:04:49.814 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.815 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.815 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:49.815 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006417+0000", 2024-04-19T01:04:49.815 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3881999999999999e-05 2024-04-19T01:04:49.815 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.815 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.816 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:49.816 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006436+0000", 2024-04-19T01:04:49.816 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.8488999999999999e-05 2024-04-19T01:04:49.816 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.816 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.816 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:49.817 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.029248+0000", 2024-04-19T01:04:49.817 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.022812136 2024-04-19T01:04:49.817 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.817 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.817 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:49.817 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.029278+0000", 2024-04-19T01:04:49.817 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.9702000000000001e-05 2024-04-19T01:04:49.818 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.818 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.818 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:49.818 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.030387+0000", 2024-04-19T01:04:49.818 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001109814 2024-04-19T01:04:49.818 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.819 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.819 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:49.819 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.030484+0000", 2024-04-19T01:04:49.819 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.6225999999999997e-05 2024-04-19T01:04:49.819 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:49.819 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:49.819 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:49.820 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.820 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.820 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:308 3.0s0 3:0abdaea3:::smithi15528762-5:head [append 0~720896] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:04:49.820 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:17.563573+0000", 2024-04-19T01:04:49.820 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 32.198832240999998, 2024-04-19T01:04:49.820 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.022395149, 2024-04-19T01:04:49.820 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:49.820 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:49.820 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:49.820 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:49.821 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:49.821 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:49.821 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 308 2024-04-19T01:04:49.821 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.821 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:49.821 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.821 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:49.821 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.563573+0000", 2024-04-19T01:04:49.821 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:49.822 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.822 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.822 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:49.822 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.563573+0000", 2024-04-19T01:04:49.822 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:49.822 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.822 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.822 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:49.823 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.563575+0000", 2024-04-19T01:04:49.823 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6339999999999999e-06 2024-04-19T01:04:49.823 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.823 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.823 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:49.823 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564347+0000", 2024-04-19T01:04:49.823 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00077217599999999996 2024-04-19T01:04:49.823 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.823 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.824 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:49.824 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564361+0000", 2024-04-19T01:04:49.824 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.3665e-05 2024-04-19T01:04:49.824 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.824 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.824 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:49.825 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564372+0000", 2024-04-19T01:04:49.825 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.0834e-05 2024-04-19T01:04:49.825 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.825 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.825 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:49.825 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564954+0000", 2024-04-19T01:04:49.825 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00058264700000000005 2024-04-19T01:04:49.825 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.826 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.826 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:49.826 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.565068+0000", 2024-04-19T01:04:49.826 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000113174 2024-04-19T01:04:49.826 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.826 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.826 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:49.826 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.566092+0000", 2024-04-19T01:04:49.826 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0010239649999999999 2024-04-19T01:04:49.827 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.828 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.828 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:49.828 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.574730+0000", 2024-04-19T01:04:49.828 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0086380120000000005 2024-04-19T01:04:49.828 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.828 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.828 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:49.829 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.585859+0000", 2024-04-19T01:04:49.829 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.011129515 2024-04-19T01:04:49.829 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.829 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.829 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:49.829 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.585969+0000", 2024-04-19T01:04:49.829 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000109527 2024-04-19T01:04:49.829 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:49.829 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:49.829 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:49.829 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.829 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.829 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:309 3.0s0 3:0abdaea3:::smithi15528762-5:head [append 0~49152] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:04:49.829 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:17.564572+0000", 2024-04-19T01:04:49.830 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 32.197833416000002, 2024-04-19T01:04:49.830 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.022344373000000001, 2024-04-19T01:04:49.830 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:49.830 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:49.830 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:49.830 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:49.830 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:49.830 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:49.830 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 309 2024-04-19T01:04:49.830 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.830 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:49.830 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.830 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:49.831 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564572+0000", 2024-04-19T01:04:49.831 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:49.831 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.831 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.831 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:49.831 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564572+0000", 2024-04-19T01:04:49.831 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:49.831 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.831 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.831 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:49.831 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564573+0000", 2024-04-19T01:04:49.831 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.5600000000000004e-07 2024-04-19T01:04:49.831 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.831 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.831 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:49.832 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564647+0000", 2024-04-19T01:04:49.832 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.4212e-05 2024-04-19T01:04:49.832 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.832 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.832 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:49.832 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564659+0000", 2024-04-19T01:04:49.832 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.1516e-05 2024-04-19T01:04:49.832 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.832 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.832 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:49.832 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564669+0000", 2024-04-19T01:04:49.832 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.9250000000000007e-06 2024-04-19T01:04:49.832 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.832 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.832 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:49.833 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.573061+0000", 2024-04-19T01:04:49.833 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0083917249999999992 2024-04-19T01:04:49.833 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.833 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.833 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:49.833 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.573228+0000", 2024-04-19T01:04:49.833 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00016785899999999999 2024-04-19T01:04:49.833 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.833 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.833 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:49.833 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.573957+0000", 2024-04-19T01:04:49.833 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00072819900000000001 2024-04-19T01:04:49.833 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.833 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.834 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:49.834 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.575986+0000", 2024-04-19T01:04:49.834 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0020295640000000002 2024-04-19T01:04:49.834 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.834 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.834 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:49.834 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.586818+0000", 2024-04-19T01:04:49.834 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.010831685000000001 2024-04-19T01:04:49.834 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.834 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.834 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:49.834 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.586917+0000", 2024-04-19T01:04:49.834 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.8732000000000003e-05 2024-04-19T01:04:49.834 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:49.835 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:49.835 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:49.835 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.835 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.835 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:310 3.0s0 3:0abdaea3:::smithi15528762-5:head [setxattr _header (55) in=7b] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:04:49.835 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:17.564685+0000", 2024-04-19T01:04:49.835 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 32.197720617999998, 2024-04-19T01:04:49.835 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.022864789999999999, 2024-04-19T01:04:49.835 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:49.835 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:49.836 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:49.836 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:49.836 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:49.837 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:49.837 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 310 2024-04-19T01:04:49.837 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.837 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:49.837 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.837 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:49.837 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564685+0000", 2024-04-19T01:04:49.837 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:49.837 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.837 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.837 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:49.837 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564685+0000", 2024-04-19T01:04:49.837 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:49.838 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.838 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.838 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:49.838 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564686+0000", 2024-04-19T01:04:49.838 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.0999999999999997e-07 2024-04-19T01:04:49.838 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.838 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.838 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:49.838 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564690+0000", 2024-04-19T01:04:49.838 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.3100000000000002e-06 2024-04-19T01:04:49.838 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.838 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.838 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:49.838 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564699+0000", 2024-04-19T01:04:49.838 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.6640000000000004e-06 2024-04-19T01:04:49.839 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.839 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.839 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:49.839 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564705+0000", 2024-04-19T01:04:49.839 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.4910000000000002e-06 2024-04-19T01:04:49.839 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.839 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.839 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:49.839 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.575044+0000", 2024-04-19T01:04:49.839 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.010338293 2024-04-19T01:04:49.839 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.839 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.839 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:49.839 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.575112+0000", 2024-04-19T01:04:49.840 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.7898999999999995e-05 2024-04-19T01:04:49.840 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.840 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.840 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:49.840 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.575501+0000", 2024-04-19T01:04:49.840 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000389491 2024-04-19T01:04:49.840 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.840 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.840 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:49.840 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.576333+0000", 2024-04-19T01:04:49.840 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00083234599999999998 2024-04-19T01:04:49.840 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.840 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.841 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:49.841 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.587286+0000", 2024-04-19T01:04:49.841 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.010952437000000001 2024-04-19T01:04:49.841 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.841 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.841 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:49.841 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.587550+0000", 2024-04-19T01:04:49.841 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00026404899999999998 2024-04-19T01:04:49.841 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:49.841 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:49.841 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:49.841 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.841 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.841 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:311 3.0s0 3:0abdaea3:::smithi15528762-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:04:49.842 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:17.564725+0000", 2024-04-19T01:04:49.842 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 32.197680718000001, 2024-04-19T01:04:49.842 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.026756121000000001, 2024-04-19T01:04:49.842 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:49.842 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:49.842 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:04:49.842 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:49.842 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:49.842 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:49.842 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 311 2024-04-19T01:04:49.842 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.842 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:49.842 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.843 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:49.843 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564725+0000", 2024-04-19T01:04:49.843 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:49.843 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.843 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.843 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:49.843 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564725+0000", 2024-04-19T01:04:49.843 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:49.843 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.843 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.843 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:49.843 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564726+0000", 2024-04-19T01:04:49.843 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.5700000000000002e-07 2024-04-19T01:04:49.844 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.844 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.844 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:49.844 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564730+0000", 2024-04-19T01:04:49.844 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.476e-06 2024-04-19T01:04:49.844 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.844 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.844 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:49.844 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564740+0000", 2024-04-19T01:04:49.844 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.2750000000000001e-06 2024-04-19T01:04:49.844 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.844 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.844 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:49.844 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564749+0000", 2024-04-19T01:04:49.845 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.1640000000000007e-06 2024-04-19T01:04:49.845 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.845 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.845 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:49.845 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.576057+0000", 2024-04-19T01:04:49.845 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.011308573000000001 2024-04-19T01:04:49.845 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.845 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.845 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting for rw locks", 2024-04-19T01:04:49.845 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.576308+0000", 2024-04-19T01:04:49.845 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00025028300000000002 2024-04-19T01:04:49.845 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.845 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.845 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:49.846 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.588445+0000", 2024-04-19T01:04:49.846 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.012137129 2024-04-19T01:04:49.846 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.846 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.846 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:49.846 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.588680+0000", 2024-04-19T01:04:49.846 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00023515400000000001 2024-04-19T01:04:49.846 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.846 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.846 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:49.846 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.591481+0000", 2024-04-19T01:04:49.846 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00280131 2024-04-19T01:04:49.846 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:49.847 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:49.847 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:49.847 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.847 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.847 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs5 48/31 ECSubWrite(tid=76, reqid=client.4604.0:345, at_version=48'22, trim_to=0'0, roll_forward_to=47'19))", 2024-04-19T01:04:49.847 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.538139+0000", 2024-04-19T01:04:49.847 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 31.22426669, 2024-04-19T01:04:49.847 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.030818128, 2024-04-19T01:04:49.847 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:49.847 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:49.847 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:49.847 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:49.847 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.847 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:49.847 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538139+0000", 2024-04-19T01:04:49.848 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:49.848 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.848 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.848 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:49.848 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538139+0000", 2024-04-19T01:04:49.848 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:49.848 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.848 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.848 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:49.848 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538141+0000", 2024-04-19T01:04:49.848 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.796e-06 2024-04-19T01:04:49.848 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.848 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.848 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:49.848 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538876+0000", 2024-04-19T01:04:49.849 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00073526699999999995 2024-04-19T01:04:49.849 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.849 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.849 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:49.849 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538919+0000", 2024-04-19T01:04:49.849 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.3065000000000002e-05 2024-04-19T01:04:49.849 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.849 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.849 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:49.849 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538943+0000", 2024-04-19T01:04:49.850 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3578999999999999e-05 2024-04-19T01:04:49.850 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.850 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.850 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:49.850 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.539114+0000", 2024-04-19T01:04:49.850 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000171383 2024-04-19T01:04:49.850 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.850 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.850 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:49.850 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.539140+0000", 2024-04-19T01:04:49.850 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5709000000000001e-05 2024-04-19T01:04:49.850 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.850 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.850 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:49.851 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.568880+0000", 2024-04-19T01:04:49.851 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.029740128000000001 2024-04-19T01:04:49.851 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.851 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.851 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:49.851 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.568957+0000", 2024-04-19T01:04:49.851 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.7200999999999995e-05 2024-04-19T01:04:49.851 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:49.851 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:49.851 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:49.851 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.851 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.851 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs5 48/31 ECSubWrite(tid=78, reqid=client.4604.0:346, at_version=48'23, trim_to=0'0, roll_forward_to=48'21))", 2024-04-19T01:04:49.851 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.548746+0000", 2024-04-19T01:04:49.851 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 31.213660014999999, 2024-04-19T01:04:49.852 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.023711396999999999, 2024-04-19T01:04:49.852 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:49.852 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:49.852 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:49.852 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:49.852 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.852 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:49.852 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.548746+0000", 2024-04-19T01:04:49.852 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:49.852 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.852 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.852 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:49.852 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.548746+0000", 2024-04-19T01:04:49.852 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:49.852 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.853 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.853 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:49.853 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.548747+0000", 2024-04-19T01:04:49.853 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5799999999999999e-06 2024-04-19T01:04:49.853 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.853 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.853 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:49.853 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.549144+0000", 2024-04-19T01:04:49.853 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00039714900000000002 2024-04-19T01:04:49.853 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.853 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.853 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:49.853 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.549174+0000", 2024-04-19T01:04:49.853 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.9652999999999999e-05 2024-04-19T01:04:49.853 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.854 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.854 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:49.854 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.549195+0000", 2024-04-19T01:04:49.854 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0585e-05 2024-04-19T01:04:49.854 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.854 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.854 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:49.854 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.554787+0000", 2024-04-19T01:04:49.854 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0055927160000000002 2024-04-19T01:04:49.854 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.854 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.854 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:49.854 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.554813+0000", 2024-04-19T01:04:49.854 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5669000000000001e-05 2024-04-19T01:04:49.855 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.855 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.855 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:49.855 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.572370+0000", 2024-04-19T01:04:49.855 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.017556852000000001 2024-04-19T01:04:49.855 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.855 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.855 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:49.855 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.572457+0000", 2024-04-19T01:04:49.855 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.7193000000000004e-05 2024-04-19T01:04:49.855 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:49.855 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:49.855 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:49.855 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.856 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.856 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs5 48/31 ECSubWrite(tid=79, reqid=client.4604.0:347, at_version=48'24, trim_to=0'0, roll_forward_to=48'21))", 2024-04-19T01:04:49.856 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.556212+0000", 2024-04-19T01:04:49.856 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 31.206193248000002, 2024-04-19T01:04:49.856 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.018061275000000002, 2024-04-19T01:04:49.856 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:49.856 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:49.856 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:49.856 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:49.856 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.856 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:49.856 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.556212+0000", 2024-04-19T01:04:49.856 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:49.856 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.856 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.857 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:49.857 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.556212+0000", 2024-04-19T01:04:49.857 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:49.857 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.857 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.857 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:49.857 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.556214+0000", 2024-04-19T01:04:49.857 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.883e-06 2024-04-19T01:04:49.857 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.857 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.857 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:49.857 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.556284+0000", 2024-04-19T01:04:49.857 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.9491000000000003e-05 2024-04-19T01:04:49.857 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.857 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.858 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:49.858 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.556314+0000", 2024-04-19T01:04:49.858 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.0233e-05 2024-04-19T01:04:49.858 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.858 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.858 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:49.858 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.556335+0000", 2024-04-19T01:04:49.858 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1094000000000002e-05 2024-04-19T01:04:49.858 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.858 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.858 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:49.858 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.569301+0000", 2024-04-19T01:04:49.858 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.012965617 2024-04-19T01:04:49.858 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.859 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.859 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:49.859 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.569340+0000", 2024-04-19T01:04:49.859 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.9232e-05 2024-04-19T01:04:49.859 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.859 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.859 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:49.859 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.574195+0000", 2024-04-19T01:04:49.859 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0048553779999999996 2024-04-19T01:04:49.859 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.859 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.859 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:49.859 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.574274+0000", 2024-04-19T01:04:49.859 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.8347000000000001e-05 2024-04-19T01:04:49.859 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:49.860 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:49.860 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:49.860 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.860 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.860 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.as5 48/31 ECSubWrite(tid=71, reqid=client.4604.0:354, at_version=48'10, trim_to=0'0, roll_forward_to=34'8))", 2024-04-19T01:04:49.860 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.557068+0000", 2024-04-19T01:04:49.860 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 31.205337992, 2024-04-19T01:04:49.860 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.029586061, 2024-04-19T01:04:49.860 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:49.860 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:49.860 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:49.860 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:49.860 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.861 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:49.861 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557068+0000", 2024-04-19T01:04:49.861 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:49.861 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.861 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.861 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:49.861 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557068+0000", 2024-04-19T01:04:49.861 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:49.861 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.861 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.861 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:49.861 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557069+0000", 2024-04-19T01:04:49.861 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.9500000000000001e-07 2024-04-19T01:04:49.861 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.861 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.862 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:49.862 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.558172+0000", 2024-04-19T01:04:49.862 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001103637 2024-04-19T01:04:49.862 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.862 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.862 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:49.862 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.558206+0000", 2024-04-19T01:04:49.862 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.4069000000000002e-05 2024-04-19T01:04:49.862 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.862 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.862 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:49.862 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.558229+0000", 2024-04-19T01:04:49.862 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2495999999999999e-05 2024-04-19T01:04:49.862 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.863 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.863 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:49.863 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.558450+0000", 2024-04-19T01:04:49.863 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000221747 2024-04-19T01:04:49.863 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.863 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.863 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:49.863 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.558476+0000", 2024-04-19T01:04:49.863 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5755999999999999e-05 2024-04-19T01:04:49.863 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.863 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.863 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:49.863 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.586611+0000", 2024-04-19T01:04:49.863 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.028134731999999999 2024-04-19T01:04:49.863 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.864 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.864 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:49.864 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.586654+0000", 2024-04-19T01:04:49.864 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.2828999999999999e-05 2024-04-19T01:04:49.864 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:49.864 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:49.864 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:49.864 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.864 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.864 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.8s2 49/31 ECSubWrite(tid=202, reqid=osd.11.0:201, at_version=0'0, trim_to=0'0, roll_forward_to=49'24))", 2024-04-19T01:04:49.864 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:19.535477+0000", 2024-04-19T01:04:49.864 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 30.226928787999999, 2024-04-19T01:04:49.864 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.020657227, 2024-04-19T01:04:49.864 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:49.864 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:49.865 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:49.865 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:49.865 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.865 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:49.865 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.535477+0000", 2024-04-19T01:04:49.865 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:49.865 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.865 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.865 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:49.865 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.535477+0000", 2024-04-19T01:04:49.865 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:49.865 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.865 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.866 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:49.866 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.535479+0000", 2024-04-19T01:04:49.866 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6700000000000001e-06 2024-04-19T01:04:49.866 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.866 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.866 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:49.866 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.535504+0000", 2024-04-19T01:04:49.866 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5626000000000002e-05 2024-04-19T01:04:49.866 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.866 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.866 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:49.866 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.535538+0000", 2024-04-19T01:04:49.866 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.3457999999999997e-05 2024-04-19T01:04:49.866 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.866 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.867 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:49.867 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.535560+0000", 2024-04-19T01:04:49.867 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2503000000000001e-05 2024-04-19T01:04:49.867 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.867 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.867 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:49.867 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.536980+0000", 2024-04-19T01:04:49.867 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0014200700000000001 2024-04-19T01:04:49.867 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.867 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.867 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:49.867 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.537005+0000", 2024-04-19T01:04:49.867 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4579e-05 2024-04-19T01:04:49.867 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.868 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.868 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:49.868 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.556071+0000", 2024-04-19T01:04:49.868 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.019066422999999999 2024-04-19T01:04:49.868 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.868 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.868 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:49.868 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.556134+0000", 2024-04-19T01:04:49.868 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.2898000000000007e-05 2024-04-19T01:04:49.868 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:49.868 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:49.868 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:49.868 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.869 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.869 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.8s2 49/31 ECSubWrite(tid=203, reqid=client.4604.0:360, at_version=49'25, trim_to=0'0, roll_forward_to=49'24))", 2024-04-19T01:04:49.869 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:19.538361+0000", 2024-04-19T01:04:49.869 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 30.224044555999999, 2024-04-19T01:04:49.869 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.028842518000000001, 2024-04-19T01:04:49.869 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:49.869 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:49.869 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:49.869 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:49.869 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.869 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:49.869 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.538361+0000", 2024-04-19T01:04:49.869 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:49.869 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.869 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.870 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:49.870 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.538361+0000", 2024-04-19T01:04:49.870 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:49.870 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.870 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.870 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:49.870 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.538363+0000", 2024-04-19T01:04:49.870 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7600000000000001e-06 2024-04-19T01:04:49.870 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.870 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.870 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:49.870 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.540481+0000", 2024-04-19T01:04:49.870 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.002117833 2024-04-19T01:04:49.870 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.871 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.871 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:49.871 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.540516+0000", 2024-04-19T01:04:49.871 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.5435000000000002e-05 2024-04-19T01:04:49.871 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.871 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.871 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:49.871 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.540539+0000", 2024-04-19T01:04:49.871 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3136e-05 2024-04-19T01:04:49.871 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.871 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.871 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:49.871 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.540906+0000", 2024-04-19T01:04:49.871 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00036633199999999997 2024-04-19T01:04:49.871 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.872 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.872 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:49.872 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.540932+0000", 2024-04-19T01:04:49.872 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5972000000000001e-05 2024-04-19T01:04:49.872 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.872 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.872 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:49.872 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.567132+0000", 2024-04-19T01:04:49.872 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.026200655999999999 2024-04-19T01:04:49.872 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.872 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.872 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:49.872 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.567204+0000", 2024-04-19T01:04:49.873 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.1394000000000001e-05 2024-04-19T01:04:49.873 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:49.873 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:49.873 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:49.873 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.873 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.873 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:365 3.0s0 3:0e44b6f6:::smithi15528762-26:head [read 0~1605632,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-04-19T01:04:49.873 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:19.549248+0000", 2024-04-19T01:04:49.873 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 30.213157968000001, 2024-04-19T01:04:49.873 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.026478386999999999, 2024-04-19T01:04:49.873 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:49.873 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:49.873 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:04:49.873 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:49.874 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:49.874 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:49.874 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 365 2024-04-19T01:04:49.874 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.874 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:49.874 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.874 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:49.874 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549248+0000", 2024-04-19T01:04:49.874 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:49.874 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.874 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.874 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:49.874 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549248+0000", 2024-04-19T01:04:49.874 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:49.874 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.875 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.875 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:49.875 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549251+0000", 2024-04-19T01:04:49.875 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.8169999999999999e-06 2024-04-19T01:04:49.875 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.875 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.875 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:49.875 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549262+0000", 2024-04-19T01:04:49.875 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.1449e-05 2024-04-19T01:04:49.875 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.875 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.875 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:49.875 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549288+0000", 2024-04-19T01:04:49.875 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6169e-05 2024-04-19T01:04:49.876 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.876 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.876 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:49.876 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549312+0000", 2024-04-19T01:04:49.876 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3833e-05 2024-04-19T01:04:49.876 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.876 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.876 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:49.876 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.566277+0000", 2024-04-19T01:04:49.876 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.016964888000000001 2024-04-19T01:04:49.876 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.876 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.876 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:49.876 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.566503+0000", 2024-04-19T01:04:49.876 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00022616800000000001 2024-04-19T01:04:49.877 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.877 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.877 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:49.877 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.575726+0000", 2024-04-19T01:04:49.877 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0092230630000000001 2024-04-19T01:04:49.877 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:49.877 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:49.877 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:49.877 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.877 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.877 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:366 3.0s0 3:0e44b6f6:::smithi15528762-26:head [read 0~1605632,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-04-19T01:04:49.877 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:19.549572+0000", 2024-04-19T01:04:49.877 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 30.212833543999999, 2024-04-19T01:04:49.877 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.028385865999999999, 2024-04-19T01:04:49.877 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:49.878 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:49.878 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:04:49.878 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:49.878 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:49.878 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:49.878 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 366 2024-04-19T01:04:49.878 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.878 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:49.878 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.878 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:49.878 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549572+0000", 2024-04-19T01:04:49.878 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:49.878 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.878 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.879 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:49.879 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549572+0000", 2024-04-19T01:04:49.879 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:49.879 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.879 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.879 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:49.879 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549574+0000", 2024-04-19T01:04:49.879 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0260000000000002e-06 2024-04-19T01:04:49.879 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.879 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.879 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:49.879 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549588+0000", 2024-04-19T01:04:49.879 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.3737000000000001e-05 2024-04-19T01:04:49.879 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.879 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.880 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:49.880 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549616+0000", 2024-04-19T01:04:49.880 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.7619000000000001e-05 2024-04-19T01:04:49.880 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.880 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.880 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:49.880 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549633+0000", 2024-04-19T01:04:49.880 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7025999999999999e-05 2024-04-19T01:04:49.880 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.880 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.880 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:49.880 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.567197+0000", 2024-04-19T01:04:49.880 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.017564377999999999 2024-04-19T01:04:49.880 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.881 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.881 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:49.881 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.567461+0000", 2024-04-19T01:04:49.881 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00026393899999999998 2024-04-19T01:04:49.881 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.881 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.881 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:49.881 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.577958+0000", 2024-04-19T01:04:49.881 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.010497141 2024-04-19T01:04:49.881 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:49.881 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:49.881 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:49.881 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.881 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.882 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:367 3.0s0 3:0e44b6f6:::smithi15528762-26:head [read 0~1605632,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-04-19T01:04:49.882 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:19.549659+0000", 2024-04-19T01:04:49.882 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 30.212746737, 2024-04-19T01:04:49.882 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.030292887000000001, 2024-04-19T01:04:49.882 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:49.882 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:49.882 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:04:49.882 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:49.882 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:49.882 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:49.882 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 367 2024-04-19T01:04:49.882 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.882 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:49.883 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.883 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:49.883 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549659+0000", 2024-04-19T01:04:49.883 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:49.883 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.883 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.883 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:49.883 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549659+0000", 2024-04-19T01:04:49.883 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:49.883 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.883 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.883 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:49.883 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549661+0000", 2024-04-19T01:04:49.883 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2500000000000001e-06 2024-04-19T01:04:49.883 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.884 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.884 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:49.884 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549669+0000", 2024-04-19T01:04:49.884 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.7060000000000004e-06 2024-04-19T01:04:49.884 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.884 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.884 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:49.884 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549688+0000", 2024-04-19T01:04:49.884 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9309000000000001e-05 2024-04-19T01:04:49.884 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.884 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.884 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:49.884 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549704+0000", 2024-04-19T01:04:49.884 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5398999999999999e-05 2024-04-19T01:04:49.884 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.885 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.885 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:49.885 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.570745+0000", 2024-04-19T01:04:49.885 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.021041432999999998 2024-04-19T01:04:49.885 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.885 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.885 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:49.885 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.570983+0000", 2024-04-19T01:04:49.885 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00023819299999999999 2024-04-19T01:04:49.885 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.885 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.885 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:49.885 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.579952+0000", 2024-04-19T01:04:49.885 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.008968597 2024-04-19T01:04:49.886 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:49.886 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:49.886 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:49.886 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.886 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.886 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.8s2 60/31 ECSubWrite(tid=225, reqid=client.4604.0:433, at_version=60'43, trim_to=0'0, roll_forward_to=59'41))", 2024-04-19T01:04:49.886 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:30.570124+0000", 2024-04-19T01:04:49.886 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 19.192281452, 2024-04-19T01:04:49.886 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.027307438, 2024-04-19T01:04:49.886 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:49.886 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:49.886 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:49.886 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:49.886 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.887 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:49.887 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.570124+0000", 2024-04-19T01:04:49.887 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:49.887 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.887 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.887 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:49.887 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.570124+0000", 2024-04-19T01:04:49.887 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:49.887 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.887 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.887 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:49.887 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.570125+0000", 2024-04-19T01:04:49.887 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.5400000000000001e-07 2024-04-19T01:04:49.887 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.887 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.888 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:49.888 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.570710+0000", 2024-04-19T01:04:49.888 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000585405 2024-04-19T01:04:49.888 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.888 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.888 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:49.888 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.570736+0000", 2024-04-19T01:04:49.888 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5511999999999999e-05 2024-04-19T01:04:49.888 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.888 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.888 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:49.888 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.570751+0000", 2024-04-19T01:04:49.888 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.4817000000000001e-05 2024-04-19T01:04:49.888 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.888 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.889 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:49.889 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.570881+0000", 2024-04-19T01:04:49.889 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000130586 2024-04-19T01:04:49.889 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.889 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.889 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:49.889 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.570900+0000", 2024-04-19T01:04:49.889 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.8868999999999998e-05 2024-04-19T01:04:49.889 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.889 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.889 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:49.889 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.597353+0000", 2024-04-19T01:04:49.889 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.026452750000000001 2024-04-19T01:04:49.889 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.889 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.890 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:49.890 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.597432+0000", 2024-04-19T01:04:49.890 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.8844999999999995e-05 2024-04-19T01:04:49.890 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:49.890 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:49.890 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:49.890 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.890 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.890 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.as5 65/31 ECSubWrite(tid=108, reqid=client.4604.0:479, at_version=65'27, trim_to=0'0, roll_forward_to=64'24))", 2024-04-19T01:04:49.890 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:35.655493+0000", 2024-04-19T01:04:49.890 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 14.106912855999999, 2024-04-19T01:04:49.890 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.020988125999999999, 2024-04-19T01:04:49.890 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:49.890 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:49.891 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:49.891 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:49.891 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.891 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:49.891 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.655493+0000", 2024-04-19T01:04:49.891 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:49.891 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.891 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.891 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:49.891 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.655493+0000", 2024-04-19T01:04:49.891 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:49.891 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.891 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.891 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:49.891 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.655495+0000", 2024-04-19T01:04:49.892 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.717e-06 2024-04-19T01:04:49.892 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.892 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.892 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:49.892 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.655648+0000", 2024-04-19T01:04:49.892 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015314199999999999 2024-04-19T01:04:49.892 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.892 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.892 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:49.892 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.655679+0000", 2024-04-19T01:04:49.892 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.1238999999999997e-05 2024-04-19T01:04:49.892 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.892 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.892 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:49.892 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.655695+0000", 2024-04-19T01:04:49.893 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5727e-05 2024-04-19T01:04:49.893 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.893 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.893 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:49.893 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.655802+0000", 2024-04-19T01:04:49.893 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00010730699999999999 2024-04-19T01:04:49.893 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.893 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.893 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:49.893 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.655820+0000", 2024-04-19T01:04:49.893 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.8076000000000001e-05 2024-04-19T01:04:49.893 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.893 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.893 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:49.893 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.676385+0000", 2024-04-19T01:04:49.894 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.020564747000000001 2024-04-19T01:04:49.894 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:49.894 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:49.894 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:49.894 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.676481+0000", 2024-04-19T01:04:49.894 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.6170999999999998e-05 2024-04-19T01:04:49.894 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:49.894 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:49.894 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:49.894 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:49.894 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:49.894 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:49.896 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-19T01:04:49.896 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:04:49.896+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:04:49.997 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:04:49.995+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:04:50.069 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:50.069 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [], 2024-04-19T01:04:50.069 INFO:teuthology.orchestra.run.smithi110.stdout: "num_ops": 0 2024-04-19T01:04:50.070 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:50.085 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-19T01:04:50.131 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:50.130+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:50.199 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:04:50.200+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:04:50.202 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:50.202 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [], 2024-04-19T01:04:50.202 INFO:teuthology.orchestra.run.smithi110.stdout: "complaint_time": 30, 2024-04-19T01:04:50.202 INFO:teuthology.orchestra.run.smithi110.stdout: "num_blocked_ops": 0 2024-04-19T01:04:50.202 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:50.216 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-19T01:04:50.301 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:04:50.300+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:04:50.378 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:50.378 INFO:teuthology.orchestra.run.smithi110.stdout: "size": 20, 2024-04-19T01:04:50.379 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 600, 2024-04-19T01:04:50.379 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [ 2024-04-19T01:04:50.379 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.379 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs2 43/31 ECSubWrite(tid=47, reqid=client.4604.0:263, at_version=43'8, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:04:50.379 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.995028+0000", 2024-04-19T01:04:50.379 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 36.38275265, 2024-04-19T01:04:50.379 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.038142937000000002, 2024-04-19T01:04:50.379 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:50.380 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:50.380 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:50.380 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:50.380 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.380 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:50.380 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995028+0000", 2024-04-19T01:04:50.380 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.380 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.380 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.380 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:50.380 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995028+0000", 2024-04-19T01:04:50.380 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.380 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.380 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.381 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:50.381 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995029+0000", 2024-04-19T01:04:50.381 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.47e-07 2024-04-19T01:04:50.381 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.381 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.381 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:50.381 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995397+0000", 2024-04-19T01:04:50.381 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00036862599999999999 2024-04-19T01:04:50.381 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.381 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.381 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:50.382 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995437+0000", 2024-04-19T01:04:50.382 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.9564999999999998e-05 2024-04-19T01:04:50.382 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.382 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.382 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:50.382 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995461+0000", 2024-04-19T01:04:50.382 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4261999999999999e-05 2024-04-19T01:04:50.382 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.382 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.382 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:50.382 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995661+0000", 2024-04-19T01:04:50.382 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00019937500000000001 2024-04-19T01:04:50.382 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.383 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.383 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:50.383 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995689+0000", 2024-04-19T01:04:50.383 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.8282000000000001e-05 2024-04-19T01:04:50.383 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.383 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.383 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:50.383 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.033059+0000", 2024-04-19T01:04:50.383 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.037370288000000002 2024-04-19T01:04:50.383 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.383 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.383 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:50.383 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.033171+0000", 2024-04-19T01:04:50.384 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000111692 2024-04-19T01:04:50.384 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.384 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:50.384 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.384 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.384 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.384 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs2 43/31 ECSubWrite(tid=48, reqid=client.4604.0:264, at_version=43'9, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:04:50.384 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.005925+0000", 2024-04-19T01:04:50.384 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 36.371855994000001, 2024-04-19T01:04:50.384 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.028007243000000001, 2024-04-19T01:04:50.384 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:50.384 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:50.384 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:50.385 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:50.385 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.385 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:50.385 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.005925+0000", 2024-04-19T01:04:50.385 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.385 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.385 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.385 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:50.385 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.005925+0000", 2024-04-19T01:04:50.385 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.385 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.385 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.385 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:50.386 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.005926+0000", 2024-04-19T01:04:50.386 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9700000000000002e-06 2024-04-19T01:04:50.386 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.386 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.386 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:50.386 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006164+0000", 2024-04-19T01:04:50.386 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00023784199999999999 2024-04-19T01:04:50.386 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.386 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.386 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:50.386 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006203+0000", 2024-04-19T01:04:50.386 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.9171999999999997e-05 2024-04-19T01:04:50.386 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.387 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.387 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:50.387 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006235+0000", 2024-04-19T01:04:50.387 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.1937999999999998e-05 2024-04-19T01:04:50.387 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.387 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.387 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:50.387 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.015013+0000", 2024-04-19T01:04:50.387 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0087775909999999995 2024-04-19T01:04:50.387 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.387 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.387 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:50.387 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.015043+0000", 2024-04-19T01:04:50.388 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.9739000000000001e-05 2024-04-19T01:04:50.388 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.388 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.388 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:50.388 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.033841+0000", 2024-04-19T01:04:50.388 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.018797861999999999 2024-04-19T01:04:50.388 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.388 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.388 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:50.388 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.033932+0000", 2024-04-19T01:04:50.388 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.1129000000000001e-05 2024-04-19T01:04:50.388 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.388 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:50.388 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.389 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.389 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.389 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs2 43/31 ECSubWrite(tid=49, reqid=client.4604.0:265, at_version=43'10, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:04:50.389 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.006295+0000", 2024-04-19T01:04:50.389 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 36.371485223999997, 2024-04-19T01:04:50.389 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.02899026, 2024-04-19T01:04:50.389 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:50.389 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:50.389 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:50.389 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:50.389 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.389 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:50.389 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006295+0000", 2024-04-19T01:04:50.390 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.390 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.390 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.390 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:50.390 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006295+0000", 2024-04-19T01:04:50.390 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.390 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.390 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.390 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:50.390 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006296+0000", 2024-04-19T01:04:50.390 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.1230000000000001e-06 2024-04-19T01:04:50.390 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.390 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.390 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:50.391 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006329+0000", 2024-04-19T01:04:50.391 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.2465000000000002e-05 2024-04-19T01:04:50.391 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.391 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.391 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:50.391 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006362+0000", 2024-04-19T01:04:50.391 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.3118999999999999e-05 2024-04-19T01:04:50.391 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.391 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.391 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:50.391 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006401+0000", 2024-04-19T01:04:50.391 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.9347999999999997e-05 2024-04-19T01:04:50.391 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.391 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.392 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:50.392 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.033284+0000", 2024-04-19T01:04:50.392 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.026882553999999999 2024-04-19T01:04:50.392 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.392 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.392 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:50.392 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.033308+0000", 2024-04-19T01:04:50.392 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4295999999999999e-05 2024-04-19T01:04:50.392 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.392 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.392 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:50.392 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.035213+0000", 2024-04-19T01:04:50.392 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0019049749999999999 2024-04-19T01:04:50.392 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.392 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.393 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:50.393 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.035286+0000", 2024-04-19T01:04:50.393 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.2379999999999995e-05 2024-04-19T01:04:50.393 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.393 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:50.393 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.393 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.393 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.393 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs2 48/31 ECSubWrite(tid=76, reqid=client.4604.0:345, at_version=48'22, trim_to=0'0, roll_forward_to=47'19))", 2024-04-19T01:04:50.393 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.538229+0000", 2024-04-19T01:04:50.393 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 31.839551017000002, 2024-04-19T01:04:50.393 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.041621538999999999, 2024-04-19T01:04:50.393 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:50.393 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:50.394 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:50.394 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:50.394 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.394 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:50.394 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538229+0000", 2024-04-19T01:04:50.394 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.394 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.394 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.394 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:50.394 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538229+0000", 2024-04-19T01:04:50.394 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.394 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.394 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.395 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:50.395 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538232+0000", 2024-04-19T01:04:50.395 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1730000000000002e-06 2024-04-19T01:04:50.395 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.395 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.395 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:50.395 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538714+0000", 2024-04-19T01:04:50.395 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00048223299999999999 2024-04-19T01:04:50.395 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.395 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.395 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:50.395 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538755+0000", 2024-04-19T01:04:50.395 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.1044999999999998e-05 2024-04-19T01:04:50.395 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.396 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.396 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:50.396 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538779+0000", 2024-04-19T01:04:50.396 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3828999999999998e-05 2024-04-19T01:04:50.396 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.396 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.396 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:50.397 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.539007+0000", 2024-04-19T01:04:50.397 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00022818400000000001 2024-04-19T01:04:50.397 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.397 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.397 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:50.397 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.539033+0000", 2024-04-19T01:04:50.397 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6455999999999999e-05 2024-04-19T01:04:50.397 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.397 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.397 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:50.397 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.579777+0000", 2024-04-19T01:04:50.397 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.040743133000000001 2024-04-19T01:04:50.397 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.398 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.398 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:50.398 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.579851+0000", 2024-04-19T01:04:50.398 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.4486000000000005e-05 2024-04-19T01:04:50.398 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.398 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:50.398 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.398 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.398 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.398 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs2 48/31 ECSubWrite(tid=78, reqid=client.4604.0:346, at_version=48'23, trim_to=0'0, roll_forward_to=48'21))", 2024-04-19T01:04:50.398 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.549034+0000", 2024-04-19T01:04:50.398 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 31.828746974000001, 2024-04-19T01:04:50.398 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.030907911999999999, 2024-04-19T01:04:50.398 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:50.399 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:50.399 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:50.399 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:50.399 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.399 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:50.399 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.549034+0000", 2024-04-19T01:04:50.399 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.399 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.399 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.399 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:50.399 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.549034+0000", 2024-04-19T01:04:50.399 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.399 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.400 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.400 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:50.400 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.549035+0000", 2024-04-19T01:04:50.400 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9429999999999999e-06 2024-04-19T01:04:50.400 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.400 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.400 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:50.400 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.549509+0000", 2024-04-19T01:04:50.400 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00047399999999999997 2024-04-19T01:04:50.400 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.400 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.400 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:50.400 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.549541+0000", 2024-04-19T01:04:50.400 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.1843e-05 2024-04-19T01:04:50.401 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.401 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.401 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:50.401 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.549563+0000", 2024-04-19T01:04:50.401 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2045e-05 2024-04-19T01:04:50.401 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.401 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.401 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:50.401 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.558487+0000", 2024-04-19T01:04:50.401 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0089238470000000004 2024-04-19T01:04:50.401 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.401 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.401 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:50.402 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.558508+0000", 2024-04-19T01:04:50.402 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1236000000000001e-05 2024-04-19T01:04:50.402 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.402 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.402 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:50.402 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.579870+0000", 2024-04-19T01:04:50.402 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.021361601000000001 2024-04-19T01:04:50.402 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.402 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.402 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:50.402 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.579941+0000", 2024-04-19T01:04:50.402 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.1396999999999994e-05 2024-04-19T01:04:50.403 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.403 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:50.403 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.403 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.403 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.403 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs2 48/31 ECSubWrite(tid=79, reqid=client.4604.0:347, at_version=48'24, trim_to=0'0, roll_forward_to=48'21))", 2024-04-19T01:04:50.403 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.555772+0000", 2024-04-19T01:04:50.403 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 31.822008147999998, 2024-04-19T01:04:50.403 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.024949988999999999, 2024-04-19T01:04:50.403 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:50.403 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:50.403 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:50.403 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:50.403 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.404 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:50.404 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.555772+0000", 2024-04-19T01:04:50.404 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.404 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.404 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.404 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:50.404 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.555772+0000", 2024-04-19T01:04:50.404 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.404 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.404 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.404 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:50.404 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.555774+0000", 2024-04-19T01:04:50.404 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.734e-06 2024-04-19T01:04:50.405 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.405 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.405 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:50.405 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.555841+0000", 2024-04-19T01:04:50.405 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.6434999999999997e-05 2024-04-19T01:04:50.405 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.405 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.405 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:50.405 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.555877+0000", 2024-04-19T01:04:50.405 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.6544000000000001e-05 2024-04-19T01:04:50.405 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.405 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.406 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:50.406 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.555895+0000", 2024-04-19T01:04:50.406 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7558999999999999e-05 2024-04-19T01:04:50.406 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.406 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.406 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:50.406 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.576874+0000", 2024-04-19T01:04:50.406 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.020979564999999999 2024-04-19T01:04:50.406 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.406 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.406 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:50.406 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.576898+0000", 2024-04-19T01:04:50.406 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3787000000000001e-05 2024-04-19T01:04:50.407 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.407 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.407 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:50.407 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.580616+0000", 2024-04-19T01:04:50.407 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0037177289999999999 2024-04-19T01:04:50.407 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.407 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.407 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:50.407 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.580722+0000", 2024-04-19T01:04:50.407 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000106636 2024-04-19T01:04:50.408 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.408 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:50.408 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.408 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.408 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.408 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.8s6 48/31 ECSubWrite(tid=199, reqid=client.4604.0:353, at_version=0'0, trim_to=0'0, roll_forward_to=48'20))", 2024-04-19T01:04:50.408 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.557241+0000", 2024-04-19T01:04:50.408 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 31.820539813, 2024-04-19T01:04:50.408 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.020337572000000002, 2024-04-19T01:04:50.408 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:50.409 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:50.409 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:50.409 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:50.409 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.409 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:50.409 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557241+0000", 2024-04-19T01:04:50.409 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.409 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.409 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.409 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:50.409 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557241+0000", 2024-04-19T01:04:50.409 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.409 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.410 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.410 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:50.410 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557242+0000", 2024-04-19T01:04:50.410 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.4449999999999999e-06 2024-04-19T01:04:50.410 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.410 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.410 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:50.410 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557262+0000", 2024-04-19T01:04:50.410 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0103999999999999e-05 2024-04-19T01:04:50.411 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.411 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.411 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:50.411 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557287+0000", 2024-04-19T01:04:50.411 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4556e-05 2024-04-19T01:04:50.411 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.411 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.411 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:50.411 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557304+0000", 2024-04-19T01:04:50.411 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7592e-05 2024-04-19T01:04:50.411 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.411 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.411 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:50.411 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557935+0000", 2024-04-19T01:04:50.412 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00063064499999999999 2024-04-19T01:04:50.412 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.412 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.412 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:50.412 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557959+0000", 2024-04-19T01:04:50.412 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3736000000000001e-05 2024-04-19T01:04:50.412 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.412 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.412 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:50.412 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.577484+0000", 2024-04-19T01:04:50.412 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.019525415000000001 2024-04-19T01:04:50.412 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.412 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.412 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:50.413 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.577578+0000", 2024-04-19T01:04:50.413 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.4079000000000005e-05 2024-04-19T01:04:50.413 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.413 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:50.413 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.413 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.413 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.413 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.es2 48/31 ECSubWrite(tid=118, reqid=client.4604.0:357, at_version=48'25, trim_to=0'0, roll_forward_to=48'23))", 2024-04-19T01:04:50.413 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.557362+0000", 2024-04-19T01:04:50.413 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 31.820418726, 2024-04-19T01:04:50.413 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.020223954999999998, 2024-04-19T01:04:50.413 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:50.413 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:50.413 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:50.413 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:50.414 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.414 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:50.414 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557362+0000", 2024-04-19T01:04:50.414 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.414 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.414 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.414 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:50.414 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557362+0000", 2024-04-19T01:04:50.414 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.414 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.414 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.414 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:50.414 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557363+0000", 2024-04-19T01:04:50.414 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.7499999999999999e-07 2024-04-19T01:04:50.415 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.415 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.415 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:50.415 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557396+0000", 2024-04-19T01:04:50.415 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.3878999999999999e-05 2024-04-19T01:04:50.415 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.415 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.415 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:50.415 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557421+0000", 2024-04-19T01:04:50.415 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4627999999999999e-05 2024-04-19T01:04:50.415 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.415 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.415 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:50.415 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557435+0000", 2024-04-19T01:04:50.416 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.4185000000000001e-05 2024-04-19T01:04:50.416 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.416 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.416 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:50.416 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.558839+0000", 2024-04-19T01:04:50.416 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0014035860000000001 2024-04-19T01:04:50.416 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.416 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.416 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:50.416 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.558863+0000", 2024-04-19T01:04:50.416 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4434e-05 2024-04-19T01:04:50.416 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.416 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.417 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:50.417 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.577527+0000", 2024-04-19T01:04:50.417 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.018663800000000001 2024-04-19T01:04:50.417 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.417 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.417 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:50.417 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.577586+0000", 2024-04-19T01:04:50.417 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.8668000000000001e-05 2024-04-19T01:04:50.417 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.417 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:50.417 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.417 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.417 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.417 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.es2 48/31 ECSubWrite(tid=120, reqid=client.4604.0:358, at_version=48'26, trim_to=0'0, roll_forward_to=48'23))", 2024-04-19T01:04:50.418 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.558551+0000", 2024-04-19T01:04:50.418 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 31.819229775, 2024-04-19T01:04:50.418 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.019082335999999998, 2024-04-19T01:04:50.418 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:50.418 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:50.418 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:50.418 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:50.418 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.418 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:50.418 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.558551+0000", 2024-04-19T01:04:50.418 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.418 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.418 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.419 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:50.419 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.558551+0000", 2024-04-19T01:04:50.419 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.419 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.419 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.419 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:50.419 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.558552+0000", 2024-04-19T01:04:50.419 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.4929999999999999e-06 2024-04-19T01:04:50.419 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.419 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.419 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:50.419 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.558588+0000", 2024-04-19T01:04:50.420 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.6125999999999999e-05 2024-04-19T01:04:50.420 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.420 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.420 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:50.420 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.558611+0000", 2024-04-19T01:04:50.420 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2959000000000002e-05 2024-04-19T01:04:50.420 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.420 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.420 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:50.420 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.558633+0000", 2024-04-19T01:04:50.420 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1815e-05 2024-04-19T01:04:50.420 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.420 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.420 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:50.421 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.565266+0000", 2024-04-19T01:04:50.421 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0066330060000000003 2024-04-19T01:04:50.421 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.421 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.421 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:50.421 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.565300+0000", 2024-04-19T01:04:50.421 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.3671999999999999e-05 2024-04-19T01:04:50.421 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.421 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.421 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:50.421 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.577592+0000", 2024-04-19T01:04:50.421 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.012291905000000001 2024-04-19T01:04:50.421 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.421 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.422 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:50.422 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.577633+0000", 2024-04-19T01:04:50.422 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.1359999999999997e-05 2024-04-19T01:04:50.422 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.422 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:50.422 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.422 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.422 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.422 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs2 48/31 ECSubWrite(tid=80, reqid=client.4604.0:348, at_version=48'25, trim_to=0'0, roll_forward_to=48'21))", 2024-04-19T01:04:50.422 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.561184+0000", 2024-04-19T01:04:50.423 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 31.816596776000001, 2024-04-19T01:04:50.423 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.021208286, 2024-04-19T01:04:50.423 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:50.423 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:50.423 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:50.423 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:50.423 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.423 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:50.423 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.561184+0000", 2024-04-19T01:04:50.423 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.423 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.423 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.423 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:50.423 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.561184+0000", 2024-04-19T01:04:50.424 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.424 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.424 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.424 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:50.424 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.561185+0000", 2024-04-19T01:04:50.424 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.685e-06 2024-04-19T01:04:50.424 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.424 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.424 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:50.424 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.561230+0000", 2024-04-19T01:04:50.424 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.4867000000000003e-05 2024-04-19T01:04:50.424 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.425 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.425 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:50.425 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.561282+0000", 2024-04-19T01:04:50.425 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.1712999999999998e-05 2024-04-19T01:04:50.425 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.425 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.425 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:50.425 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.561304+0000", 2024-04-19T01:04:50.425 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2209999999999999e-05 2024-04-19T01:04:50.425 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.425 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.425 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:50.425 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.580080+0000", 2024-04-19T01:04:50.425 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.018775475999999999 2024-04-19T01:04:50.425 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.426 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.426 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:50.426 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.580193+0000", 2024-04-19T01:04:50.426 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000113244 2024-04-19T01:04:50.426 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.426 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.426 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:50.426 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.582324+0000", 2024-04-19T01:04:50.426 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0021313790000000001 2024-04-19T01:04:50.426 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.426 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.426 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:50.426 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.582392+0000", 2024-04-19T01:04:50.427 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.7712000000000005e-05 2024-04-19T01:04:50.427 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.427 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:50.427 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.427 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.427 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.427 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.8s6 49/31 ECSubWrite(tid=203, reqid=client.4604.0:360, at_version=49'25, trim_to=0'0, roll_forward_to=49'24))", 2024-04-19T01:04:50.427 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:19.538676+0000", 2024-04-19T01:04:50.427 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 30.839104758000001, 2024-04-19T01:04:50.427 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.019513513, 2024-04-19T01:04:50.427 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:50.427 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:50.427 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:50.427 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:50.427 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.428 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:50.428 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.538676+0000", 2024-04-19T01:04:50.428 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.428 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.428 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.428 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:50.428 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.538676+0000", 2024-04-19T01:04:50.428 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.428 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.429 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.429 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:50.429 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.538678+0000", 2024-04-19T01:04:50.429 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0260000000000002e-06 2024-04-19T01:04:50.429 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.429 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.429 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:50.429 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.538968+0000", 2024-04-19T01:04:50.429 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000290172 2024-04-19T01:04:50.429 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.429 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.430 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:50.430 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.539364+0000", 2024-04-19T01:04:50.430 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00039640399999999999 2024-04-19T01:04:50.430 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.430 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.430 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:50.430 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.539393+0000", 2024-04-19T01:04:50.430 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.8912e-05 2024-04-19T01:04:50.430 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.430 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.431 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:50.431 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.539604+0000", 2024-04-19T01:04:50.431 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00021122900000000001 2024-04-19T01:04:50.431 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.431 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.431 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:50.431 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.539630+0000", 2024-04-19T01:04:50.431 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5151999999999999e-05 2024-04-19T01:04:50.431 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.431 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.431 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:50.431 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.558098+0000", 2024-04-19T01:04:50.431 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.018468232000000001 2024-04-19T01:04:50.432 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.432 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.432 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:50.432 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.558189+0000", 2024-04-19T01:04:50.432 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.1385999999999995e-05 2024-04-19T01:04:50.432 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.432 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:50.432 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.432 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.432 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.432 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.8s6 49/31 ECSubWrite(tid=204, reqid=client.4604.0:361, at_version=49'26, trim_to=0'0, roll_forward_to=49'24))", 2024-04-19T01:04:50.432 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:19.560727+0000", 2024-04-19T01:04:50.432 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 30.817053131000002, 2024-04-19T01:04:50.432 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.018075543999999999, 2024-04-19T01:04:50.433 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:50.433 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:50.433 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:50.433 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:50.433 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.433 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:50.433 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.560727+0000", 2024-04-19T01:04:50.433 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.433 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.433 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.433 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:50.433 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.560727+0000", 2024-04-19T01:04:50.433 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.433 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.434 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.434 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:50.434 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.560730+0000", 2024-04-19T01:04:50.434 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.9299999999999999e-06 2024-04-19T01:04:50.434 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.434 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.434 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:50.434 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.560956+0000", 2024-04-19T01:04:50.434 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00022553100000000001 2024-04-19T01:04:50.434 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.434 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.434 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:50.435 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.560989+0000", 2024-04-19T01:04:50.435 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.3646000000000003e-05 2024-04-19T01:04:50.435 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.435 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.435 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:50.435 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.561012+0000", 2024-04-19T01:04:50.435 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2742e-05 2024-04-19T01:04:50.435 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.435 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.436 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:50.436 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.561272+0000", 2024-04-19T01:04:50.436 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000260067 2024-04-19T01:04:50.436 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.436 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.436 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:50.436 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.561298+0000", 2024-04-19T01:04:50.436 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5862000000000001e-05 2024-04-19T01:04:50.436 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.436 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.436 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:50.436 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.578729+0000", 2024-04-19T01:04:50.436 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.017430977 2024-04-19T01:04:50.436 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.436 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.437 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:50.437 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.578803+0000", 2024-04-19T01:04:50.437 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.3788999999999994e-05 2024-04-19T01:04:50.437 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.437 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:50.437 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.437 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.437 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.437 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.5s6 59/31 ECSubWrite(tid=66, reqid=client.4604.0:423, at_version=59'51, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:04:50.437 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:29.540851+0000", 2024-04-19T01:04:50.438 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 20.836929482999999, 2024-04-19T01:04:50.438 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.030615177, 2024-04-19T01:04:50.438 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:50.438 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:50.438 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:50.438 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:50.438 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.438 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:50.438 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.540851+0000", 2024-04-19T01:04:50.438 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.438 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.438 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.438 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:50.438 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.540851+0000", 2024-04-19T01:04:50.439 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.439 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.439 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.439 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:50.439 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.540852+0000", 2024-04-19T01:04:50.439 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.6400000000000002e-07 2024-04-19T01:04:50.439 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.439 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.439 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:50.439 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.541398+0000", 2024-04-19T01:04:50.439 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00054620600000000004 2024-04-19T01:04:50.439 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.439 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.439 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:50.440 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.541433+0000", 2024-04-19T01:04:50.440 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.4853e-05 2024-04-19T01:04:50.440 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.440 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.440 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:50.440 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.541457+0000", 2024-04-19T01:04:50.440 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4641999999999999e-05 2024-04-19T01:04:50.440 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.440 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.440 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:50.440 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.541639+0000", 2024-04-19T01:04:50.440 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00018139100000000001 2024-04-19T01:04:50.440 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.440 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.441 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:50.441 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.541667+0000", 2024-04-19T01:04:50.441 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.7943e-05 2024-04-19T01:04:50.441 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.441 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.441 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:50.441 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.571413+0000", 2024-04-19T01:04:50.441 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0297462 2024-04-19T01:04:50.441 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.441 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.441 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:50.441 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.571466+0000", 2024-04-19T01:04:50.441 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.3378000000000003e-05 2024-04-19T01:04:50.441 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.442 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:50.442 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.442 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.442 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.442 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.5s6 59/31 ECSubWrite(tid=67, reqid=client.4604.0:424, at_version=59'52, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:04:50.442 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:29.548369+0000", 2024-04-19T01:04:50.442 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 20.829411739000001, 2024-04-19T01:04:50.442 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.023140674999999999, 2024-04-19T01:04:50.442 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:50.442 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:50.442 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:50.442 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:50.442 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.442 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:50.443 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.548369+0000", 2024-04-19T01:04:50.443 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.443 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.443 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.443 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:50.443 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.548369+0000", 2024-04-19T01:04:50.443 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.443 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.443 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.443 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:50.443 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.548370+0000", 2024-04-19T01:04:50.443 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.066e-06 2024-04-19T01:04:50.443 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.443 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.443 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:50.444 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.548631+0000", 2024-04-19T01:04:50.444 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00026157799999999999 2024-04-19T01:04:50.444 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.444 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.444 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:50.444 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.548648+0000", 2024-04-19T01:04:50.444 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7079e-05 2024-04-19T01:04:50.444 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.444 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.444 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:50.444 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.548657+0000", 2024-04-19T01:04:50.444 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.9500000000000007e-06 2024-04-19T01:04:50.444 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.444 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.445 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:50.445 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.555857+0000", 2024-04-19T01:04:50.445 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0071992569999999997 2024-04-19T01:04:50.445 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.445 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.445 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:50.445 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.555883+0000", 2024-04-19T01:04:50.445 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6522000000000001e-05 2024-04-19T01:04:50.445 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.445 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.445 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:50.446 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.571475+0000", 2024-04-19T01:04:50.446 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.015591727999999999 2024-04-19T01:04:50.446 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.446 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.446 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:50.446 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.571509+0000", 2024-04-19T01:04:50.446 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.4495000000000001e-05 2024-04-19T01:04:50.446 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.446 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:50.446 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.446 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.446 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.446 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.8s6 60/31 ECSubWrite(tid=225, reqid=client.4604.0:433, at_version=60'43, trim_to=0'0, roll_forward_to=59'41))", 2024-04-19T01:04:50.447 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:30.569927+0000", 2024-04-19T01:04:50.447 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 19.807853442999999, 2024-04-19T01:04:50.447 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.027044315999999999, 2024-04-19T01:04:50.447 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:50.447 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:50.447 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:50.447 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:50.447 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.447 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:50.447 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.569927+0000", 2024-04-19T01:04:50.447 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.447 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.447 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.447 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:50.448 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.569927+0000", 2024-04-19T01:04:50.448 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.448 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.448 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.448 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:50.448 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.569927+0000", 2024-04-19T01:04:50.448 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.34e-07 2024-04-19T01:04:50.448 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.448 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.448 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:50.448 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.570430+0000", 2024-04-19T01:04:50.448 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00050269400000000004 2024-04-19T01:04:50.448 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.448 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.449 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:50.449 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.570465+0000", 2024-04-19T01:04:50.449 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.4898999999999999e-05 2024-04-19T01:04:50.449 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.449 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.449 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:50.449 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.570490+0000", 2024-04-19T01:04:50.449 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4805999999999999e-05 2024-04-19T01:04:50.449 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.449 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.449 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:50.449 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.570639+0000", 2024-04-19T01:04:50.449 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00014890899999999999 2024-04-19T01:04:50.450 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.450 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.450 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:50.450 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.570665+0000", 2024-04-19T01:04:50.450 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5825000000000001e-05 2024-04-19T01:04:50.450 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.450 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.450 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:50.450 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.596871+0000", 2024-04-19T01:04:50.450 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.026206291999999999 2024-04-19T01:04:50.450 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.450 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.450 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:50.450 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.596971+0000", 2024-04-19T01:04:50.451 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000100457 2024-04-19T01:04:50.451 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.451 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:50.451 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.451 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.451 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.451 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.5s6 65/31 ECSubWrite(tid=75, reqid=client.4604.0:486, at_version=65'56, trim_to=0'0, roll_forward_to=59'54))", 2024-04-19T01:04:50.451 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:35.660833+0000", 2024-04-19T01:04:50.451 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 14.716947289, 2024-04-19T01:04:50.451 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.026740791999999999, 2024-04-19T01:04:50.451 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:50.452 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:50.452 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:50.452 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:50.452 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.452 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:50.452 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.660833+0000", 2024-04-19T01:04:50.452 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.452 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.452 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.452 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:50.452 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.660833+0000", 2024-04-19T01:04:50.452 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.453 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.453 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.453 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:50.453 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.660834+0000", 2024-04-19T01:04:50.453 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.7299999999999995e-07 2024-04-19T01:04:50.453 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.453 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.453 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:50.453 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.661764+0000", 2024-04-19T01:04:50.453 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00092978500000000003 2024-04-19T01:04:50.453 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.453 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.453 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:50.453 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.661794+0000", 2024-04-19T01:04:50.453 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.021e-05 2024-04-19T01:04:50.454 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.454 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.454 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:50.454 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.661810+0000", 2024-04-19T01:04:50.454 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6082e-05 2024-04-19T01:04:50.454 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.454 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.454 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:50.454 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.662014+0000", 2024-04-19T01:04:50.454 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000203625 2024-04-19T01:04:50.455 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.455 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.455 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:50.455 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.662030+0000", 2024-04-19T01:04:50.455 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6293000000000001e-05 2024-04-19T01:04:50.455 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.455 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.455 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:50.455 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.687509+0000", 2024-04-19T01:04:50.455 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.025479485999999999 2024-04-19T01:04:50.455 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.455 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.455 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:50.455 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.687574+0000", 2024-04-19T01:04:50.456 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.4638e-05 2024-04-19T01:04:50.456 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.456 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:50.456 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.456 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.456 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.456 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.5s6 65/31 ECSubWrite(tid=76, reqid=client.4604.0:487, at_version=65'57, trim_to=0'0, roll_forward_to=59'54))", 2024-04-19T01:04:50.456 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:35.673705+0000", 2024-04-19T01:04:50.456 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 14.704075249000001, 2024-04-19T01:04:50.456 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.019261430999999999, 2024-04-19T01:04:50.456 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:50.456 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:50.456 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:50.456 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:50.457 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.457 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:50.457 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.673705+0000", 2024-04-19T01:04:50.457 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.457 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.457 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.457 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:50.457 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.673705+0000", 2024-04-19T01:04:50.457 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.457 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.457 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.457 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:50.457 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.673707+0000", 2024-04-19T01:04:50.457 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9230000000000001e-06 2024-04-19T01:04:50.458 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.458 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.458 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:50.458 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.673908+0000", 2024-04-19T01:04:50.458 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00020053099999999999 2024-04-19T01:04:50.458 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.458 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.458 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:50.458 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.673939+0000", 2024-04-19T01:04:50.458 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.0825999999999999e-05 2024-04-19T01:04:50.458 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.458 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.458 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:50.458 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.673954+0000", 2024-04-19T01:04:50.459 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5642999999999999e-05 2024-04-19T01:04:50.459 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.459 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.459 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:50.459 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.679260+0000", 2024-04-19T01:04:50.459 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0053059739999999998 2024-04-19T01:04:50.459 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.459 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.459 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:50.459 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.679287+0000", 2024-04-19T01:04:50.459 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.7046000000000002e-05 2024-04-19T01:04:50.459 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.459 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.459 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:50.460 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.692868+0000", 2024-04-19T01:04:50.460 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.013580411000000001 2024-04-19T01:04:50.460 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.460 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.460 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:50.460 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.692967+0000", 2024-04-19T01:04:50.460 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.9077e-05 2024-04-19T01:04:50.460 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.460 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:50.460 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.460 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.460 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.460 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.5s6 74/31 ECSubWrite(tid=88, reqid=osd.8.0:88, at_version=74'69, trim_to=0'0, roll_forward_to=72'65))", 2024-04-19T01:04:50.460 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:44.559798+0000", 2024-04-19T01:04:50.461 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 5.8179824589999996, 2024-04-19T01:04:50.461 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.024864188999999998, 2024-04-19T01:04:50.461 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:50.461 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:50.461 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:50.461 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:50.461 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.461 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:50.461 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.559798+0000", 2024-04-19T01:04:50.461 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.461 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.461 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.461 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:50.461 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.559798+0000", 2024-04-19T01:04:50.461 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.462 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.462 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.462 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:50.462 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.559800+0000", 2024-04-19T01:04:50.462 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.88e-06 2024-04-19T01:04:50.462 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.462 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.462 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:50.462 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.559851+0000", 2024-04-19T01:04:50.462 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.0739000000000003e-05 2024-04-19T01:04:50.462 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.462 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.462 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:50.462 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.559882+0000", 2024-04-19T01:04:50.463 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.1038999999999999e-05 2024-04-19T01:04:50.463 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.463 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.463 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:50.463 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.559908+0000", 2024-04-19T01:04:50.463 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6460000000000001e-05 2024-04-19T01:04:50.463 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.463 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.463 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:50.463 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.564678+0000", 2024-04-19T01:04:50.464 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0047699079999999998 2024-04-19T01:04:50.464 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.464 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.464 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:50.464 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.564694+0000", 2024-04-19T01:04:50.464 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6056e-05 2024-04-19T01:04:50.464 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.464 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.464 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:50.464 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.584573+0000", 2024-04-19T01:04:50.464 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.019878939000000002 2024-04-19T01:04:50.465 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.465 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.465 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:50.465 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.584662+0000", 2024-04-19T01:04:50.465 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.9167999999999997e-05 2024-04-19T01:04:50.465 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.465 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:50.465 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.465 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.465 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.465 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.5s6 74/31 ECSubWrite(tid=89, reqid=client.4604.0:544, at_version=74'70, trim_to=0'0, roll_forward_to=72'65))", 2024-04-19T01:04:50.465 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:44.566121+0000", 2024-04-19T01:04:50.465 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 5.8116599530000004, 2024-04-19T01:04:50.465 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.026764973000000001, 2024-04-19T01:04:50.466 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:50.466 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:50.466 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:50.466 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:50.466 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.466 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:50.466 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.566121+0000", 2024-04-19T01:04:50.466 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.466 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.466 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.466 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:50.466 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.566121+0000", 2024-04-19T01:04:50.466 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.467 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.467 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.467 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:50.467 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.566123+0000", 2024-04-19T01:04:50.467 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0770000000000002e-06 2024-04-19T01:04:50.467 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.467 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.467 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:50.467 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.566618+0000", 2024-04-19T01:04:50.467 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000495296 2024-04-19T01:04:50.467 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.467 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.467 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:50.467 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.566651+0000", 2024-04-19T01:04:50.468 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.2795e-05 2024-04-19T01:04:50.468 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.468 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.468 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:50.468 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.566673+0000", 2024-04-19T01:04:50.468 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1926999999999999e-05 2024-04-19T01:04:50.468 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.468 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.468 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:50.468 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.566823+0000", 2024-04-19T01:04:50.468 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015024999999999999 2024-04-19T01:04:50.468 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.468 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.468 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:50.468 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.566852+0000", 2024-04-19T01:04:50.469 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.9493e-05 2024-04-19T01:04:50.469 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.469 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.469 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:50.469 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.592770+0000", 2024-04-19T01:04:50.469 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.025917557000000001 2024-04-19T01:04:50.469 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.469 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.469 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:50.469 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.592886+0000", 2024-04-19T01:04:50.469 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000115578 2024-04-19T01:04:50.469 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.469 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:50.469 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.470 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.470 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.470 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.5s6 74/31 ECSubWrite(tid=90, reqid=client.4604.0:545, at_version=74'71, trim_to=0'0, roll_forward_to=72'65))", 2024-04-19T01:04:50.470 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:44.574719+0000", 2024-04-19T01:04:50.470 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 5.8030617979999999, 2024-04-19T01:04:50.470 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.018279548999999999, 2024-04-19T01:04:50.470 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:50.470 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:50.470 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:50.470 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:50.470 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.470 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:50.470 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.574719+0000", 2024-04-19T01:04:50.471 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.471 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.471 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.471 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:50.471 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.574719+0000", 2024-04-19T01:04:50.471 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.471 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.471 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.472 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:50.472 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.574721+0000", 2024-04-19T01:04:50.472 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.007e-06 2024-04-19T01:04:50.472 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.472 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.472 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:50.472 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.574963+0000", 2024-04-19T01:04:50.472 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000242451 2024-04-19T01:04:50.472 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.473 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.473 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:50.473 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.574997+0000", 2024-04-19T01:04:50.473 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.3433000000000003e-05 2024-04-19T01:04:50.473 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.473 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.473 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:50.473 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.575022+0000", 2024-04-19T01:04:50.473 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5599e-05 2024-04-19T01:04:50.473 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.473 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.473 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:50.473 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.584812+0000", 2024-04-19T01:04:50.473 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0097900939999999992 2024-04-19T01:04:50.474 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.474 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.474 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:50.474 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.584842+0000", 2024-04-19T01:04:50.474 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.9675999999999998e-05 2024-04-19T01:04:50.474 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.474 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.474 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:50.474 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.592931+0000", 2024-04-19T01:04:50.474 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0080894599999999997 2024-04-19T01:04:50.475 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.475 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.475 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:50.475 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.592998+0000", 2024-04-19T01:04:50.475 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.6828999999999993e-05 2024-04-19T01:04:50.475 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.475 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:50.475 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.475 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.475 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:50.475 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:50.477 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-19T01:04:50.477 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:04:50.474+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:04:50.578 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:04:50.578+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:04:50.620 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.13 to 1 (10000) 2024-04-19T01:04:50.621 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2024-04-19T01:04:50.626 INFO:tasks.rados.rados.0.smithi155.stdout:245: done (5 left) 2024-04-19T01:04:50.626 INFO:tasks.rados.rados.0.smithi155.stdout:246: done (4 left) 2024-04-19T01:04:50.626 INFO:tasks.rados.rados.0.smithi155.stdout:247: done (3 left) 2024-04-19T01:04:50.627 INFO:tasks.rados.rados.0.smithi155.stdout:248: done (2 left) 2024-04-19T01:04:50.627 INFO:tasks.rados.rados.0.smithi155.stdout:249: done (1 left) 2024-04-19T01:04:50.627 INFO:tasks.rados.rados.0.smithi155.stdout:250: done (0 left) 2024-04-19T01:04:50.627 INFO:tasks.rados.rados.0.smithi155.stdout:251: delete oid 12 current snap is 26 2024-04-19T01:04:50.637 INFO:tasks.rados.rados.0.smithi155.stdout:251: done (0 left) 2024-04-19T01:04:50.637 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 23 current snap is 26 2024-04-19T01:04:50.637 INFO:tasks.rados.rados.0.smithi155.stdout:252: seq_num 111 ranges {262144=704512,966656=655360,1622016=507904} 2024-04-19T01:04:50.649 INFO:tasks.rados.rados.0.smithi155.stdout:252: writing smithi15528762-23 from 262144 to 966656 tid 1 2024-04-19T01:04:50.651 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:50.651 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [], 2024-04-19T01:04:50.651 INFO:teuthology.orchestra.run.smithi110.stdout: "num_ops": 0 2024-04-19T01:04:50.651 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:50.653 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 14 1 2024-04-19T01:04:50.659 INFO:tasks.rados.rados.0.smithi155.stdout:252: writing smithi15528762-23 from 966656 to 1622016 tid 2 2024-04-19T01:04:50.666 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-19T01:04:50.667 INFO:tasks.rados.rados.0.smithi155.stdout:252: writing smithi15528762-23 from 1622016 to 2129920 tid 3 2024-04-19T01:04:50.668 INFO:tasks.rados.rados.0.smithi155.stdout:253: snap_create 2024-04-19T01:04:50.679 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:04:50.679+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:04:50.685 INFO:tasks.rados.rados.0.smithi155.stdout:252: finishing write tid 1 to smithi15528762-23 2024-04-19T01:04:50.697 INFO:tasks.rados.rados.0.smithi155.stdout:252: finishing write tid 2 to smithi15528762-23 2024-04-19T01:04:50.698 INFO:tasks.rados.rados.0.smithi155.stdout:252: finishing write tid 3 to smithi15528762-23 2024-04-19T01:04:50.699 INFO:tasks.rados.rados.0.smithi155.stdout:252: finishing write tid 4 to smithi15528762-23 2024-04-19T01:04:50.707 INFO:tasks.rados.rados.0.smithi155.stdout:252: finishing write tid 5 to smithi15528762-23 2024-04-19T01:04:50.707 INFO:tasks.rados.rados.0.smithi155.stdout:252: oid 23 updating version 0 to 54 2024-04-19T01:04:50.707 INFO:tasks.rados.rados.0.smithi155.stdout:252: oid 23 version 54 is already newer than 51 2024-04-19T01:04:50.707 INFO:tasks.rados.rados.0.smithi155.stdout:252: oid 23 version 54 is already newer than 52 2024-04-19T01:04:50.707 INFO:tasks.rados.rados.0.smithi155.stdout:252: oid 23 version 54 is already newer than 53 2024-04-19T01:04:50.707 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 23 v 54 (ObjNum 111 snap 26 seq_num 111) dirty exists 2024-04-19T01:04:50.707 INFO:tasks.rados.rados.0.smithi155.stdout:252: left oid 23 (ObjNum 111 snap 26 seq_num 111) 2024-04-19T01:04:50.780 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:50.779+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:50.786 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:50.786 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [], 2024-04-19T01:04:50.787 INFO:teuthology.orchestra.run.smithi110.stdout: "complaint_time": 30, 2024-04-19T01:04:50.787 INFO:teuthology.orchestra.run.smithi110.stdout: "num_blocked_ops": 0 2024-04-19T01:04:50.787 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:50.801 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-19T01:04:50.883 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:04:50.882+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:04:50.925 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:50.925 INFO:teuthology.orchestra.run.smithi110.stdout: "size": 20, 2024-04-19T01:04:50.925 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 600, 2024-04-19T01:04:50.925 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [ 2024-04-19T01:04:50.925 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.925 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 39/31 ECSubWrite(tid=32, reqid=client.4604.0:181, at_version=39'16, trim_to=0'0, roll_forward_to=38'14))", 2024-04-19T01:04:50.925 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:09.820805+0000", 2024-04-19T01:04:50.925 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 41.104647802999999, 2024-04-19T01:04:50.925 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.013829808000000001, 2024-04-19T01:04:50.926 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:50.926 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:50.926 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:50.926 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:50.926 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.926 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:50.926 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.820805+0000", 2024-04-19T01:04:50.926 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.926 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.926 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.926 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:50.927 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.820805+0000", 2024-04-19T01:04:50.927 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.927 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.927 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.927 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:50.927 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.820806+0000", 2024-04-19T01:04:50.927 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.2699999999999999e-07 2024-04-19T01:04:50.927 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.927 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.927 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:50.927 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.821061+0000", 2024-04-19T01:04:50.927 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000255133 2024-04-19T01:04:50.927 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.927 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.928 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:50.928 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.821256+0000", 2024-04-19T01:04:50.928 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00019562400000000001 2024-04-19T01:04:50.928 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.928 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.928 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:50.928 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.821283+0000", 2024-04-19T01:04:50.928 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6712000000000001e-05 2024-04-19T01:04:50.928 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.928 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.928 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:50.928 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.821504+0000", 2024-04-19T01:04:50.928 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00022120199999999999 2024-04-19T01:04:50.928 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.929 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.929 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:50.929 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.821557+0000", 2024-04-19T01:04:50.929 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.2534000000000002e-05 2024-04-19T01:04:50.929 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.929 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.929 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:50.929 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.834585+0000", 2024-04-19T01:04:50.929 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.013027617999999999 2024-04-19T01:04:50.929 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.929 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.929 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:50.929 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.834635+0000", 2024-04-19T01:04:50.929 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.0257999999999999e-05 2024-04-19T01:04:50.930 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.930 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:50.930 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.930 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.930 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.930 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 43/31 ECSubWrite(tid=70, reqid=client.4604.0:235, at_version=43'43, trim_to=0'0, roll_forward_to=41'37))", 2024-04-19T01:04:50.930 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.870038+0000", 2024-04-19T01:04:50.930 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 37.055414441000003, 2024-04-19T01:04:50.930 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.020219719000000001, 2024-04-19T01:04:50.930 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:50.930 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:50.930 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:50.930 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:50.931 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.931 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:50.931 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.870038+0000", 2024-04-19T01:04:50.931 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.931 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.931 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.931 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:50.931 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.870038+0000", 2024-04-19T01:04:50.931 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.931 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.931 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.931 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:50.932 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.870040+0000", 2024-04-19T01:04:50.932 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.81e-06 2024-04-19T01:04:50.932 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.932 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.932 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:50.932 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.870453+0000", 2024-04-19T01:04:50.932 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000412965 2024-04-19T01:04:50.932 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.932 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.932 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:50.932 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.870494+0000", 2024-04-19T01:04:50.932 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.0938e-05 2024-04-19T01:04:50.933 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.933 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.933 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:50.933 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.870510+0000", 2024-04-19T01:04:50.933 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5458999999999999e-05 2024-04-19T01:04:50.933 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.933 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.933 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:50.933 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.870799+0000", 2024-04-19T01:04:50.933 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00028974699999999998 2024-04-19T01:04:50.933 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.933 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.933 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:50.934 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.870832+0000", 2024-04-19T01:04:50.934 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.2434999999999997e-05 2024-04-19T01:04:50.934 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.934 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.934 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:50.934 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.890185+0000", 2024-04-19T01:04:50.934 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.019353467999999999 2024-04-19T01:04:50.934 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.934 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.934 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:50.934 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.890258+0000", 2024-04-19T01:04:50.934 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.2897000000000004e-05 2024-04-19T01:04:50.934 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.935 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:50.935 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.935 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.935 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.935 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.ds2 43/31 ECSubWrite(tid=38, reqid=client.4604.0:253, at_version=43'14, trim_to=0'0, roll_forward_to=43'12))", 2024-04-19T01:04:50.935 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.992478+0000", 2024-04-19T01:04:50.935 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 36.932975235999997, 2024-04-19T01:04:50.935 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.025708154, 2024-04-19T01:04:50.935 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:50.935 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:50.935 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:50.935 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:50.935 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.936 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:50.936 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.992478+0000", 2024-04-19T01:04:50.936 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.936 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.936 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.936 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:50.936 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.992478+0000", 2024-04-19T01:04:50.936 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.936 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.936 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.936 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:50.936 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.992480+0000", 2024-04-19T01:04:50.936 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9870000000000002e-06 2024-04-19T01:04:50.937 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.937 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.937 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:50.937 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.992860+0000", 2024-04-19T01:04:50.937 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00038019600000000002 2024-04-19T01:04:50.937 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.937 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.937 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:50.937 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.992891+0000", 2024-04-19T01:04:50.937 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.0982000000000003e-05 2024-04-19T01:04:50.937 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.937 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.937 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:50.938 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.992906+0000", 2024-04-19T01:04:50.938 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5503e-05 2024-04-19T01:04:50.938 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.938 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.938 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:50.938 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.993064+0000", 2024-04-19T01:04:50.938 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015782900000000001 2024-04-19T01:04:50.938 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.938 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.938 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:50.938 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.993111+0000", 2024-04-19T01:04:50.939 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.7138000000000002e-05 2024-04-19T01:04:50.939 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.939 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.939 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:50.939 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.018092+0000", 2024-04-19T01:04:50.939 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.024980822999999999 2024-04-19T01:04:50.939 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.939 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.939 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:50.939 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.018186+0000", 2024-04-19T01:04:50.939 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.3696000000000006e-05 2024-04-19T01:04:50.939 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.939 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:50.939 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.940 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.940 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.940 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.ds2 43/31 ECSubWrite(tid=39, reqid=client.4604.0:254, at_version=43'15, trim_to=0'0, roll_forward_to=43'12))", 2024-04-19T01:04:50.940 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.996882+0000", 2024-04-19T01:04:50.940 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 36.928571116999997, 2024-04-19T01:04:50.940 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.032356653999999999, 2024-04-19T01:04:50.940 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:50.940 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:50.940 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:50.940 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:50.940 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.940 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:50.940 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.996882+0000", 2024-04-19T01:04:50.940 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.941 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.941 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.941 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:50.941 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.996882+0000", 2024-04-19T01:04:50.941 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.941 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.941 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.941 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:50.941 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.996884+0000", 2024-04-19T01:04:50.941 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2029999999999999e-06 2024-04-19T01:04:50.941 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.941 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.941 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:50.941 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.997205+0000", 2024-04-19T01:04:50.941 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00032076899999999999 2024-04-19T01:04:50.942 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.942 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.942 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:50.942 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.997249+0000", 2024-04-19T01:04:50.942 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.4654000000000003e-05 2024-04-19T01:04:50.942 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.942 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.942 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:50.942 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.997273+0000", 2024-04-19T01:04:50.942 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3589000000000001e-05 2024-04-19T01:04:50.942 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.942 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.943 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:50.943 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.007691+0000", 2024-04-19T01:04:50.943 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.010418495999999999 2024-04-19T01:04:50.943 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.943 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.943 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:50.943 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.007717+0000", 2024-04-19T01:04:50.943 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5876000000000001e-05 2024-04-19T01:04:50.943 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.943 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.943 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:50.943 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.029151+0000", 2024-04-19T01:04:50.944 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.021433651000000001 2024-04-19T01:04:50.944 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.944 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.944 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:50.944 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.029238+0000", 2024-04-19T01:04:50.944 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.7416000000000005e-05 2024-04-19T01:04:50.944 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.944 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:50.944 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.944 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.944 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.944 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.ds2 43/31 ECSubWrite(tid=40, reqid=client.4604.0:255, at_version=43'16, trim_to=0'0, roll_forward_to=43'12))", 2024-04-19T01:04:50.944 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.002736+0000", 2024-04-19T01:04:50.945 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 36.922716878000003, 2024-04-19T01:04:50.945 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.026570379000000002, 2024-04-19T01:04:50.945 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:50.945 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:50.945 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:50.945 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:50.945 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.945 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:50.945 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.002736+0000", 2024-04-19T01:04:50.945 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.945 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.945 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.945 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:50.945 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.002736+0000", 2024-04-19T01:04:50.945 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.946 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.946 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.946 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:50.946 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.002738+0000", 2024-04-19T01:04:50.946 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0899999999999999e-06 2024-04-19T01:04:50.946 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.946 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.946 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:50.946 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.002782+0000", 2024-04-19T01:04:50.946 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.3611e-05 2024-04-19T01:04:50.946 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.946 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.946 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:50.947 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.002823+0000", 2024-04-19T01:04:50.947 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.0954999999999997e-05 2024-04-19T01:04:50.947 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.947 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.947 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:50.947 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.002845+0000", 2024-04-19T01:04:50.947 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2775000000000001e-05 2024-04-19T01:04:50.947 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.947 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.947 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:50.947 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.019866+0000", 2024-04-19T01:04:50.947 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.017021162999999999 2024-04-19T01:04:50.947 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.947 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.947 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:50.948 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.019893+0000", 2024-04-19T01:04:50.948 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6176000000000001e-05 2024-04-19T01:04:50.948 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.948 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.948 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:50.948 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.029258+0000", 2024-04-19T01:04:50.948 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0093652990000000005 2024-04-19T01:04:50.948 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.948 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.948 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:50.948 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.029306+0000", 2024-04-19T01:04:50.948 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.8310000000000003e-05 2024-04-19T01:04:50.948 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.948 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:50.948 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.949 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.949 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.949 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.ds2 43/31 ECSubWrite(tid=42, reqid=client.4604.0:271, at_version=43'18, trim_to=0'0, roll_forward_to=43'14))", 2024-04-19T01:04:50.949 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.019519+0000", 2024-04-19T01:04:50.949 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 36.905934172000002, 2024-04-19T01:04:50.949 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.010349381, 2024-04-19T01:04:50.949 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:50.949 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:50.949 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:50.949 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:50.949 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.949 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:50.949 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.019519+0000", 2024-04-19T01:04:50.951 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.951 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.951 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.951 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:50.951 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.019519+0000", 2024-04-19T01:04:50.951 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.951 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.951 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.951 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:50.951 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.019520+0000", 2024-04-19T01:04:50.951 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.424e-06 2024-04-19T01:04:50.952 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.952 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.952 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:50.952 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.019678+0000", 2024-04-19T01:04:50.952 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015818599999999999 2024-04-19T01:04:50.952 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.952 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.952 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:50.952 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.019702+0000", 2024-04-19T01:04:50.952 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3561999999999999e-05 2024-04-19T01:04:50.952 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.952 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.952 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:50.953 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.019717+0000", 2024-04-19T01:04:50.953 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.4793e-05 2024-04-19T01:04:50.953 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.953 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.953 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:50.953 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.021792+0000", 2024-04-19T01:04:50.953 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0020751739999999999 2024-04-19T01:04:50.953 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.953 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.953 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:50.953 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.021855+0000", 2024-04-19T01:04:50.953 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.3436999999999996e-05 2024-04-19T01:04:50.954 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.954 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.954 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:50.954 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.029768+0000", 2024-04-19T01:04:50.954 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0079128429999999993 2024-04-19T01:04:50.954 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.954 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.954 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:50.954 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.029868+0000", 2024-04-19T01:04:50.954 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.9962000000000003e-05 2024-04-19T01:04:50.954 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.954 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:50.954 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.955 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.955 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.955 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 43/31 ECSubWrite(tid=76, reqid=client.4604.0:267, at_version=43'46, trim_to=0'0, roll_forward_to=43'45))", 2024-04-19T01:04:50.955 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.020527+0000", 2024-04-19T01:04:50.955 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 36.90492587, 2024-04-19T01:04:50.955 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.016212788999999998, 2024-04-19T01:04:50.955 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:50.955 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:50.955 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:50.955 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:50.955 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.955 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:50.955 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.020527+0000", 2024-04-19T01:04:50.956 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.956 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.956 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.956 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:50.956 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.020527+0000", 2024-04-19T01:04:50.957 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.957 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.957 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.957 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:50.957 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.020529+0000", 2024-04-19T01:04:50.957 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.046e-06 2024-04-19T01:04:50.957 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.957 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.958 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:50.958 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.020873+0000", 2024-04-19T01:04:50.958 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00034371800000000002 2024-04-19T01:04:50.958 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.958 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.958 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:50.958 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.020902+0000", 2024-04-19T01:04:50.958 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.8903000000000001e-05 2024-04-19T01:04:50.958 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.958 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.958 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:50.958 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.020924+0000", 2024-04-19T01:04:50.958 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2892e-05 2024-04-19T01:04:50.958 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.959 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.959 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:50.959 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.021071+0000", 2024-04-19T01:04:50.959 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00014612300000000001 2024-04-19T01:04:50.959 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.959 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.959 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:50.959 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.021113+0000", 2024-04-19T01:04:50.959 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.2573e-05 2024-04-19T01:04:50.959 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.959 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.959 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:50.959 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.036641+0000", 2024-04-19T01:04:50.960 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.015527501000000001 2024-04-19T01:04:50.960 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.960 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.960 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:50.960 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.036740+0000", 2024-04-19T01:04:50.960 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.9032999999999999e-05 2024-04-19T01:04:50.960 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.960 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:50.960 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.960 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.960 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.960 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.1s4 44/31 ECSubWrite(tid=15, reqid=client.4604.0:289, at_version=44'8, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:04:50.960 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.875457+0000", 2024-04-19T01:04:50.960 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 36.049995553000002, 2024-04-19T01:04:50.960 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.036567641999999997, 2024-04-19T01:04:50.961 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:50.961 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:50.961 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:50.961 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:50.961 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.961 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:50.961 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875457+0000", 2024-04-19T01:04:50.961 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.961 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.961 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.961 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:50.961 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875457+0000", 2024-04-19T01:04:50.961 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.961 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.962 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.962 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:50.962 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875458+0000", 2024-04-19T01:04:50.962 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.8400000000000004e-07 2024-04-19T01:04:50.962 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.962 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.962 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:50.962 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875519+0000", 2024-04-19T01:04:50.964 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.1377000000000005e-05 2024-04-19T01:04:50.964 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.964 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.964 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:50.965 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875588+0000", 2024-04-19T01:04:50.965 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.9099999999999999e-05 2024-04-19T01:04:50.965 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.965 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.965 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:50.965 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875615+0000", 2024-04-19T01:04:50.965 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6902000000000001e-05 2024-04-19T01:04:50.965 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.965 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.965 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:50.966 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875846+0000", 2024-04-19T01:04:50.966 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00023076699999999999 2024-04-19T01:04:50.966 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.966 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.966 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:50.966 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875874+0000", 2024-04-19T01:04:50.966 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.7725e-05 2024-04-19T01:04:50.966 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.966 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.966 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:50.966 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.911826+0000", 2024-04-19T01:04:50.966 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.035952348000000002 2024-04-19T01:04:50.966 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.967 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.967 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:50.967 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.912025+0000", 2024-04-19T01:04:50.967 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000198739 2024-04-19T01:04:50.967 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.967 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:50.967 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.967 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.967 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.967 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 44/31 ECSubWrite(tid=82, reqid=client.4604.0:280, at_version=44'50, trim_to=0'0, roll_forward_to=43'48))", 2024-04-19T01:04:50.967 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.880504+0000", 2024-04-19T01:04:50.968 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 36.044948458999997, 2024-04-19T01:04:50.968 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.041548157000000002, 2024-04-19T01:04:50.968 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:50.968 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:50.968 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:50.968 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:50.968 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.968 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:50.968 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.880504+0000", 2024-04-19T01:04:50.968 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.968 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.968 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.968 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:50.968 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.880504+0000", 2024-04-19T01:04:50.968 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.969 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.969 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.969 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:50.969 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.880506+0000", 2024-04-19T01:04:50.969 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9439999999999999e-06 2024-04-19T01:04:50.974 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.974 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.974 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:50.974 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.880844+0000", 2024-04-19T01:04:50.974 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00033782799999999998 2024-04-19T01:04:50.974 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.974 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.974 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:50.975 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.880877+0000", 2024-04-19T01:04:50.975 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.2524999999999998e-05 2024-04-19T01:04:50.975 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.975 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.975 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:50.975 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.880899+0000", 2024-04-19T01:04:50.975 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2656000000000001e-05 2024-04-19T01:04:50.975 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.975 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.975 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:50.975 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.881129+0000", 2024-04-19T01:04:50.975 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00022929600000000001 2024-04-19T01:04:50.976 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.976 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.976 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:50.976 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.881155+0000", 2024-04-19T01:04:50.976 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6729000000000001e-05 2024-04-19T01:04:50.976 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.976 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.976 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:50.977 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.921810+0000", 2024-04-19T01:04:50.977 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.040654807000000001 2024-04-19T01:04:50.977 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.977 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.977 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:50.977 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.922052+0000", 2024-04-19T01:04:50.977 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000242372 2024-04-19T01:04:50.977 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.977 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:50.977 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.978 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.978 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.978 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.es5 44/31 ECSubWrite(tid=83, reqid=osd.13.0:81, at_version=0'0, trim_to=0'0, roll_forward_to=44'14))", 2024-04-19T01:04:50.978 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.887116+0000", 2024-04-19T01:04:50.978 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 36.038337097000003, 2024-04-19T01:04:50.978 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0081527319999999993, 2024-04-19T01:04:50.978 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:50.979 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:50.979 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:50.979 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:50.979 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.979 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:50.979 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.887116+0000", 2024-04-19T01:04:50.979 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.979 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.979 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.979 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:50.979 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.887116+0000", 2024-04-19T01:04:50.979 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.979 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.980 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.980 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:50.980 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.887118+0000", 2024-04-19T01:04:50.980 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1560000000000002e-06 2024-04-19T01:04:50.980 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.980 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.980 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:50.980 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.887144+0000", 2024-04-19T01:04:50.980 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6495999999999998e-05 2024-04-19T01:04:50.980 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.980 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.980 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:50.981 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.887178+0000", 2024-04-19T01:04:50.981 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.3241999999999998e-05 2024-04-19T01:04:50.981 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.981 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.981 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:50.981 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.887200+0000", 2024-04-19T01:04:50.982 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2665000000000001e-05 2024-04-19T01:04:50.982 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.982 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.982 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:50.982 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.887400+0000", 2024-04-19T01:04:50.982 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000199278 2024-04-19T01:04:50.982 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.982 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.982 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:50.982 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.887424+0000", 2024-04-19T01:04:50.982 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4545999999999998e-05 2024-04-19T01:04:50.984 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.984 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.985 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:50.985 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.895177+0000", 2024-04-19T01:04:50.985 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.007752643 2024-04-19T01:04:50.985 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.985 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.985 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:50.985 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.895268+0000", 2024-04-19T01:04:50.985 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.1706000000000006e-05 2024-04-19T01:04:50.985 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.985 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:50.985 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.985 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.985 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.985 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 44/31 ECSubWrite(tid=84, reqid=client.4604.0:281, at_version=44'51, trim_to=0'0, roll_forward_to=43'48))", 2024-04-19T01:04:50.986 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.906686+0000", 2024-04-19T01:04:50.986 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 36.018767062000002, 2024-04-19T01:04:50.986 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.018013254999999999, 2024-04-19T01:04:50.986 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:50.986 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:50.986 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:50.986 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:50.986 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.986 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:50.986 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.906686+0000", 2024-04-19T01:04:50.986 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.986 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.986 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.986 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:50.987 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.906686+0000", 2024-04-19T01:04:50.987 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.987 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.987 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.987 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:50.987 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.906688+0000", 2024-04-19T01:04:50.987 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1569999999999998e-06 2024-04-19T01:04:50.987 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.987 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.987 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:50.987 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.906789+0000", 2024-04-19T01:04:50.987 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000101488 2024-04-19T01:04:50.987 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.987 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.988 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:50.988 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.906831+0000", 2024-04-19T01:04:50.988 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.1488e-05 2024-04-19T01:04:50.988 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.988 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.988 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:50.988 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.906855+0000", 2024-04-19T01:04:50.988 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4269000000000001e-05 2024-04-19T01:04:50.988 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.988 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.988 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:50.988 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.913161+0000", 2024-04-19T01:04:50.988 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0063058070000000001 2024-04-19T01:04:50.988 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.988 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.989 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:50.989 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.913193+0000", 2024-04-19T01:04:50.989 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.2007e-05 2024-04-19T01:04:50.989 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.989 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.989 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:50.989 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.924624+0000", 2024-04-19T01:04:50.989 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.011430879999999999 2024-04-19T01:04:50.989 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.989 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.991 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:50.991 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.924699+0000", 2024-04-19T01:04:50.991 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.5159000000000003e-05 2024-04-19T01:04:50.991 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.991 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:50.991 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.991 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.991 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.991 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 44/31 ECSubWrite(tid=85, reqid=client.4604.0:282, at_version=44'52, trim_to=0'0, roll_forward_to=43'48))", 2024-04-19T01:04:50.991 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.916938+0000", 2024-04-19T01:04:50.992 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 36.008514787000003, 2024-04-19T01:04:50.992 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0078204020000000006, 2024-04-19T01:04:50.992 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:50.992 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:50.992 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:50.992 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:50.992 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.992 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:50.992 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.916938+0000", 2024-04-19T01:04:50.992 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.992 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.992 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.992 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:50.992 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.916938+0000", 2024-04-19T01:04:50.992 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.993 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.993 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.993 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:50.993 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.916940+0000", 2024-04-19T01:04:50.993 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0159999999999998e-06 2024-04-19T01:04:50.993 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.993 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.993 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:50.993 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.916992+0000", 2024-04-19T01:04:50.993 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.2494999999999998e-05 2024-04-19T01:04:50.993 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.993 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.994 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:50.994 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.917033+0000", 2024-04-19T01:04:50.994 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.0908000000000002e-05 2024-04-19T01:04:50.994 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.994 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.994 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:50.994 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.917057+0000", 2024-04-19T01:04:50.994 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4042000000000001e-05 2024-04-19T01:04:50.994 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.994 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.994 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:50.995 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.922174+0000", 2024-04-19T01:04:50.995 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.005116308 2024-04-19T01:04:50.995 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.995 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.995 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:50.995 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.922203+0000", 2024-04-19T01:04:50.995 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.9615e-05 2024-04-19T01:04:50.995 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.995 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.995 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:50.995 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.924709+0000", 2024-04-19T01:04:50.995 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0025060299999999998 2024-04-19T01:04:50.995 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.995 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.995 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:50.996 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.924758+0000", 2024-04-19T01:04:50.996 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.8987999999999999e-05 2024-04-19T01:04:50.996 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.996 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:50.996 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:50.996 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.996 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.996 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.0s1 44/31 ECSubWrite(tid=25, reqid=client.4604.0:284, at_version=44'22, trim_to=0'0, roll_forward_to=44'20))", 2024-04-19T01:04:50.996 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.922536+0000", 2024-04-19T01:04:50.996 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 36.002916784, 2024-04-19T01:04:50.996 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.029007871000000001, 2024-04-19T01:04:50.996 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:50.996 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:50.996 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:50.997 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:50.997 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.997 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:50.997 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.922536+0000", 2024-04-19T01:04:50.997 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.997 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.997 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.997 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:50.997 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.922536+0000", 2024-04-19T01:04:50.997 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:50.997 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.997 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.997 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:50.997 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.922538+0000", 2024-04-19T01:04:50.997 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0829999999999998e-06 2024-04-19T01:04:50.998 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.998 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.998 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:50.998 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.923282+0000", 2024-04-19T01:04:50.998 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000743554 2024-04-19T01:04:50.998 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.998 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.998 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:50.998 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.923311+0000", 2024-04-19T01:04:50.998 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.9147999999999999e-05 2024-04-19T01:04:50.998 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.998 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.998 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:50.999 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.923337+0000", 2024-04-19T01:04:50.999 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6648999999999998e-05 2024-04-19T01:04:50.999 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.999 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.999 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:50.999 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.923466+0000", 2024-04-19T01:04:50.999 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00012821100000000001 2024-04-19T01:04:50.999 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.999 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.999 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:50.999 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.923478+0000", 2024-04-19T01:04:50.999 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.2291e-05 2024-04-19T01:04:50.999 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:50.999 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:50.999 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:51.000 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.951479+0000", 2024-04-19T01:04:51.002 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.02800066 2024-04-19T01:04:51.002 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.002 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.002 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:51.002 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.951544+0000", 2024-04-19T01:04:51.002 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.5275000000000001e-05 2024-04-19T01:04:51.002 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.002 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:51.002 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.002 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.002 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.002 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.ds2 47/31 ECSubWrite(tid=61, reqid=client.4604.0:330, at_version=47'32, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:04:51.002 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:17.677293+0000", 2024-04-19T01:04:51.003 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 33.248159655000002, 2024-04-19T01:04:51.003 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0087219389999999997, 2024-04-19T01:04:51.003 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:51.003 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:51.003 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:51.003 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:51.003 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.003 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:51.003 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.677293+0000", 2024-04-19T01:04:51.003 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.003 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.003 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.003 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:51.003 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.677293+0000", 2024-04-19T01:04:51.003 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.004 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.004 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.004 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:51.004 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.677294+0000", 2024-04-19T01:04:51.004 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.1289999999999999e-06 2024-04-19T01:04:51.004 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.004 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.004 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:51.004 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.677592+0000", 2024-04-19T01:04:51.004 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00029753199999999998 2024-04-19T01:04:51.004 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.004 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.004 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:51.004 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.677607+0000", 2024-04-19T01:04:51.004 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5367999999999999e-05 2024-04-19T01:04:51.005 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.005 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.005 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:51.005 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.677618+0000", 2024-04-19T01:04:51.005 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.1201e-05 2024-04-19T01:04:51.005 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.005 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.005 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:51.005 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.677706+0000", 2024-04-19T01:04:51.005 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.7769000000000006e-05 2024-04-19T01:04:51.005 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.005 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.005 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:51.005 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.677713+0000", 2024-04-19T01:04:51.006 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.8399999999999997e-06 2024-04-19T01:04:51.006 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.006 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.006 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:51.006 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.685970+0000", 2024-04-19T01:04:51.006 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0082570330000000004 2024-04-19T01:04:51.006 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.006 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.006 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:51.006 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.686015+0000", 2024-04-19T01:04:51.006 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.5067000000000001e-05 2024-04-19T01:04:51.006 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.006 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:51.006 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.007 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.007 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.007 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 48/31 ECSubWrite(tid=115, reqid=osd.13.0:112, at_version=0'0, trim_to=0'0, roll_forward_to=48'69))", 2024-04-19T01:04:51.007 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.539886+0000", 2024-04-19T01:04:51.007 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 32.385566490999999, 2024-04-19T01:04:51.007 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.011017575, 2024-04-19T01:04:51.007 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:51.007 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:51.007 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:51.007 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:51.007 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.007 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:51.007 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.539886+0000", 2024-04-19T01:04:51.007 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.008 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.008 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.008 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:51.008 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.539886+0000", 2024-04-19T01:04:51.008 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.008 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.008 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.008 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:51.008 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.539888+0000", 2024-04-19T01:04:51.008 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.24e-06 2024-04-19T01:04:51.008 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.008 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.008 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:51.008 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.539911+0000", 2024-04-19T01:04:51.008 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3785999999999999e-05 2024-04-19T01:04:51.009 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.009 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.009 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:51.009 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.539937+0000", 2024-04-19T01:04:51.009 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5252000000000001e-05 2024-04-19T01:04:51.009 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.009 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.009 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:51.009 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.539958+0000", 2024-04-19T01:04:51.009 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1223e-05 2024-04-19T01:04:51.009 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.009 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.009 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:51.009 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.540206+0000", 2024-04-19T01:04:51.009 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00024783599999999998 2024-04-19T01:04:51.010 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.010 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.010 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:51.010 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.540231+0000", 2024-04-19T01:04:51.010 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5318999999999999e-05 2024-04-19T01:04:51.010 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.010 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.010 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:51.010 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.550811+0000", 2024-04-19T01:04:51.010 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.010580042 2024-04-19T01:04:51.010 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.012 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.012 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:51.012 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.550904+0000", 2024-04-19T01:04:51.012 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.2876999999999999e-05 2024-04-19T01:04:51.013 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.013 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:51.013 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.013 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.013 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.013 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 63/31 ECSubWrite(tid=152, reqid=client.4604.0:465, at_version=63'100, trim_to=0'0, roll_forward_to=62'98))", 2024-04-19T01:04:51.013 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:33.566417+0000", 2024-04-19T01:04:51.013 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 17.359035644999999, 2024-04-19T01:04:51.013 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.010107771, 2024-04-19T01:04:51.013 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:51.013 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:51.013 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:51.013 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:51.013 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.014 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:51.014 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.566417+0000", 2024-04-19T01:04:51.014 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.014 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.014 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.014 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:51.014 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.566417+0000", 2024-04-19T01:04:51.014 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.014 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.014 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.014 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:51.014 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.566417+0000", 2024-04-19T01:04:51.014 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.3000000000000002e-07 2024-04-19T01:04:51.014 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.015 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.015 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:51.015 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.566642+0000", 2024-04-19T01:04:51.015 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00022461900000000001 2024-04-19T01:04:51.015 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.015 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.015 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:51.015 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.566656+0000", 2024-04-19T01:04:51.015 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.3740999999999999e-05 2024-04-19T01:04:51.015 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.015 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.015 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:51.015 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.566669+0000", 2024-04-19T01:04:51.015 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.2806000000000001e-05 2024-04-19T01:04:51.015 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.016 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.016 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:51.016 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.566764+0000", 2024-04-19T01:04:51.016 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.5223999999999995e-05 2024-04-19T01:04:51.016 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.016 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.016 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:51.016 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.566779+0000", 2024-04-19T01:04:51.016 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5081e-05 2024-04-19T01:04:51.016 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.016 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.016 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:51.016 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.576481+0000", 2024-04-19T01:04:51.016 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0097020909999999995 2024-04-19T01:04:51.017 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.017 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.017 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:51.017 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.576525+0000", 2024-04-19T01:04:51.017 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.3878999999999998e-05 2024-04-19T01:04:51.017 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.017 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:51.017 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.017 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.017 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.017 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 74/31 ECSubWrite(tid=174, reqid=client.4604.0:548, at_version=74'110, trim_to=0'0, roll_forward_to=70'108))", 2024-04-19T01:04:51.017 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:44.595181+0000", 2024-04-19T01:04:51.017 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 6.3302718420000001, 2024-04-19T01:04:51.017 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.023202622999999999, 2024-04-19T01:04:51.018 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:51.018 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:51.018 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:51.018 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:51.018 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.018 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:51.018 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.595181+0000", 2024-04-19T01:04:51.018 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.018 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.018 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.018 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:51.018 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.595181+0000", 2024-04-19T01:04:51.018 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.018 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.019 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.019 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:51.019 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.595182+0000", 2024-04-19T01:04:51.019 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.13e-06 2024-04-19T01:04:51.019 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.019 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.019 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:51.019 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.595558+0000", 2024-04-19T01:04:51.019 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00037566500000000002 2024-04-19T01:04:51.019 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.019 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.019 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:51.019 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.595596+0000", 2024-04-19T01:04:51.019 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.7815999999999998e-05 2024-04-19T01:04:51.020 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.020 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.020 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:51.020 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.595624+0000", 2024-04-19T01:04:51.020 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.8906000000000001e-05 2024-04-19T01:04:51.020 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.020 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.020 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:51.020 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.595857+0000", 2024-04-19T01:04:51.020 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00023279099999999999 2024-04-19T01:04:51.020 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.020 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.020 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:51.020 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.595886+0000", 2024-04-19T01:04:51.023 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.8770000000000001e-05 2024-04-19T01:04:51.023 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.023 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.024 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:51.024 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.618262+0000", 2024-04-19T01:04:51.024 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.022375757999999999 2024-04-19T01:04:51.024 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.024 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.024 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:51.024 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.618384+0000", 2024-04-19T01:04:51.024 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000121787 2024-04-19T01:04:51.024 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.024 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:51.024 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.024 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.024 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.024 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 74/31 ECSubWrite(tid=175, reqid=client.4604.0:549, at_version=74'111, trim_to=0'0, roll_forward_to=70'108))", 2024-04-19T01:04:51.025 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:44.609147+0000", 2024-04-19T01:04:51.025 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 6.3163059529999996, 2024-04-19T01:04:51.025 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.010316297, 2024-04-19T01:04:51.025 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:51.025 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:51.025 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:51.025 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:51.025 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.025 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:51.025 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.609147+0000", 2024-04-19T01:04:51.025 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.025 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.025 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.025 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:51.025 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.609147+0000", 2024-04-19T01:04:51.026 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.026 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.026 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.026 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:51.026 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.609150+0000", 2024-04-19T01:04:51.026 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.7769999999999998e-06 2024-04-19T01:04:51.026 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.026 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.026 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:51.026 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.609271+0000", 2024-04-19T01:04:51.026 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000121848 2024-04-19T01:04:51.026 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.026 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.026 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:51.027 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.609307+0000", 2024-04-19T01:04:51.027 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.5695000000000003e-05 2024-04-19T01:04:51.027 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.027 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.027 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:51.027 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.609327+0000", 2024-04-19T01:04:51.027 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0360000000000002e-05 2024-04-19T01:04:51.027 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.027 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.027 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:51.027 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.609544+0000", 2024-04-19T01:04:51.027 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00021615800000000001 2024-04-19T01:04:51.027 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.027 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.027 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:51.028 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.609574+0000", 2024-04-19T01:04:51.028 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.0645999999999998e-05 2024-04-19T01:04:51.028 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.028 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.028 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:51.028 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.619399+0000", 2024-04-19T01:04:51.028 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0098242009999999994 2024-04-19T01:04:51.028 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.028 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.028 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:51.028 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.619463+0000", 2024-04-19T01:04:51.028 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.4611999999999997e-05 2024-04-19T01:04:51.028 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.028 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:51.028 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.029 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.029 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.029 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 74/31 ECSubWrite(tid=177, reqid=client.4604.0:550, at_version=74'112, trim_to=0'0, roll_forward_to=74'111))", 2024-04-19T01:04:51.029 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:44.623353+0000", 2024-04-19T01:04:51.029 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 6.3020994879999996, 2024-04-19T01:04:51.029 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.012116458, 2024-04-19T01:04:51.029 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:51.029 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:51.029 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:51.029 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:51.029 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.029 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:51.029 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.623353+0000", 2024-04-19T01:04:51.029 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.030 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.030 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.030 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:51.030 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.623353+0000", 2024-04-19T01:04:51.030 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.030 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.030 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.030 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:51.030 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.623355+0000", 2024-04-19T01:04:51.031 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.4330000000000001e-06 2024-04-19T01:04:51.031 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.031 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.031 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:51.031 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.623547+0000", 2024-04-19T01:04:51.031 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00019249600000000001 2024-04-19T01:04:51.031 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.031 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.031 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:51.031 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.623571+0000", 2024-04-19T01:04:51.031 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3448999999999999e-05 2024-04-19T01:04:51.032 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.032 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.032 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:51.032 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.623585+0000", 2024-04-19T01:04:51.032 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.414e-05 2024-04-19T01:04:51.032 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.032 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.032 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:51.032 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.623784+0000", 2024-04-19T01:04:51.032 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000199285 2024-04-19T01:04:51.032 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.032 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.032 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:51.033 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.623830+0000", 2024-04-19T01:04:51.033 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.6276000000000002e-05 2024-04-19T01:04:51.033 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.033 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.033 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:51.033 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.635402+0000", 2024-04-19T01:04:51.033 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.011571744 2024-04-19T01:04:51.033 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.033 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.034 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:51.034 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.635470+0000", 2024-04-19T01:04:51.034 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.7634999999999999e-05 2024-04-19T01:04:51.034 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.034 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:51.034 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.034 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.034 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.034 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.0s1 76/31 ECSubWrite(tid=66, reqid=client.4604.0:563, at_version=76'56, trim_to=0'0, roll_forward_to=67'54))", 2024-04-19T01:04:51.034 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:46.555265+0000", 2024-04-19T01:04:51.034 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 4.3701875509999999, 2024-04-19T01:04:51.034 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.014802898, 2024-04-19T01:04:51.035 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:51.035 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:51.035 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:51.035 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:51.035 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.035 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:51.035 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:46.555265+0000", 2024-04-19T01:04:51.035 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.035 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.035 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.035 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:51.035 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:46.555265+0000", 2024-04-19T01:04:51.035 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.035 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.035 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.036 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:51.036 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:46.555266+0000", 2024-04-19T01:04:51.036 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.4700000000000001e-07 2024-04-19T01:04:51.036 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.036 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.036 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:51.036 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:46.555593+0000", 2024-04-19T01:04:51.036 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00032677999999999998 2024-04-19T01:04:51.036 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.036 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.036 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:51.036 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:46.555633+0000", 2024-04-19T01:04:51.037 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.0092000000000002e-05 2024-04-19T01:04:51.037 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.037 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.037 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:51.037 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:46.555658+0000", 2024-04-19T01:04:51.037 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5122000000000001e-05 2024-04-19T01:04:51.037 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.037 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.037 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:51.037 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:46.555833+0000", 2024-04-19T01:04:51.037 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00017493399999999999 2024-04-19T01:04:51.037 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.037 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.037 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:51.037 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:46.555860+0000", 2024-04-19T01:04:51.038 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.7589e-05 2024-04-19T01:04:51.038 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.038 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.038 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:51.038 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:46.569978+0000", 2024-04-19T01:04:51.038 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.014117299999999999 2024-04-19T01:04:51.038 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.038 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.038 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:51.038 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:46.570068+0000", 2024-04-19T01:04:51.038 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.0334000000000006e-05 2024-04-19T01:04:51.038 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.038 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:51.039 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.039 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.039 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:51.039 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:51.040 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-04-19T01:04:51.041 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:51.038+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:51.142 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:04:51.138+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:04:51.209 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:51.209 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [], 2024-04-19T01:04:51.209 INFO:teuthology.orchestra.run.smithi110.stdout: "num_ops": 0 2024-04-19T01:04:51.209 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:51.224 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-04-19T01:04:51.267 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:04:51.266+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:04:51.342 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-19T01:04:51.343 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:04:51.342+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:04:51.392 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:51.393 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [], 2024-04-19T01:04:51.393 INFO:teuthology.orchestra.run.smithi110.stdout: "complaint_time": 30, 2024-04-19T01:04:51.393 INFO:teuthology.orchestra.run.smithi110.stdout: "num_blocked_ops": 0 2024-04-19T01:04:51.393 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:51.406 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-04-19T01:04:51.444 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:51.443+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:51.503 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-04-19T01:04:51.503 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T01:04:51.505 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.14 to 1 (10000) 2024-04-19T01:04:51.513 INFO:tasks.rados.rados.0.smithi155.stdout:252: done (1 left) 2024-04-19T01:04:51.513 INFO:tasks.rados.rados.0.smithi155.stdout:253: done (0 left) 2024-04-19T01:04:51.513 INFO:tasks.rados.rados.0.smithi155.stdout:254: rmattr oid 32 current snap is 27 2024-04-19T01:04:51.513 INFO:tasks.rados.rados.0.smithi155.stdout:254: done (0 left) 2024-04-19T01:04:51.513 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 3 current snap is 27 2024-04-19T01:04:51.513 INFO:tasks.rados.rados.0.smithi155.stdout:255: seq_num 112 ranges {49152=720896,770048=802816,1572864=360448} 2024-04-19T01:04:51.524 INFO:tasks.rados.rados.0.smithi155.stdout:255: writing smithi15528762-3 from 49152 to 770048 tid 1 2024-04-19T01:04:51.530 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-04-19T01:04:51.530 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T01:04:51.531 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 15 1 2024-04-19T01:04:51.536 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:51.536 INFO:teuthology.orchestra.run.smithi110.stdout: "size": 20, 2024-04-19T01:04:51.536 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 600, 2024-04-19T01:04:51.537 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [ 2024-04-19T01:04:51.537 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.537 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:235 3.9s0 3:93d266e5:::smithi15528762-34:head [append 0~688128] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:51.537 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.857683+0000", 2024-04-19T01:04:51.537 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 37.676649118999997, 2024-04-19T01:04:51.537 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.044416514999999997, 2024-04-19T01:04:51.537 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:51.537 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:51.537 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:51.537 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:51.537 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:51.537 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:51.537 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 235 2024-04-19T01:04:51.538 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.538 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:51.538 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.538 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:51.538 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.857683+0000", 2024-04-19T01:04:51.538 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.538 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.538 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.538 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:51.538 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.857683+0000", 2024-04-19T01:04:51.538 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.538 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.539 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.539 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:51.539 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.857687+0000", 2024-04-19T01:04:51.539 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.1130000000000001e-06 2024-04-19T01:04:51.539 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.539 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.539 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:51.539 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.859118+0000", 2024-04-19T01:04:51.539 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001431114 2024-04-19T01:04:51.539 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.539 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.539 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:51.540 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.859150+0000", 2024-04-19T01:04:51.540 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.2589000000000003e-05 2024-04-19T01:04:51.540 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.540 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.540 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:51.540 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.860377+0000", 2024-04-19T01:04:51.541 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001226878 2024-04-19T01:04:51.541 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.541 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.541 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:51.541 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.866898+0000", 2024-04-19T01:04:51.541 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0065209250000000003 2024-04-19T01:04:51.541 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.541 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.541 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:51.541 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.867163+0000", 2024-04-19T01:04:51.542 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000264901 2024-04-19T01:04:51.542 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.542 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.542 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:51.542 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.869905+0000", 2024-04-19T01:04:51.542 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0027423550000000001 2024-04-19T01:04:51.542 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.542 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.542 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:51.542 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.897422+0000", 2024-04-19T01:04:51.542 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.027516445 2024-04-19T01:04:51.542 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.542 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.542 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:51.543 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.901877+0000", 2024-04-19T01:04:51.543 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.004454965 2024-04-19T01:04:51.543 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.543 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.543 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:51.543 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.902100+0000", 2024-04-19T01:04:51.543 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00022322999999999999 2024-04-19T01:04:51.543 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.543 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:51.543 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.544 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.544 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.544 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:236 3.9s0 3:93d266e5:::smithi15528762-34:head [append 0~376832] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:51.544 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.860599+0000", 2024-04-19T01:04:51.544 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 37.673733099000003, 2024-04-19T01:04:51.544 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.042094899999999998, 2024-04-19T01:04:51.544 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:51.544 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:51.544 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:51.544 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:51.544 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:51.545 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:51.545 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 236 2024-04-19T01:04:51.545 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.545 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:51.545 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.545 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:51.545 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.860599+0000", 2024-04-19T01:04:51.545 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.545 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.545 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.545 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:51.545 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.860599+0000", 2024-04-19T01:04:51.545 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.545 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.546 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.546 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:51.546 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.860602+0000", 2024-04-19T01:04:51.546 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4899999999999999e-06 2024-04-19T01:04:51.546 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.546 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.546 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:51.546 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861320+0000", 2024-04-19T01:04:51.546 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00071774399999999998 2024-04-19T01:04:51.546 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.546 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.546 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:51.547 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861368+0000", 2024-04-19T01:04:51.547 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.8016999999999998e-05 2024-04-19T01:04:51.547 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.547 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.547 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:51.547 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861389+0000", 2024-04-19T01:04:51.547 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1175999999999999e-05 2024-04-19T01:04:51.547 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.547 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.547 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:51.547 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.888682+0000", 2024-04-19T01:04:51.547 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.027292739 2024-04-19T01:04:51.547 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.547 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.548 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:51.548 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.888867+0000", 2024-04-19T01:04:51.548 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00018496499999999999 2024-04-19T01:04:51.548 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.548 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.548 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:51.548 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.890430+0000", 2024-04-19T01:04:51.548 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001563069 2024-04-19T01:04:51.548 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.548 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.548 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:51.548 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.898524+0000", 2024-04-19T01:04:51.548 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0080939800000000006 2024-04-19T01:04:51.548 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.549 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.549 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:51.549 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.902524+0000", 2024-04-19T01:04:51.549 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0039999930000000003 2024-04-19T01:04:51.549 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.549 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.549 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:51.549 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.902694+0000", 2024-04-19T01:04:51.549 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00017072699999999999 2024-04-19T01:04:51.549 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.549 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:51.549 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.549 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.550 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.550 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:237 3.9s0 3:93d266e5:::smithi15528762-34:head [setxattr _header (56) in=7b] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:51.550 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.861460+0000", 2024-04-19T01:04:51.550 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 37.672872542, 2024-04-19T01:04:51.550 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.042252307000000003, 2024-04-19T01:04:51.550 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:51.551 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:51.551 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:51.551 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:51.551 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:51.551 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:51.551 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 237 2024-04-19T01:04:51.551 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.551 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:51.551 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.551 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:51.552 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861460+0000", 2024-04-19T01:04:51.552 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.552 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.552 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.552 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:51.552 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861460+0000", 2024-04-19T01:04:51.552 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.552 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.552 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.552 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:51.552 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861462+0000", 2024-04-19T01:04:51.552 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9929999999999998e-06 2024-04-19T01:04:51.552 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.553 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.553 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:51.553 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861472+0000", 2024-04-19T01:04:51.553 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.9070000000000002e-06 2024-04-19T01:04:51.553 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.553 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.553 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:51.553 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861493+0000", 2024-04-19T01:04:51.553 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1386000000000002e-05 2024-04-19T01:04:51.553 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.554 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.554 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:51.554 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861510+0000", 2024-04-19T01:04:51.554 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7198999999999999e-05 2024-04-19T01:04:51.554 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.554 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.554 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:51.555 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.899576+0000", 2024-04-19T01:04:51.555 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.038065486000000003 2024-04-19T01:04:51.555 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.555 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.555 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:51.555 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.899753+0000", 2024-04-19T01:04:51.555 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000176522 2024-04-19T01:04:51.555 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.555 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.555 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:51.555 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.900579+0000", 2024-04-19T01:04:51.555 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00082676899999999998 2024-04-19T01:04:51.555 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.556 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.556 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:51.556 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.902843+0000", 2024-04-19T01:04:51.556 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0022635329999999999 2024-04-19T01:04:51.556 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.556 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.556 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:51.556 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.903504+0000", 2024-04-19T01:04:51.556 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00066085099999999995 2024-04-19T01:04:51.556 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.556 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.556 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:51.556 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.903712+0000", 2024-04-19T01:04:51.556 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00020866099999999999 2024-04-19T01:04:51.557 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.557 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:51.557 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.557 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.557 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.557 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:238 3.9s0 3:93d266e5:::smithi15528762-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:51.557 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.861534+0000", 2024-04-19T01:04:51.557 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 37.672798354999998, 2024-04-19T01:04:51.557 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.046352598000000002, 2024-04-19T01:04:51.557 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:51.557 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:51.557 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:04:51.557 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:51.557 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:51.558 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:51.558 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 238 2024-04-19T01:04:51.558 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.558 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:51.558 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.558 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:51.558 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861534+0000", 2024-04-19T01:04:51.558 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.558 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.558 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.558 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:51.558 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861534+0000", 2024-04-19T01:04:51.558 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.558 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.559 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.559 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:51.559 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861536+0000", 2024-04-19T01:04:51.559 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.607e-06 2024-04-19T01:04:51.559 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.559 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.560 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:51.560 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861542+0000", 2024-04-19T01:04:51.560 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.1689999999999997e-06 2024-04-19T01:04:51.560 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.560 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.560 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:51.560 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861562+0000", 2024-04-19T01:04:51.560 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0043000000000001e-05 2024-04-19T01:04:51.560 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.560 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.561 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:51.561 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861882+0000", 2024-04-19T01:04:51.561 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000320398 2024-04-19T01:04:51.561 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.561 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.561 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:51.561 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.901266+0000", 2024-04-19T01:04:51.561 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.039383373999999999 2024-04-19T01:04:51.561 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.561 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.561 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting for rw locks", 2024-04-19T01:04:51.561 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.901474+0000", 2024-04-19T01:04:51.561 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00020846999999999999 2024-04-19T01:04:51.562 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.562 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.562 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:51.562 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.904551+0000", 2024-04-19T01:04:51.562 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0030766690000000002 2024-04-19T01:04:51.562 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.562 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.562 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:51.562 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.904806+0000", 2024-04-19T01:04:51.562 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00025481899999999999 2024-04-19T01:04:51.562 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.563 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.563 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:51.563 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.907887+0000", 2024-04-19T01:04:51.563 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0030810490000000002 2024-04-19T01:04:51.563 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.563 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:51.563 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.563 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.563 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.563 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:267 3.9s0 3:93d266e5:::smithi15528762-34:head [append 0~737280] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:51.563 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.013235+0000", 2024-04-19T01:04:51.563 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 37.521097234000003, 2024-04-19T01:04:51.563 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.043578804999999998, 2024-04-19T01:04:51.564 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:51.564 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:51.564 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:51.564 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:51.564 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:51.564 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:51.564 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 267 2024-04-19T01:04:51.564 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.564 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:51.564 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.564 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:51.564 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.013235+0000", 2024-04-19T01:04:51.564 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.565 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.565 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.565 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:51.565 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.013235+0000", 2024-04-19T01:04:51.565 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.565 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.565 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.565 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:51.565 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.013239+0000", 2024-04-19T01:04:51.565 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.6200000000000001e-06 2024-04-19T01:04:51.565 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.566 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.566 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:51.566 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.014549+0000", 2024-04-19T01:04:51.566 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001310075 2024-04-19T01:04:51.566 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.566 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.566 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:51.566 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.014585+0000", 2024-04-19T01:04:51.566 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.5494999999999998e-05 2024-04-19T01:04:51.566 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.566 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.566 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:51.566 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.014607+0000", 2024-04-19T01:04:51.567 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2371999999999999e-05 2024-04-19T01:04:51.567 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.567 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.567 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:51.567 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.014744+0000", 2024-04-19T01:04:51.567 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000136988 2024-04-19T01:04:51.567 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.567 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.567 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:51.567 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.015097+0000", 2024-04-19T01:04:51.567 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00035269400000000002 2024-04-19T01:04:51.567 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.567 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.568 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:51.568 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.020447+0000", 2024-04-19T01:04:51.568 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0053503739999999998 2024-04-19T01:04:51.568 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.568 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.568 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:51.568 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.047175+0000", 2024-04-19T01:04:51.568 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.026728372 2024-04-19T01:04:51.568 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.568 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.568 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:51.568 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.056595+0000", 2024-04-19T01:04:51.569 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0094197320000000001 2024-04-19T01:04:51.569 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.569 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.569 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:51.569 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.056814+0000", 2024-04-19T01:04:51.569 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00021908299999999999 2024-04-19T01:04:51.569 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.570 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:51.570 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.570 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.570 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.570 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:268 3.9s0 3:93d266e5:::smithi15528762-34:head [append 0~376832] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:51.570 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.015660+0000", 2024-04-19T01:04:51.570 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 37.518672199000001, 2024-04-19T01:04:51.571 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.041602957000000003, 2024-04-19T01:04:51.571 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:51.571 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:51.571 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:51.571 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:51.571 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:51.571 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:51.571 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 268 2024-04-19T01:04:51.571 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.571 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:51.571 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.571 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:51.571 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.015660+0000", 2024-04-19T01:04:51.571 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.572 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.572 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.572 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:51.572 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.015660+0000", 2024-04-19T01:04:51.572 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.572 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.572 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.572 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:51.572 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.015663+0000", 2024-04-19T01:04:51.572 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.0299999999999998e-06 2024-04-19T01:04:51.572 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.572 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.572 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:51.572 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016328+0000", 2024-04-19T01:04:51.573 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00066432000000000004 2024-04-19T01:04:51.573 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.573 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.573 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:51.573 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016362+0000", 2024-04-19T01:04:51.573 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.4709000000000003e-05 2024-04-19T01:04:51.573 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.573 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.573 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:51.573 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016384+0000", 2024-04-19T01:04:51.573 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1172999999999999e-05 2024-04-19T01:04:51.573 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.573 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.573 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:51.574 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.038612+0000", 2024-04-19T01:04:51.574 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.022227925999999999 2024-04-19T01:04:51.574 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.574 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.574 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:51.574 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.038788+0000", 2024-04-19T01:04:51.574 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000176095 2024-04-19T01:04:51.574 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.574 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.574 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:51.574 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.040120+0000", 2024-04-19T01:04:51.574 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001331917 2024-04-19T01:04:51.574 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.574 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.574 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:51.575 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.047665+0000", 2024-04-19T01:04:51.575 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0075450789999999997 2024-04-19T01:04:51.575 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.575 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.576 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:51.576 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.057085+0000", 2024-04-19T01:04:51.576 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0094201930000000003 2024-04-19T01:04:51.576 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.576 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.576 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:51.576 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.057263+0000", 2024-04-19T01:04:51.576 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00017851499999999999 2024-04-19T01:04:51.576 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.576 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:51.576 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.576 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.576 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.577 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:269 3.9s0 3:93d266e5:::smithi15528762-34:head [setxattr _header (56) in=7b] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:51.577 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.016434+0000", 2024-04-19T01:04:51.577 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 37.517898236999997, 2024-04-19T01:04:51.577 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.041381567000000001, 2024-04-19T01:04:51.577 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:51.577 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:51.577 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:51.577 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:51.577 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:51.577 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:51.577 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 269 2024-04-19T01:04:51.577 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.577 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:51.577 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.578 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:51.578 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016434+0000", 2024-04-19T01:04:51.578 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.578 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.578 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.578 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:51.578 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016434+0000", 2024-04-19T01:04:51.578 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.578 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.578 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.578 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:51.578 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016436+0000", 2024-04-19T01:04:51.578 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5939999999999999e-06 2024-04-19T01:04:51.579 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.579 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.579 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:51.579 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016445+0000", 2024-04-19T01:04:51.579 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.9889999999999998e-06 2024-04-19T01:04:51.579 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.579 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.579 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:51.579 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016465+0000", 2024-04-19T01:04:51.579 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0403000000000001e-05 2024-04-19T01:04:51.579 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.579 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.579 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:51.579 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016481+0000", 2024-04-19T01:04:51.580 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5772000000000001e-05 2024-04-19T01:04:51.580 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.580 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.580 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:51.580 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.047733+0000", 2024-04-19T01:04:51.580 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.031251689999999999 2024-04-19T01:04:51.580 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.580 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.580 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:51.580 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.047885+0000", 2024-04-19T01:04:51.580 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015215300000000001 2024-04-19T01:04:51.580 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.580 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.580 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:51.581 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.048690+0000", 2024-04-19T01:04:51.581 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00080468100000000002 2024-04-19T01:04:51.581 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.581 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.581 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:51.581 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.049873+0000", 2024-04-19T01:04:51.581 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0011833200000000001 2024-04-19T01:04:51.581 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.581 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.581 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:51.581 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.057577+0000", 2024-04-19T01:04:51.581 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0077039860000000003 2024-04-19T01:04:51.581 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.581 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.582 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:51.582 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.057816+0000", 2024-04-19T01:04:51.582 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000238979 2024-04-19T01:04:51.582 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.582 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:51.582 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.582 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.582 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.582 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:270 3.9s0 3:93d266e5:::smithi15528762-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:51.582 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.016503+0000", 2024-04-19T01:04:51.582 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 37.517829069999998, 2024-04-19T01:04:51.582 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.044494194000000001, 2024-04-19T01:04:51.582 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:51.582 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:51.583 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:04:51.583 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:51.583 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:51.583 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:51.583 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 270 2024-04-19T01:04:51.583 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.583 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:51.583 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.583 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:51.583 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016503+0000", 2024-04-19T01:04:51.583 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.583 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.583 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.583 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:51.583 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016503+0000", 2024-04-19T01:04:51.584 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.584 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.584 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.584 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:51.584 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016505+0000", 2024-04-19T01:04:51.584 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.2729999999999999e-06 2024-04-19T01:04:51.584 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.584 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.584 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:51.584 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016511+0000", 2024-04-19T01:04:51.584 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.7570000000000001e-06 2024-04-19T01:04:51.584 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.584 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.584 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:51.585 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016530+0000", 2024-04-19T01:04:51.585 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9565999999999999e-05 2024-04-19T01:04:51.585 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.585 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.585 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:51.586 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016545+0000", 2024-04-19T01:04:51.586 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.4963000000000001e-05 2024-04-19T01:04:51.586 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.586 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.586 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:51.586 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.049233+0000", 2024-04-19T01:04:51.586 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.032687815000000002 2024-04-19T01:04:51.586 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.586 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.586 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting for rw locks", 2024-04-19T01:04:51.586 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.049365+0000", 2024-04-19T01:04:51.586 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00013258399999999999 2024-04-19T01:04:51.587 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.587 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.587 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:51.587 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.058814+0000", 2024-04-19T01:04:51.587 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0094481080000000002 2024-04-19T01:04:51.587 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.587 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.587 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:51.587 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.059010+0000", 2024-04-19T01:04:51.587 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00019622699999999999 2024-04-19T01:04:51.587 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.587 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.588 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:51.588 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.060998+0000", 2024-04-19T01:04:51.588 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0019879009999999998 2024-04-19T01:04:51.588 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.588 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:51.588 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.588 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.588 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.588 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:280 3.9s0 3:98647342:::smithi15528762-43:head [append 0~753664] snapc 7=[7,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-19T01:04:51.588 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.869239+0000", 2024-04-19T01:04:51.588 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 36.665093200999998, 2024-04-19T01:04:51.589 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.062785076999999995, 2024-04-19T01:04:51.589 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:51.589 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:51.589 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:51.589 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:51.589 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:51.589 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:51.589 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 280 2024-04-19T01:04:51.589 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.589 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:51.589 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.589 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:51.589 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.869239+0000", 2024-04-19T01:04:51.589 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.589 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.590 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.590 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:51.590 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.869239+0000", 2024-04-19T01:04:51.590 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.590 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.590 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.590 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:51.590 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.869243+0000", 2024-04-19T01:04:51.590 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.444e-06 2024-04-19T01:04:51.590 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.590 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.590 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:51.590 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.870594+0000", 2024-04-19T01:04:51.591 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001351662 2024-04-19T01:04:51.591 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.591 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.591 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:51.591 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.870629+0000", 2024-04-19T01:04:51.591 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.4329000000000003e-05 2024-04-19T01:04:51.591 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.591 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.591 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:51.591 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.870647+0000", 2024-04-19T01:04:51.591 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7768999999999999e-05 2024-04-19T01:04:51.591 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.591 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.591 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:51.591 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.870806+0000", 2024-04-19T01:04:51.592 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015962300000000001 2024-04-19T01:04:51.592 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.592 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.592 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "op must wait for map", 2024-04-19T01:04:51.592 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.870819+0000", 2024-04-19T01:04:51.592 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.3266e-05 2024-04-19T01:04:51.592 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.592 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.592 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:51.592 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875741+0000", 2024-04-19T01:04:51.592 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0049211599999999999 2024-04-19T01:04:51.592 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.592 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.592 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:51.592 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.876018+0000", 2024-04-19T01:04:51.593 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000276944 2024-04-19T01:04:51.593 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.593 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.593 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:51.593 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.880141+0000", 2024-04-19T01:04:51.593 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0041231030000000004 2024-04-19T01:04:51.593 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.593 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.593 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:51.593 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.915254+0000", 2024-04-19T01:04:51.593 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.035113610000000003 2024-04-19T01:04:51.593 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.593 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.593 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:51.594 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.931834+0000", 2024-04-19T01:04:51.594 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.016580212 2024-04-19T01:04:51.594 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.594 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.594 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:51.594 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.932024+0000", 2024-04-19T01:04:51.594 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00018995500000000001 2024-04-19T01:04:51.594 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.594 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:51.594 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.594 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.594 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.594 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:281 3.9s0 3:98647342:::smithi15528762-43:head [append 0~376832] snapc 7=[7,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-19T01:04:51.594 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.872157+0000", 2024-04-19T01:04:51.594 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 36.662175578000003, 2024-04-19T01:04:51.595 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.061221235999999998, 2024-04-19T01:04:51.595 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:51.595 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:51.595 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:51.595 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:51.595 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:51.595 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:51.595 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 281 2024-04-19T01:04:51.595 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.595 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:51.595 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.595 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:51.595 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872157+0000", 2024-04-19T01:04:51.595 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.596 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.596 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.596 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:51.596 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872157+0000", 2024-04-19T01:04:51.596 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.596 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.596 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.596 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:51.597 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872159+0000", 2024-04-19T01:04:51.597 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5129999999999999e-06 2024-04-19T01:04:51.597 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.597 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.597 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:51.597 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872435+0000", 2024-04-19T01:04:51.597 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00027559500000000001 2024-04-19T01:04:51.597 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.597 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.597 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:51.597 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872470+0000", 2024-04-19T01:04:51.597 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.5170999999999999e-05 2024-04-19T01:04:51.597 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.597 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.597 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:51.598 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872488+0000", 2024-04-19T01:04:51.598 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.8136000000000001e-05 2024-04-19T01:04:51.598 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.598 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.598 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:51.598 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872639+0000", 2024-04-19T01:04:51.598 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015092 2024-04-19T01:04:51.598 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.598 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.598 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:04:51.598 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872658+0000", 2024-04-19T01:04:51.598 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.8882999999999999e-05 2024-04-19T01:04:51.598 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.599 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.599 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:51.599 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.903660+0000", 2024-04-19T01:04:51.599 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.031001846999999999 2024-04-19T01:04:51.599 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.599 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.599 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:51.599 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.904268+0000", 2024-04-19T01:04:51.599 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00060793999999999996 2024-04-19T01:04:51.599 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.599 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.599 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:51.599 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.906536+0000", 2024-04-19T01:04:51.599 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0022680180000000001 2024-04-19T01:04:51.599 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.600 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.600 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:51.600 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.918698+0000", 2024-04-19T01:04:51.600 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.012161852000000001 2024-04-19T01:04:51.600 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.600 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.600 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:51.600 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.933228+0000", 2024-04-19T01:04:51.600 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.014529781 2024-04-19T01:04:51.600 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.600 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.600 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:51.600 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.933378+0000", 2024-04-19T01:04:51.601 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015058000000000001 2024-04-19T01:04:51.601 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.601 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:51.601 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.601 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.601 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.601 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:282 3.9s0 3:98647342:::smithi15528762-43:head [setxattr _header (56) in=7b] snapc 7=[7,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-19T01:04:51.601 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.872536+0000", 2024-04-19T01:04:51.601 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 36.661796074999998, 2024-04-19T01:04:51.601 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.062122945999999998, 2024-04-19T01:04:51.601 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:51.601 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:51.601 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:51.601 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:51.601 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:51.602 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:51.602 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 282 2024-04-19T01:04:51.602 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.602 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:51.602 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.602 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:51.602 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872536+0000", 2024-04-19T01:04:51.602 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.602 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.602 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.602 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:51.602 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872536+0000", 2024-04-19T01:04:51.602 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.602 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.603 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.603 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:51.603 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872538+0000", 2024-04-19T01:04:51.603 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.423e-06 2024-04-19T01:04:51.603 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.603 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.603 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:51.603 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872548+0000", 2024-04-19T01:04:51.603 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.8430000000000005e-06 2024-04-19T01:04:51.603 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.603 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.603 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:51.603 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872569+0000", 2024-04-19T01:04:51.603 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1336e-05 2024-04-19T01:04:51.603 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.604 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.604 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:51.604 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872585+0000", 2024-04-19T01:04:51.604 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6288999999999999e-05 2024-04-19T01:04:51.604 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.604 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.604 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:51.604 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872783+0000", 2024-04-19T01:04:51.604 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000197488 2024-04-19T01:04:51.604 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.604 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.604 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:04:51.604 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872804+0000", 2024-04-19T01:04:51.604 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0839000000000002e-05 2024-04-19T01:04:51.605 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.605 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.605 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:51.605 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.915520+0000", 2024-04-19T01:04:51.605 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.042715887000000001 2024-04-19T01:04:51.605 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.605 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.605 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:51.605 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.915852+0000", 2024-04-19T01:04:51.605 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00033199899999999998 2024-04-19T01:04:51.605 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.605 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.605 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:51.605 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.916938+0000", 2024-04-19T01:04:51.605 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0010863940000000001 2024-04-19T01:04:51.606 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.606 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.606 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:51.606 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.918701+0000", 2024-04-19T01:04:51.606 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001763351 2024-04-19T01:04:51.606 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.606 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.607 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:51.607 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.934283+0000", 2024-04-19T01:04:51.607 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.015581526 2024-04-19T01:04:51.607 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.607 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.607 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:51.607 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.934659+0000", 2024-04-19T01:04:51.607 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00037657100000000002 2024-04-19T01:04:51.607 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.607 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:51.607 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.607 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.607 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.607 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:283 3.9s0 3:98647342:::smithi15528762-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-04-19T01:04:51.608 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.872609+0000", 2024-04-19T01:04:51.608 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 36.661723225000003, 2024-04-19T01:04:51.608 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.067525222999999995, 2024-04-19T01:04:51.608 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:51.608 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:51.608 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:04:51.608 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:51.608 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:51.608 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:51.608 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 283 2024-04-19T01:04:51.608 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.608 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:51.608 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.608 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:51.608 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872609+0000", 2024-04-19T01:04:51.609 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.609 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.609 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.609 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:51.609 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872609+0000", 2024-04-19T01:04:51.609 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.609 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.609 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.609 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:51.609 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872611+0000", 2024-04-19T01:04:51.609 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.367e-06 2024-04-19T01:04:51.609 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.609 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.609 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:51.610 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872617+0000", 2024-04-19T01:04:51.610 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.2400000000000004e-06 2024-04-19T01:04:51.610 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.610 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.610 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:51.610 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872636+0000", 2024-04-19T01:04:51.610 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.8941999999999999e-05 2024-04-19T01:04:51.610 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.610 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.610 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:51.610 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872652+0000", 2024-04-19T01:04:51.610 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5651999999999999e-05 2024-04-19T01:04:51.610 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.610 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.611 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:51.611 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872858+0000", 2024-04-19T01:04:51.611 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00020631100000000001 2024-04-19T01:04:51.611 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.611 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.611 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:04:51.611 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872867+0000", 2024-04-19T01:04:51.611 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.5400000000000001e-06 2024-04-19T01:04:51.611 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.611 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.611 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:51.611 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.917636+0000", 2024-04-19T01:04:51.611 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.044768372000000001 2024-04-19T01:04:51.611 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.611 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.612 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting for rw locks", 2024-04-19T01:04:51.612 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.917899+0000", 2024-04-19T01:04:51.612 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00026336799999999999 2024-04-19T01:04:51.612 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.612 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.612 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:51.612 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.935711+0000", 2024-04-19T01:04:51.612 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.017811862000000001 2024-04-19T01:04:51.612 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.612 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.612 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:51.612 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.935937+0000", 2024-04-19T01:04:51.612 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00022568800000000001 2024-04-19T01:04:51.612 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.613 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.613 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:51.613 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.940135+0000", 2024-04-19T01:04:51.613 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0041978809999999997 2024-04-19T01:04:51.613 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.613 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:51.613 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.613 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.613 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.613 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs1 48/31 ECSubWrite(tid=76, reqid=client.4604.0:345, at_version=48'22, trim_to=0'0, roll_forward_to=47'19))", 2024-04-19T01:04:51.613 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.538686+0000", 2024-04-19T01:04:51.613 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 32.995646620999999, 2024-04-19T01:04:51.613 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.040628868999999998, 2024-04-19T01:04:51.613 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:51.614 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:51.614 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:04:51.614 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:51.614 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.614 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:51.614 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538686+0000", 2024-04-19T01:04:51.614 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.614 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.614 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.614 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:51.614 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538686+0000", 2024-04-19T01:04:51.614 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.614 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.615 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.615 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:51.615 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538688+0000", 2024-04-19T01:04:51.615 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5799999999999999e-06 2024-04-19T01:04:51.615 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.615 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.615 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:51.615 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.539128+0000", 2024-04-19T01:04:51.615 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00044051499999999999 2024-04-19T01:04:51.615 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.615 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.615 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:51.615 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.539160+0000", 2024-04-19T01:04:51.615 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.2255999999999997e-05 2024-04-19T01:04:51.616 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.616 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.616 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:51.616 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.539184+0000", 2024-04-19T01:04:51.616 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3699000000000001e-05 2024-04-19T01:04:51.616 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.616 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.616 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:51.616 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.539555+0000", 2024-04-19T01:04:51.616 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00037067799999999999 2024-04-19T01:04:51.616 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.616 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.616 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:51.617 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.539639+0000", 2024-04-19T01:04:51.617 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.4625999999999999e-05 2024-04-19T01:04:51.617 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.617 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.617 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:51.617 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.579272+0000", 2024-04-19T01:04:51.617 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.039632435000000001 2024-04-19T01:04:51.617 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.618 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.618 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:51.618 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.579315+0000", 2024-04-19T01:04:51.618 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.3080000000000001e-05 2024-04-19T01:04:51.618 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.618 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:51.618 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.618 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.618 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.618 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:352 3.9s0 3:98647342:::smithi15528762-43:head [append 0~0] snapc 9=[8,7] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:04:51.618 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.542861+0000", 2024-04-19T01:04:51.618 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 32.991471570999998, 2024-04-19T01:04:51.618 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.033209177999999999, 2024-04-19T01:04:51.618 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:51.619 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:51.619 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:51.619 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:51.619 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:51.619 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:51.619 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 352 2024-04-19T01:04:51.619 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.619 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:51.619 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.619 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:51.619 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.542861+0000", 2024-04-19T01:04:51.619 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.619 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.619 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.619 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:51.620 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.542861+0000", 2024-04-19T01:04:51.620 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.620 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.620 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.620 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:51.620 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.542862+0000", 2024-04-19T01:04:51.620 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.2130000000000001e-06 2024-04-19T01:04:51.620 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.620 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.620 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:51.620 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.542868+0000", 2024-04-19T01:04:51.620 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.4770000000000002e-06 2024-04-19T01:04:51.620 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.620 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.621 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:51.621 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.542886+0000", 2024-04-19T01:04:51.621 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.8119000000000001e-05 2024-04-19T01:04:51.621 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.621 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.621 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:51.621 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.542900+0000", 2024-04-19T01:04:51.621 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.4083000000000001e-05 2024-04-19T01:04:51.621 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.621 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.621 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:51.621 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.562653+0000", 2024-04-19T01:04:51.621 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.019752998000000001 2024-04-19T01:04:51.621 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.622 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.622 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:51.622 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.563024+0000", 2024-04-19T01:04:51.622 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00037130000000000003 2024-04-19T01:04:51.622 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.622 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.622 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:51.622 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.564055+0000", 2024-04-19T01:04:51.622 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0010311000000000001 2024-04-19T01:04:51.622 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.622 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.622 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:51.622 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.574688+0000", 2024-04-19T01:04:51.622 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.010632561 2024-04-19T01:04:51.623 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.623 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.623 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:51.623 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.574784+0000", 2024-04-19T01:04:51.623 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.6129000000000001e-05 2024-04-19T01:04:51.623 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.623 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.623 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:51.623 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.576070+0000", 2024-04-19T01:04:51.623 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0012861979999999999 2024-04-19T01:04:51.623 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.623 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:51.623 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.623 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.624 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.624 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:465 3.9s0 3:971ff57a:::smithi15528762-32:head [copy-from ver 9] snapc 11=[11,10,d] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-19T01:04:51.624 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:33.550965+0000", 2024-04-19T01:04:51.624 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 17.983367634, 2024-04-19T01:04:51.624 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.035302247000000002, 2024-04-19T01:04:51.624 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:51.624 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:51.624 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:51.624 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:51.624 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:51.624 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:51.624 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 465 2024-04-19T01:04:51.624 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.624 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:51.624 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.625 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:51.625 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.550965+0000", 2024-04-19T01:04:51.625 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.625 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.625 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.625 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:51.625 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.550965+0000", 2024-04-19T01:04:51.625 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.625 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.625 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.625 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:51.625 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.550968+0000", 2024-04-19T01:04:51.625 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.0599999999999999e-06 2024-04-19T01:04:51.625 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.626 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.626 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:51.626 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.550983+0000", 2024-04-19T01:04:51.626 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5396999999999998e-05 2024-04-19T01:04:51.626 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.626 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.626 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:51.626 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.551016+0000", 2024-04-19T01:04:51.626 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.2305000000000003e-05 2024-04-19T01:04:51.626 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.626 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.626 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:51.626 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.551032+0000", 2024-04-19T01:04:51.626 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6730000000000001e-05 2024-04-19T01:04:51.627 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.627 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.627 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:51.627 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.551143+0000", 2024-04-19T01:04:51.627 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000110433 2024-04-19T01:04:51.627 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.627 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.627 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:51.627 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.551331+0000", 2024-04-19T01:04:51.627 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000188462 2024-04-19T01:04:51.627 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.627 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.627 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:51.627 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.565853+0000", 2024-04-19T01:04:51.628 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.014521479 2024-04-19T01:04:51.628 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.628 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.628 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:51.628 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.573502+0000", 2024-04-19T01:04:51.628 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0076495579999999999 2024-04-19T01:04:51.628 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.628 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.628 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:51.628 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.586193+0000", 2024-04-19T01:04:51.628 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.012690450000000001 2024-04-19T01:04:51.628 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.628 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.628 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:51.629 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.586267+0000", 2024-04-19T01:04:51.629 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.4373000000000001e-05 2024-04-19T01:04:51.629 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.629 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:51.629 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.629 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.629 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.629 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:465 3.9s0 3:971ff57a:::smithi15528762-32:head [copy-from ver 9] snapc 11=[11,10,d] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e63)", 2024-04-19T01:04:51.629 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:33.552705+0000", 2024-04-19T01:04:51.629 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 17.981627851999999, 2024-04-19T01:04:51.629 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.033497738999999999, 2024-04-19T01:04:51.629 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:51.629 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:51.629 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "waiting for ondisk", 2024-04-19T01:04:51.629 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:51.630 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:51.630 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:51.630 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 465 2024-04-19T01:04:51.630 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.630 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:51.630 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.630 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:51.630 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.552705+0000", 2024-04-19T01:04:51.630 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.630 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.630 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.630 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:51.630 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.552705+0000", 2024-04-19T01:04:51.630 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.631 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.631 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.631 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:51.631 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.552706+0000", 2024-04-19T01:04:51.631 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7e-06 2024-04-19T01:04:51.631 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.631 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.631 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:51.631 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.552714+0000", 2024-04-19T01:04:51.631 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.4800000000000004e-06 2024-04-19T01:04:51.631 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.631 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.631 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:51.631 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.552737+0000", 2024-04-19T01:04:51.632 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3048999999999999e-05 2024-04-19T01:04:51.632 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.632 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.632 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:51.632 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.552761+0000", 2024-04-19T01:04:51.632 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4266000000000001e-05 2024-04-19T01:04:51.632 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.632 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.632 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:51.632 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.552862+0000", 2024-04-19T01:04:51.632 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00010106399999999999 2024-04-19T01:04:51.632 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.632 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.633 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting for blocked object", 2024-04-19T01:04:51.633 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.552952+0000", 2024-04-19T01:04:51.633 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.9524000000000005e-05 2024-04-19T01:04:51.633 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.633 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.633 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:51.633 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.572361+0000", 2024-04-19T01:04:51.633 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.019409182000000001 2024-04-19T01:04:51.633 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.633 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.633 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting for ondisk", 2024-04-19T01:04:51.633 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.572613+0000", 2024-04-19T01:04:51.633 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00025244999999999998 2024-04-19T01:04:51.633 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.634 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.634 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:51.634 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:33.586202+0000", 2024-04-19T01:04:51.634 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.013589024 2024-04-19T01:04:51.634 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.634 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:51.634 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.634 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.634 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.634 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:550 3.9s0 3:9a4f6987:::smithi15528762-44:head [append 0~622592] snapc 17=[17,15,14,12] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-04-19T01:04:51.634 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:44.603203+0000", 2024-04-19T01:04:51.634 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 6.9311296129999995, 2024-04-19T01:04:51.634 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.036180287999999998, 2024-04-19T01:04:51.634 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:51.635 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:51.635 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:51.635 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:51.635 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:51.635 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:51.635 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 550 2024-04-19T01:04:51.635 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.635 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:51.635 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.635 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:51.635 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.603203+0000", 2024-04-19T01:04:51.635 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.635 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.635 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.636 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:51.636 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.603203+0000", 2024-04-19T01:04:51.636 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.636 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.636 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.636 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:51.636 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.603205+0000", 2024-04-19T01:04:51.636 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.823e-06 2024-04-19T01:04:51.636 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.636 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.636 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:51.636 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.604075+0000", 2024-04-19T01:04:51.636 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00086994399999999999 2024-04-19T01:04:51.636 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.636 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.637 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:51.637 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.604100+0000", 2024-04-19T01:04:51.637 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5559e-05 2024-04-19T01:04:51.637 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.637 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.637 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:51.637 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.604114+0000", 2024-04-19T01:04:51.637 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.3426e-05 2024-04-19T01:04:51.637 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.637 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.637 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:51.637 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.620943+0000", 2024-04-19T01:04:51.637 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.016828824999999999 2024-04-19T01:04:51.637 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.638 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.638 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:51.638 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.621155+0000", 2024-04-19T01:04:51.638 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00021226900000000001 2024-04-19T01:04:51.638 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.638 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.638 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:51.638 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.623073+0000", 2024-04-19T01:04:51.638 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0019186260000000001 2024-04-19T01:04:51.638 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.638 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.638 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:51.638 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.637490+0000", 2024-04-19T01:04:51.638 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.014417021 2024-04-19T01:04:51.639 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.639 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.639 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:51.639 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.639218+0000", 2024-04-19T01:04:51.639 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0017279719999999999 2024-04-19T01:04:51.639 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.639 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.639 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:51.639 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.639383+0000", 2024-04-19T01:04:51.639 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000164823 2024-04-19T01:04:51.639 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.639 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:51.639 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.639 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.639 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.640 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:551 3.9s0 3:9a4f6987:::smithi15528762-44:head [append 0~32768] snapc 17=[17,15,14,12] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-04-19T01:04:51.640 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:44.604141+0000", 2024-04-19T01:04:51.640 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 6.9301919940000003, 2024-04-19T01:04:51.640 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.036022647999999997, 2024-04-19T01:04:51.640 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:51.640 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:51.640 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:51.640 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:51.640 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:51.640 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:51.640 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 551 2024-04-19T01:04:51.640 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.640 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:51.640 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.641 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:51.641 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.604141+0000", 2024-04-19T01:04:51.641 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.641 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.641 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.641 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:51.641 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.604141+0000", 2024-04-19T01:04:51.641 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.641 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.641 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.641 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:51.641 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.604142+0000", 2024-04-19T01:04:51.641 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.22e-06 2024-04-19T01:04:51.641 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.642 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.642 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:51.642 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.604201+0000", 2024-04-19T01:04:51.642 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.9500999999999998e-05 2024-04-19T01:04:51.642 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.642 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.642 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:51.642 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.604217+0000", 2024-04-19T01:04:51.642 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6039999999999999e-05 2024-04-19T01:04:51.642 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.642 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.642 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:51.642 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.604228+0000", 2024-04-19T01:04:51.643 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.0273e-05 2024-04-19T01:04:51.643 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.643 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.643 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:51.643 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.634657+0000", 2024-04-19T01:04:51.643 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.030429235999999998 2024-04-19T01:04:51.643 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.643 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.643 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:51.643 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.634829+0000", 2024-04-19T01:04:51.643 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00017189199999999999 2024-04-19T01:04:51.643 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.643 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.643 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:51.643 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.635933+0000", 2024-04-19T01:04:51.644 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001104128 2024-04-19T01:04:51.644 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.644 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.644 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:51.644 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.638571+0000", 2024-04-19T01:04:51.644 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0026382440000000001 2024-04-19T01:04:51.644 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.644 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.644 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:51.644 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.640013+0000", 2024-04-19T01:04:51.644 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001441814 2024-04-19T01:04:51.644 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.644 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.644 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:51.645 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.640163+0000", 2024-04-19T01:04:51.645 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015029999999999999 2024-04-19T01:04:51.645 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.645 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:51.645 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.645 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.645 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.645 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:552 3.9s0 3:9a4f6987:::smithi15528762-44:head [setxattr _header (57) in=7b] snapc 17=[17,15,14,12] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-04-19T01:04:51.645 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:44.604243+0000", 2024-04-19T01:04:51.645 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 6.9300900439999999, 2024-04-19T01:04:51.645 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.040674366000000003, 2024-04-19T01:04:51.645 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:51.646 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:51.646 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:51.646 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:51.646 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:51.646 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:51.646 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 552 2024-04-19T01:04:51.646 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.646 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:51.646 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.646 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:51.646 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.604243+0000", 2024-04-19T01:04:51.646 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.646 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.647 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.647 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:51.647 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.604243+0000", 2024-04-19T01:04:51.647 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.647 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.647 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.647 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:51.647 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.604243+0000", 2024-04-19T01:04:51.647 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.6400000000000001e-07 2024-04-19T01:04:51.647 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.647 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.647 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:51.647 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.604255+0000", 2024-04-19T01:04:51.647 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.1345999999999999e-05 2024-04-19T01:04:51.647 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.648 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.648 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:51.648 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.604269+0000", 2024-04-19T01:04:51.648 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.3857e-05 2024-04-19T01:04:51.648 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.648 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.648 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:51.648 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.604278+0000", 2024-04-19T01:04:51.648 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.2690000000000005e-06 2024-04-19T01:04:51.648 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.648 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.648 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:51.648 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.641028+0000", 2024-04-19T01:04:51.648 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.036749969 2024-04-19T01:04:51.649 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.649 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.649 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:51.649 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.641253+0000", 2024-04-19T01:04:51.649 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00022485199999999999 2024-04-19T01:04:51.649 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.649 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.649 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:04:51.649 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.642194+0000", 2024-04-19T01:04:51.649 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00094166499999999995 2024-04-19T01:04:51.649 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.649 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.649 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:04:51.649 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.644146+0000", 2024-04-19T01:04:51.649 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001951232 2024-04-19T01:04:51.650 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.650 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.650 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:04:51.650 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.644774+0000", 2024-04-19T01:04:51.650 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00062884599999999996 2024-04-19T01:04:51.650 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.650 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.650 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:51.650 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.644917+0000", 2024-04-19T01:04:51.650 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000142466 2024-04-19T01:04:51.650 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.650 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:51.650 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.650 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.651 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.651 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:553 3.9s0 3:9a4f6987:::smithi15528762-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2024-04-19T01:04:51.651 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:44.604289+0000", 2024-04-19T01:04:51.651 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 6.9300432509999998, 2024-04-19T01:04:51.651 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.043277413000000001, 2024-04-19T01:04:51.651 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:04:51.651 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:04:51.651 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:04:51.651 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:04:51.651 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:04:51.651 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:51.651 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 553 2024-04-19T01:04:51.651 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.651 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:04:51.652 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.652 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:04:51.652 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.604289+0000", 2024-04-19T01:04:51.652 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.652 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.652 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.652 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:04:51.652 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.604289+0000", 2024-04-19T01:04:51.652 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:04:51.652 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.652 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.652 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:04:51.652 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.604290+0000", 2024-04-19T01:04:51.652 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.9999999999999996e-07 2024-04-19T01:04:51.652 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.653 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.653 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:04:51.653 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.604294+0000", 2024-04-19T01:04:51.653 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.7900000000000001e-06 2024-04-19T01:04:51.653 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.653 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.653 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:04:51.653 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.604307+0000", 2024-04-19T01:04:51.653 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.3322999999999999e-05 2024-04-19T01:04:51.653 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.653 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.653 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:04:51.653 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.604316+0000", 2024-04-19T01:04:51.653 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.9660000000000002e-06 2024-04-19T01:04:51.653 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.654 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.654 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:51.654 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.642854+0000", 2024-04-19T01:04:51.654 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.038537760999999997 2024-04-19T01:04:51.654 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.654 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.654 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting for rw locks", 2024-04-19T01:04:51.654 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.643145+0000", 2024-04-19T01:04:51.654 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00029066399999999999 2024-04-19T01:04:51.654 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.654 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.654 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:04:51.654 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.645653+0000", 2024-04-19T01:04:51.654 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0025080889999999998 2024-04-19T01:04:51.655 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.655 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.655 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:04:51.655 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.645758+0000", 2024-04-19T01:04:51.655 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000105598 2024-04-19T01:04:51.655 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:04:51.655 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:04:51.655 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:04:51.655 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.647567+0000", 2024-04-19T01:04:51.655 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0018084220000000001 2024-04-19T01:04:51.655 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.655 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:51.655 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.655 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:04:51.656 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:04:51.656 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:51.657 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-04-19T01:04:51.657 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:04:51.655+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:04:51.658 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T01:04:51.659 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-04-19T01:04:51.659 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-19T01:04:51.660 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T01:04:51.660 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-04-19T01:04:51.661 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T01:04:51.661 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-04-19T01:04:51.662 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T01:04:51.662 INFO:tasks.rados.rados.0.smithi155.stdout:255: writing smithi15528762-3 from 770048 to 1572864 tid 2 2024-04-19T01:04:51.662 INFO:tasks.rados.rados.0.smithi155.stdout:255: writing smithi15528762-3 from 1572864 to 1933312 tid 3 2024-04-19T01:04:51.662 INFO:tasks.rados.rados.0.smithi155.stdout:256: rmattr oid 47 current snap is 27 2024-04-19T01:04:51.662 INFO:tasks.rados.rados.0.smithi155.stdout:256: done (1 left) 2024-04-19T01:04:51.662 INFO:tasks.rados.rados.0.smithi155.stdout:257: read oid 25 snap -1 2024-04-19T01:04:51.662 INFO:tasks.rados.rados.0.smithi155.stdout:257: expect (ObjNum 24 snap 0 seq_num 24) 2024-04-19T01:04:51.662 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 38 current snap is 27 2024-04-19T01:04:51.662 INFO:tasks.rados.rados.0.smithi155.stdout:258: seq_num 113 ranges {2375680=737280,3112960=557056,3670016=606208} 2024-04-19T01:04:51.663 INFO:tasks.rados.rados.0.smithi155.stdout:258: writing smithi15528762-38 from 2375680 to 3112960 tid 1 2024-04-19T01:04:51.663 INFO:tasks.rados.rados.0.smithi155.stdout:258: writing smithi15528762-38 from 3112960 to 3670016 tid 2 2024-04-19T01:04:51.663 INFO:tasks.rados.rados.0.smithi155.stdout:258: writing smithi15528762-38 from 3670016 to 4276224 tid 3 2024-04-19T01:04:51.663 INFO:tasks.rados.rados.0.smithi155.stdout:259: copy_from oid 18 from oid 19 current snap is 27 2024-04-19T01:04:51.663 INFO:tasks.rados.rados.0.smithi155.stdout:257: expect (ObjNum 24 snap 0 seq_num 24) 2024-04-19T01:04:51.663 INFO:tasks.rados.rados.0.smithi155.stdout:255: finishing write tid 1 to smithi15528762-3 2024-04-19T01:04:51.663 INFO:tasks.rados.rados.0.smithi155.stdout:255: finishing write tid 2 to smithi15528762-3 2024-04-19T01:04:51.663 INFO:tasks.rados.rados.0.smithi155.stdout:255: finishing write tid 3 to smithi15528762-3 2024-04-19T01:04:51.663 INFO:tasks.rados.rados.0.smithi155.stdout:255: finishing write tid 4 to smithi15528762-3 2024-04-19T01:04:51.663 INFO:tasks.rados.rados.0.smithi155.stdout:255: finishing write tid 5 to smithi15528762-3 2024-04-19T01:04:51.663 INFO:tasks.rados.rados.0.smithi155.stdout:255: oid 3 updating version 0 to 60 2024-04-19T01:04:51.663 INFO:tasks.rados.rados.0.smithi155.stdout:255: oid 3 version 60 is already newer than 58 2024-04-19T01:04:51.663 INFO:tasks.rados.rados.0.smithi155.stdout:255: oid 3 version 60 is already newer than 59 2024-04-19T01:04:51.663 INFO:tasks.rados.rados.0.smithi155.stdout:255: oid 3 updating version 60 to 61 2024-04-19T01:04:51.663 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 3 v 61 (ObjNum 112 snap 27 seq_num 112) dirty exists 2024-04-19T01:04:51.664 INFO:tasks.rados.rados.0.smithi155.stdout:255: left oid 3 (ObjNum 112 snap 27 seq_num 112) 2024-04-19T01:04:51.664 INFO:tasks.rados.rados.0.smithi155.stdout:259: finishing copy_from racing read to smithi15528762-18 2024-04-19T01:04:51.664 INFO:tasks.rados.rados.0.smithi155.stdout:255: done (3 left) 2024-04-19T01:04:51.664 INFO:tasks.rados.rados.0.smithi155.stdout:257: done (2 left) 2024-04-19T01:04:51.664 INFO:tasks.rados.rados.0.smithi155.stdout:260: rollback oid 8 current snap is 27 2024-04-19T01:04:51.664 INFO:tasks.rados.rados.0.smithi155.stdout:rollback oid 8 to 23 2024-04-19T01:04:51.664 INFO:tasks.rados.rados.0.smithi155.stdout:261: read oid 39 snap -1 2024-04-19T01:04:51.664 INFO:tasks.rados.rados.0.smithi155.stdout:261: expect (ObjNum 0 snap 0 seq_num 0) 2024-04-19T01:04:51.664 INFO:tasks.rados.rados.0.smithi155.stdout:262: read oid 9 snap -1 2024-04-19T01:04:51.664 INFO:tasks.rados.rados.0.smithi155.stdout:262: expect (ObjNum 8 snap 0 seq_num 8) 2024-04-19T01:04:51.664 INFO:tasks.rados.rados.0.smithi155.stdout:263: snap_remove snap 22 2024-04-19T01:04:51.664 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-04-19T01:04:51.665 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T01:04:51.681 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-04-19T01:04:51.682 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T01:04:51.699 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2024-04-19T01:04:51.699 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T01:04:51.718 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2024-04-19T01:04:51.719 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T01:04:51.738 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2024-04-19T01:04:51.739 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T01:04:51.759 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2024-04-19T01:04:51.759 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T01:04:51.760 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:51.759+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:51.777 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2024-04-19T01:04:51.778 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-19T01:04:51.791 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:51.792 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:51.798 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2024-04-19T01:04:51.798 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-19T01:04:51.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:04:51.798+0000 7f6db5d9b640 0 -- 172.21.15.33:0/3300208915 >> [v2:172.21.15.118:6800/506416463,v1:172.21.15.118:6801/506416463] conn(0x7f6db821e550 msgr2=0x7f6db8220940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-04-19T01:04:51.806 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-19T01:04:51.860 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:04:51.858+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:04:51.961 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:04:51.959+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:04:51.970 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:51.970 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:51.985 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-19T01:04:52.021 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2024-04-19T01:04:52.021 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-19T01:04:52.051 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2024-04-19T01:04:52.051 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-19T01:04:52.062 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:04:52.062+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:04:52.149 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:52.149 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:52.163 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-19T01:04:52.164 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:04:52.162+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:04:52.264 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:04:52.264+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:04:52.287 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:52.287 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:52.303 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-19T01:04:52.365 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:04:52.362+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:04:52.422 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:52.422 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:52.436 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-19T01:04:52.483 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:52.483+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:52.515 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.15 to 1 (10000) 2024-04-19T01:04:52.523 INFO:tasks.rados.rados.0.smithi155.stdout:261: expect (ObjNum 0 snap 0 seq_num 0) 2024-04-19T01:04:52.533 INFO:tasks.rados.rados.0.smithi155.stdout:260: finishing rollback tid 0 to smithi15528762-8 2024-04-19T01:04:52.533 INFO:tasks.rados.rados.0.smithi155.stdout:262: expect (ObjNum 8 snap 0 seq_num 8) 2024-04-19T01:04:52.548 INFO:tasks.rados.rados.0.smithi155.stdout:260: finishing rollback tid 1 to smithi15528762-8 2024-04-19T01:04:52.548 INFO:tasks.rados.rados.0.smithi155.stdout:260: finishing rollback tid 2 to smithi15528762-8 2024-04-19T01:04:52.548 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 8 v 24 (ObjNum 33 snap 0 seq_num 1052353136) dirty exists 2024-04-19T01:04:52.548 INFO:tasks.rados.rados.0.smithi155.stdout:258: finishing write tid 1 to smithi15528762-38 2024-04-19T01:04:52.548 INFO:tasks.rados.rados.0.smithi155.stdout:258: finishing write tid 2 to smithi15528762-38 2024-04-19T01:04:52.548 INFO:tasks.rados.rados.0.smithi155.stdout:258: finishing write tid 3 to smithi15528762-38 2024-04-19T01:04:52.548 INFO:tasks.rados.rados.0.smithi155.stdout:258: finishing write tid 4 to smithi15528762-38 2024-04-19T01:04:52.549 INFO:tasks.rados.rados.0.smithi155.stdout:259: finishing copy_from to smithi15528762-18 2024-04-19T01:04:52.549 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 18 v 8 (ObjNum 56 snap 5 seq_num 56) dirty exists 2024-04-19T01:04:52.549 INFO:tasks.rados.rados.0.smithi155.stdout:258: finishing write tid 5 to smithi15528762-38 2024-04-19T01:04:52.549 INFO:tasks.rados.rados.0.smithi155.stdout:258: oid 38 updating version 0 to 97 2024-04-19T01:04:52.549 INFO:tasks.rados.rados.0.smithi155.stdout:258: oid 38 updating version 97 to 99 2024-04-19T01:04:52.549 INFO:tasks.rados.rados.0.smithi155.stdout:258: oid 38 version 99 is already newer than 96 2024-04-19T01:04:52.549 INFO:tasks.rados.rados.0.smithi155.stdout:258: oid 38 version 99 is already newer than 98 2024-04-19T01:04:52.549 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 38 v 99 (ObjNum 113 snap 27 seq_num 113) dirty exists 2024-04-19T01:04:52.549 INFO:tasks.rados.rados.0.smithi155.stdout:258: left oid 38 (ObjNum 113 snap 27 seq_num 113) 2024-04-19T01:04:52.549 INFO:tasks.rados.rados.0.smithi155.stdout:258: done (5 left) 2024-04-19T01:04:52.549 INFO:tasks.rados.rados.0.smithi155.stdout:259: done (4 left) 2024-04-19T01:04:52.549 INFO:tasks.rados.rados.0.smithi155.stdout:260: done (3 left) 2024-04-19T01:04:52.549 INFO:tasks.rados.rados.0.smithi155.stdout:261: done (2 left) 2024-04-19T01:04:52.549 INFO:tasks.rados.rados.0.smithi155.stdout:262: done (1 left) 2024-04-19T01:04:52.550 INFO:tasks.rados.rados.0.smithi155.stdout:263: done (0 left) 2024-04-19T01:04:52.550 INFO:tasks.rados.rados.0.smithi155.stdout:264: copy_from oid 42 from oid 48 current snap is 27 2024-04-19T01:04:52.550 INFO:tasks.rados.rados.0.smithi155.stdout:265: snap_create 2024-04-19T01:04:52.553 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-19T01:04:52.554 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T01:04:52.567 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:52.566+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:52.573 INFO:tasks.rados.rados.0.smithi155.stdout:264: finishing copy_from racing read to smithi15528762-42 2024-04-19T01:04:52.579 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:52.579 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:52.582 INFO:tasks.rados.rados.0.smithi155.stdout:264: finishing copy_from to smithi15528762-42 2024-04-19T01:04:52.582 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 42 v 30 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-04-19T01:04:52.594 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-19T01:04:52.655 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-19T01:04:52.701 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:04:52.702+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:04:52.762 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:52.762 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:52.770 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:52.767+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:52.776 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-19T01:04:52.871 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:04:52.871+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:04:52.913 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:04:52.914 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-04-19T01:04:52.931 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":99,"stamp":"2024-04-19T01:04:52.520684+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":66765344,"num_objects":71,"num_object_clones":21,"num_object_copies":487,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":7,"num_read":428,"num_read_kb":71717,"num_write":762,"num_write_kb":83620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":910,"ondisk_log_size":910,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":603148,"kb_used_data":129156,"kb_used_omap":24,"kb_used_meta":473703,"kb_avail":1509346292,"statfs":{"total":1546188226560,"available":1545570603008,"internally_reserved":0,"allocated":132255744,"data_stored":151561094,"data_compressed":1781331,"data_compressed_allocated":3710976,"data_compressed_original":8314880,"omap_allocated":25520,"internal_metadata":485071952},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8732672,"num_objects":7,"num_object_clones":7,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":14,"num_read_kb":7843,"num_write":24,"num_write_kb":8528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.908540"},"pg_stats":[{"pgid":"3.f","version":"79'47","reported_seq":289,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-19T01:04:50.662599+0000","last_change":"2024-04-19T01:04:49.630730+0000","last_active":"2024-04-19T01:04:50.662599+0000","last_peered":"2024-04-19T01:04:50.662599+0000","last_clean":"2024-04-19T01:04:50.662599+0000","last_became_active":"2024-04-19T01:04:03.238651+0000","last_became_peered":"2024-04-19T01:04:03.238651+0000","last_unstale":"2024-04-19T01:04:50.662599+0000","last_undegraded":"2024-04-19T01:04:50.662599+0000","last_fullsized":"2024-04-19T01:04:50.662599+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:24.895505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081676800000000005,"stat_sum":{"num_bytes":4472832,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":5288,"num_write":41,"num_write_kb":4368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,5,11,7,1,2],"acting":[0,13,5,11,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"1a","length":"1"}]},{"pgid":"3.e","version":"70'36","reported_seq":285,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-19T01:04:51.546153+0000","last_change":"2024-04-19T01:04:49.634440+0000","last_active":"2024-04-19T01:04:51.546153+0000","last_peered":"2024-04-19T01:04:51.546153+0000","last_clean":"2024-04-19T01:04:51.546153+0000","last_became_active":"2024-04-19T01:04:03.238386+0000","last_became_peered":"2024-04-19T01:04:03.238386+0000","last_unstale":"2024-04-19T01:04:51.546153+0000","last_undegraded":"2024-04-19T01:04:51.546153+0000","last_fullsized":"2024-04-19T01:04:51.546153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:22.119764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018389929999999999,"stat_sum":{"num_bytes":1703936,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":19,"num_read_kb":8695,"num_write":14,"num_write_kb":2864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11,5,4,2,9,15],"acting":[13,11,5,4,2,9,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"3.d","version":"78'64","reported_seq":289,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-19T01:04:48.619389+0000","last_change":"2024-04-19T01:04:48.620146+0000","last_active":"2024-04-19T01:04:48.619389+0000","last_peered":"2024-04-19T01:04:48.619389+0000","last_clean":"2024-04-19T01:04:48.619389+0000","last_became_active":"2024-04-19T01:04:03.025363+0000","last_became_peered":"2024-04-19T01:04:03.025363+0000","last_unstale":"2024-04-19T01:04:48.619389+0000","last_undegraded":"2024-04-19T01:04:48.619389+0000","last_fullsized":"2024-04-19T01:04:48.619389+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:25.235325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0064189090000000004,"stat_sum":{"num_bytes":4751360,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":4327,"num_write":25,"num_write_kb":5728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,9,0,8,2,6],"acting":[12,4,9,0,8,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"13","length":"1"},{"start":"16","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"3.c","version":"79'49","reported_seq":283,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-19T01:04:50.662255+0000","last_change":"2024-04-19T01:04:49.633289+0000","last_active":"2024-04-19T01:04:50.662255+0000","last_peered":"2024-04-19T01:04:50.662255+0000","last_clean":"2024-04-19T01:04:50.662255+0000","last_became_active":"2024-04-19T01:04:03.025808+0000","last_became_peered":"2024-04-19T01:04:03.025808+0000","last_unstale":"2024-04-19T01:04:50.662255+0000","last_undegraded":"2024-04-19T01:04:50.662255+0000","last_fullsized":"2024-04-19T01:04:50.662255+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:14.427417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038213980000000002,"stat_sum":{"num_bytes":1425408,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":14,"num_read_kb":2453,"num_write":36,"num_write_kb":1904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,12,1,9,2,6],"acting":[0,10,12,1,9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"1a","length":"1"}]},{"pgid":"3.b","version":"74'42","reported_seq":270,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-19T01:04:47.734927+0000","last_change":"2024-04-19T01:04:44.576547+0000","last_active":"2024-04-19T01:04:47.734927+0000","last_peered":"2024-04-19T01:04:47.734927+0000","last_clean":"2024-04-19T01:04:47.734927+0000","last_became_active":"2024-04-19T01:04:02.781569+0000","last_became_peered":"2024-04-19T01:04:02.781569+0000","last_unstale":"2024-04-19T01:04:47.734927+0000","last_undegraded":"2024-04-19T01:04:47.734927+0000","last_fullsized":"2024-04-19T01:04:47.734927+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:18.208986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014763732,"stat_sum":{"num_bytes":5095424,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":5612,"num_write":37,"num_write_kb":4976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11,15,1,4,14],"acting":[10,6,11,15,1,4,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"13","length":"1"},{"start":"16","length":"1"}]},{"pgid":"3.a","version":"74'62","reported_seq":302,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-19T01:04:48.613735+0000","last_change":"2024-04-19T01:04:48.614256+0000","last_active":"2024-04-19T01:04:48.613735+0000","last_peered":"2024-04-19T01:04:48.613735+0000","last_clean":"2024-04-19T01:04:48.613735+0000","last_became_active":"2024-04-19T01:04:03.323177+0000","last_became_peered":"2024-04-19T01:04:03.323177+0000","last_unstale":"2024-04-19T01:04:48.613735+0000","last_undegraded":"2024-04-19T01:04:48.613735+0000","last_fullsized":"2024-04-19T01:04:48.613735+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:02.976107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070001000000000004,"stat_sum":{"num_bytes":10436608,"num_objects":9,"num_object_clones":5,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":30,"num_read_kb":5819,"num_write":26,"num_write_kb":11104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,7,3,13,1,8],"acting":[12,4,7,3,13,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"13","length":"1"},{"start":"16","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"3.9","version":"79'116","reported_seq":358,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-19T01:04:51.546040+0000","last_change":"2024-04-19T01:04:49.646289+0000","last_active":"2024-04-19T01:04:51.546040+0000","last_peered":"2024-04-19T01:04:51.546040+0000","last_clean":"2024-04-19T01:04:51.546040+0000","last_became_active":"2024-04-19T01:04:03.027372+0000","last_became_peered":"2024-04-19T01:04:03.027372+0000","last_unstale":"2024-04-19T01:04:51.546040+0000","last_undegraded":"2024-04-19T01:04:51.546040+0000","last_fullsized":"2024-04-19T01:04:51.546040+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:05.226287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013506852999999999,"stat_sum":{"num_bytes":7667712,"num_objects":7,"num_object_clones":1,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":28,"num_read_kb":638,"num_write":83,"num_write_kb":11712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11,14,9,4,10,6],"acting":[13,11,14,9,4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"3.8","version":"70'57","reported_seq":285,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-19T01:04:48.617351+0000","last_change":"2024-04-19T01:04:48.618160+0000","last_active":"2024-04-19T01:04:48.617351+0000","last_peered":"2024-04-19T01:04:48.617351+0000","last_clean":"2024-04-19T01:04:48.617351+0000","last_became_active":"2024-04-19T01:04:03.237653+0000","last_became_peered":"2024-04-19T01:04:03.237653+0000","last_unstale":"2024-04-19T01:04:48.617351+0000","last_undegraded":"2024-04-19T01:04:48.617351+0000","last_fullsized":"2024-04-19T01:04:48.617351+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:16.192850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013934150000000001,"stat_sum":{"num_bytes":5046272,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":14,"num_read_kb":1647,"num_write":41,"num_write_kb":5664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,1,15,14,0,5],"acting":[11,10,1,15,14,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"13","length":"1"},{"start":"16","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"1.0","version":"25'157","reported_seq":314,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-19T01:04:48.616694+0000","last_change":"2024-04-19T01:03:50.776344+0000","last_active":"2024-04-19T01:04:48.616694+0000","last_peered":"2024-04-19T01:04:48.616694+0000","last_clean":"2024-04-19T01:04:48.616694+0000","last_became_active":"2024-04-19T01:03:50.774078+0000","last_became_peered":"2024-04-19T01:03:50.774078+0000","last_unstale":"2024-04-19T01:04:48.616694+0000","last_undegraded":"2024-04-19T01:04:48.616694+0000","last_fullsized":"2024-04-19T01:04:48.616694+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:49.580851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:49.580851+0000","last_clean_scrub_stamp":"2024-04-19T01:03:49.580851+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:07.495976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":112,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-19T01:04:48.616879+0000","last_change":"2024-04-19T01:03:53.766587+0000","last_active":"2024-04-19T01:04:48.616879+0000","last_peered":"2024-04-19T01:04:48.616879+0000","last_clean":"2024-04-19T01:04:48.616879+0000","last_became_active":"2024-04-19T01:03:53.765967+0000","last_became_peered":"2024-04-19T01:03:53.765967+0000","last_unstale":"2024-04-19T01:04:48.616879+0000","last_undegraded":"2024-04-19T01:04:48.616879+0000","last_fullsized":"2024-04-19T01:04:48.616879+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:18.833542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"76'19","reported_seq":250,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-19T01:04:48.617584+0000","last_change":"2024-04-19T01:04:48.618589+0000","last_active":"2024-04-19T01:04:48.617584+0000","last_peered":"2024-04-19T01:04:48.617584+0000","last_clean":"2024-04-19T01:04:48.617584+0000","last_became_active":"2024-04-19T01:04:03.322437+0000","last_became_peered":"2024-04-19T01:04:03.322437+0000","last_unstale":"2024-04-19T01:04:48.617584+0000","last_undegraded":"2024-04-19T01:04:48.617584+0000","last_fullsized":"2024-04-19T01:04:48.617584+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:26.640927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001582647,"stat_sum":{"num_bytes":344064,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":9,"num_read_kb":531,"num_write":7,"num_write_kb":336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0,4,8,3,7],"acting":[11,5,0,4,8,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"13","length":"1"},{"start":"16","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":112,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-19T01:04:48.617918+0000","last_change":"2024-04-19T01:03:53.760075+0000","last_active":"2024-04-19T01:04:48.617918+0000","last_peered":"2024-04-19T01:04:48.617918+0000","last_clean":"2024-04-19T01:04:48.617918+0000","last_became_active":"2024-04-19T01:03:53.759326+0000","last_became_peered":"2024-04-19T01:03:53.759326+0000","last_unstale":"2024-04-19T01:04:48.617918+0000","last_undegraded":"2024-04-19T01:04:48.617918+0000","last_fullsized":"2024-04-19T01:04:48.617918+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:02.718012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"64'20","reported_seq":252,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-19T01:04:48.623495+0000","last_change":"2024-04-19T01:04:48.624421+0000","last_active":"2024-04-19T01:04:48.623495+0000","last_peered":"2024-04-19T01:04:48.623495+0000","last_clean":"2024-04-19T01:04:48.623495+0000","last_became_active":"2024-04-19T01:04:03.323727+0000","last_became_peered":"2024-04-19T01:04:03.323727+0000","last_unstale":"2024-04-19T01:04:48.623495+0000","last_undegraded":"2024-04-19T01:04:48.623495+0000","last_fullsized":"2024-04-19T01:04:48.623495+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:23.486341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001402599,"stat_sum":{"num_bytes":1392640,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":2020,"num_write":47,"num_write_kb":1488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,11,9,1,7],"acting":[6,4,3,11,9,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"13","length":"1"},{"start":"16","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":111,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-19T01:04:48.665566+0000","last_change":"2024-04-19T01:03:53.650434+0000","last_active":"2024-04-19T01:04:48.665566+0000","last_peered":"2024-04-19T01:04:48.665566+0000","last_clean":"2024-04-19T01:04:48.665566+0000","last_became_active":"2024-04-19T01:03:53.649606+0000","last_became_peered":"2024-04-19T01:03:53.649606+0000","last_unstale":"2024-04-19T01:04:48.665566+0000","last_undegraded":"2024-04-19T01:04:48.665566+0000","last_fullsized":"2024-04-19T01:04:48.665566+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:17.424729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"76'59","reported_seq":290,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-19T01:04:48.668356+0000","last_change":"2024-04-19T01:04:48.669367+0000","last_active":"2024-04-19T01:04:48.668356+0000","last_peered":"2024-04-19T01:04:48.668356+0000","last_clean":"2024-04-19T01:04:48.668356+0000","last_became_active":"2024-04-19T01:04:03.322503+0000","last_became_peered":"2024-04-19T01:04:03.322503+0000","last_unstale":"2024-04-19T01:04:48.668356+0000","last_undegraded":"2024-04-19T01:04:48.668356+0000","last_fullsized":"2024-04-19T01:04:48.668356+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:04.082408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015423360000000001,"stat_sum":{"num_bytes":5029888,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":19,"num_read_kb":10472,"num_write":24,"num_write_kb":6896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4,3,10,11,6],"acting":[1,9,4,3,10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"13","length":"1"},{"start":"16","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":112,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-19T01:04:48.618113+0000","last_change":"2024-04-19T01:03:54.012138+0000","last_active":"2024-04-19T01:04:48.618113+0000","last_peered":"2024-04-19T01:04:48.618113+0000","last_clean":"2024-04-19T01:04:48.618113+0000","last_became_active":"2024-04-19T01:03:54.011460+0000","last_became_peered":"2024-04-19T01:03:54.011460+0000","last_unstale":"2024-04-19T01:04:48.618113+0000","last_undegraded":"2024-04-19T01:04:48.618113+0000","last_fullsized":"2024-04-19T01:04:48.618113+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:04:58.336612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"57'49","reported_seq":279,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-19T01:04:48.666436+0000","last_change":"2024-04-19T01:04:48.666877+0000","last_active":"2024-04-19T01:04:48.666436+0000","last_peered":"2024-04-19T01:04:48.666436+0000","last_clean":"2024-04-19T01:04:48.666436+0000","last_became_active":"2024-04-19T01:04:02.835716+0000","last_became_peered":"2024-04-19T01:04:02.835716+0000","last_unstale":"2024-04-19T01:04:48.666436+0000","last_undegraded":"2024-04-19T01:04:48.666436+0000","last_fullsized":"2024-04-19T01:04:48.666436+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:17.532899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073056499999999997,"stat_sum":{"num_bytes":1064960,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":1879,"num_write":75,"num_write_kb":3744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2,0,1,4,12],"acting":[15,8,2,0,1,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"13","length":"1"},{"start":"16","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"3.5","version":"76'82","reported_seq":332,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-19T01:04:48.618760+0000","last_change":"2024-04-19T01:04:48.619113+0000","last_active":"2024-04-19T01:04:48.618760+0000","last_peered":"2024-04-19T01:04:48.618760+0000","last_clean":"2024-04-19T01:04:48.618760+0000","last_became_active":"2024-04-19T01:04:03.237886+0000","last_became_peered":"2024-04-19T01:04:03.237886+0000","last_unstale":"2024-04-19T01:04:48.618760+0000","last_undegraded":"2024-04-19T01:04:48.618760+0000","last_fullsized":"2024-04-19T01:04:48.618760+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:22.331915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081840700000000005,"stat_sum":{"num_bytes":10977280,"num_objects":12,"num_object_clones":5,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":51,"num_read_kb":11742,"num_write":80,"num_write_kb":12432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,14,11,10,2,5],"acting":[8,6,14,11,10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"13","length":"1"},{"start":"16","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":112,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-19T01:04:48.622873+0000","last_change":"2024-04-19T01:03:53.716606+0000","last_active":"2024-04-19T01:04:48.622873+0000","last_peered":"2024-04-19T01:04:48.622873+0000","last_clean":"2024-04-19T01:04:48.622873+0000","last_became_active":"2024-04-19T01:03:53.716040+0000","last_became_peered":"2024-04-19T01:03:53.716040+0000","last_unstale":"2024-04-19T01:04:48.622873+0000","last_undegraded":"2024-04-19T01:04:48.622873+0000","last_fullsized":"2024-04-19T01:04:48.622873+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:01.975515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"74'30","reported_seq":262,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-19T01:04:47.735144+0000","last_change":"2024-04-19T01:04:44.581680+0000","last_active":"2024-04-19T01:04:47.735144+0000","last_peered":"2024-04-19T01:04:47.735144+0000","last_clean":"2024-04-19T01:04:47.735144+0000","last_became_active":"2024-04-19T01:04:03.322815+0000","last_became_peered":"2024-04-19T01:04:03.322815+0000","last_unstale":"2024-04-19T01:04:47.735144+0000","last_undegraded":"2024-04-19T01:04:47.735144+0000","last_fullsized":"2024-04-19T01:04:47.735144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:31.159335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.019596985000000001,"stat_sum":{"num_bytes":3686400,"num_objects":6,"num_object_clones":2,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":24,"num_read_kb":1448,"num_write":14,"num_write_kb":3936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,13,14,3,15,1,5],"acting":[10,13,14,3,15,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"13","length":"1"},{"start":"16","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":115,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-19T01:04:50.733074+0000","last_change":"2024-04-19T01:03:53.772709+0000","last_active":"2024-04-19T01:04:50.733074+0000","last_peered":"2024-04-19T01:04:50.733074+0000","last_clean":"2024-04-19T01:04:50.733074+0000","last_became_active":"2024-04-19T01:03:53.771887+0000","last_became_peered":"2024-04-19T01:03:53.771887+0000","last_unstale":"2024-04-19T01:04:50.733074+0000","last_undegraded":"2024-04-19T01:04:50.733074+0000","last_fullsized":"2024-04-19T01:04:50.733074+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:17.479766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"67'21","reported_seq":266,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-19T01:04:51.542550+0000","last_change":"2024-04-19T01:04:49.639602+0000","last_active":"2024-04-19T01:04:51.542550+0000","last_peered":"2024-04-19T01:04:51.542550+0000","last_clean":"2024-04-19T01:04:51.542550+0000","last_became_active":"2024-04-19T01:04:03.238517+0000","last_became_peered":"2024-04-19T01:04:03.238517+0000","last_unstale":"2024-04-19T01:04:51.542550+0000","last_undegraded":"2024-04-19T01:04:51.542550+0000","last_fullsized":"2024-04-19T01:04:51.542550+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:11.944695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061145419999999997,"stat_sum":{"num_bytes":3080192,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":9028,"num_write":9,"num_write_kb":3008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,7,5,11,6,0],"acting":[14,13,7,5,11,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":118,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-19T01:04:51.541870+0000","last_change":"2024-04-19T01:03:53.736290+0000","last_active":"2024-04-19T01:04:51.541870+0000","last_peered":"2024-04-19T01:04:51.541870+0000","last_clean":"2024-04-19T01:04:51.541870+0000","last_became_active":"2024-04-19T01:03:53.735549+0000","last_became_peered":"2024-04-19T01:03:53.735549+0000","last_unstale":"2024-04-19T01:04:51.541870+0000","last_undegraded":"2024-04-19T01:04:51.541870+0000","last_fullsized":"2024-04-19T01:04:51.541870+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:07.640918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13],"acting":[14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":241,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-19T01:04:51.542400+0000","last_change":"2024-04-19T01:04:49.638631+0000","last_active":"2024-04-19T01:04:51.542400+0000","last_peered":"2024-04-19T01:04:51.542400+0000","last_clean":"2024-04-19T01:04:51.542400+0000","last_became_active":"2024-04-19T01:04:02.889974+0000","last_became_peered":"2024-04-19T01:04:02.889974+0000","last_unstale":"2024-04-19T01:04:51.542400+0000","last_undegraded":"2024-04-19T01:04:51.542400+0000","last_fullsized":"2024-04-19T01:04:51.542400+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:12.784715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004575717,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7,15,8,11,0,1],"acting":[14,7,15,8,11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":112,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-19T01:04:48.617927+0000","last_change":"2024-04-19T01:03:53.939441+0000","last_active":"2024-04-19T01:04:48.617927+0000","last_peered":"2024-04-19T01:04:48.617927+0000","last_clean":"2024-04-19T01:04:48.617927+0000","last_became_active":"2024-04-19T01:03:53.938774+0000","last_became_peered":"2024-04-19T01:03:53.938774+0000","last_unstale":"2024-04-19T01:04:48.617927+0000","last_undegraded":"2024-04-19T01:04:48.617927+0000","last_fullsized":"2024-04-19T01:04:48.617927+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:14.081709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":66174976,"num_objects":69,"num_object_clones":21,"num_object_copies":483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":7,"num_read":292,"num_read_kb":71599,"num_write":559,"num_write_kb":80160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":119201792,"data_stored":141426688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":753,"ondisk_log_size":753,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":13},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":15546,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":14,"up_from":24,"seq":103079215120,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45800,"kb_used_data":10456,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94326040,"statfs":{"total":96636764160,"available":96589864960,"internally_reserved":0,"allocated":10706944,"data_stored":12177328,"data_compressed":113117,"data_compressed_allocated":233472,"data_compressed_original":466944,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,5,6,7,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":13,"up_from":24,"seq":103079215121,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43868,"kb_used_data":8524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94327972,"statfs":{"total":96636764160,"available":96591843328,"internally_reserved":0,"allocated":8728576,"data_stored":9637808,"data_compressed":113117,"data_compressed_allocated":233472,"data_compressed_original":466944,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[2,4,5,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":24,"seq":103079215122,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41592,"kb_used_data":6248,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94330248,"statfs":{"total":96636764160,"available":96594173952,"internally_reserved":0,"allocated":6397952,"data_stored":6827952,"data_compressed":113117,"data_compressed_allocated":233472,"data_compressed_original":466944,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[1,2,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313232,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37360,"kb_used_data":10340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334480,"statfs":{"total":96636764160,"available":96598507520,"internally_reserved":0,"allocated":10588160,"data_stored":12422678,"data_compressed":105111,"data_compressed_allocated":217088,"data_compressed_original":434176,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,3,4,5,6,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280526,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37512,"kb_used_data":10484,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334328,"statfs":{"total":96636764160,"available":96598351872,"internally_reserved":0,"allocated":10735616,"data_stored":12574352,"data_compressed":109108,"data_compressed_allocated":225280,"data_compressed_original":450560,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280528,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37068,"kb_used_data":10052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334772,"statfs":{"total":96636764160,"available":96598806528,"internally_reserved":0,"allocated":10293248,"data_stored":11587216,"data_compressed":109108,"data_compressed_allocated":225280,"data_compressed_original":450560,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280527,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32988,"kb_used_data":5960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338852,"statfs":{"total":96636764160,"available":96602984448,"internally_reserved":0,"allocated":6103040,"data_stored":7028368,"data_compressed":109108,"data_compressed_allocated":225280,"data_compressed_original":450560,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[2,4,5,6,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280527,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37040,"kb_used_data":10012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334800,"statfs":{"total":96636764160,"available":96598835200,"internally_reserved":0,"allocated":10252288,"data_stored":11738768,"data_compressed":109108,"data_compressed_allocated":225280,"data_compressed_original":450560,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280527,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35096,"kb_used_data":8068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336744,"statfs":{"total":96636764160,"available":96600825856,"internally_reserved":0,"allocated":8261632,"data_stored":9523376,"data_compressed":116881,"data_compressed_allocated":258048,"data_compressed_original":962560,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,6,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280528,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34620,"kb_used_data":7592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337220,"statfs":{"total":96636764160,"available":96601313280,"internally_reserved":0,"allocated":7774208,"data_stored":9252496,"data_compressed":109108,"data_compressed_allocated":225280,"data_compressed_original":450560,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[2,5,6,7,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280526,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43040,"kb_used_data":11852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328800,"statfs":{"total":96636764160,"available":96592691200,"internally_reserved":0,"allocated":12136448,"data_stored":14442672,"data_compressed":116881,"data_compressed_allocated":258048,"data_compressed_original":962560,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,3,4,5,6,7,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280526,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32204,"kb_used_data":5176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339636,"statfs":{"total":96636764160,"available":96603787264,"internally_reserved":0,"allocated":5300224,"data_stored":5742224,"data_compressed":109108,"data_compressed_allocated":225280,"data_compressed_original":450560,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280527,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31928,"kb_used_data":4912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339912,"statfs":{"total":96636764160,"available":96604069888,"internally_reserved":0,"allocated":5029888,"data_stored":5148304,"data_compressed":109108,"data_compressed_allocated":225280,"data_compressed_original":450560,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,4,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":24,"seq":103079215120,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38924,"kb_used_data":7740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332916,"statfs":{"total":96636764160,"available":96596905984,"internally_reserved":0,"allocated":7925760,"data_stored":9519024,"data_compressed":113117,"data_compressed_allocated":233472,"data_compressed_original":466944,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215121,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37532,"kb_used_data":6348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334308,"statfs":{"total":96636764160,"available":96598331392,"internally_reserved":0,"allocated":6500352,"data_stored":6987696,"data_compressed":113117,"data_compressed_allocated":233472,"data_compressed_original":466944,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":24,"seq":103079215120,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36576,"kb_used_data":5392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335264,"statfs":{"total":96636764160,"available":96599310336,"internally_reserved":0,"allocated":5521408,"data_stored":6950832,"data_compressed":113117,"data_compressed_allocated":233472,"data_compressed_original":466944,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5595136,"data_stored":6336512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9412608,"data_stored":10956800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7020544,"data_stored":8867840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":5222400,"data_stored":6397952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":9371648,"data_stored":11108352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":7327744,"data_stored":8364032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":9854976,"data_stored":11943936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4616192,"data_stored":5529600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":6893568,"data_stored":8622080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":5492736,"data_stored":6176768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":9732096,"data_stored":11812864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":11202560,"data_stored":13283328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4419584,"data_stored":5111808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":8294400,"data_stored":9469952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":10596352,"data_stored":12926976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":4149248,"data_stored":4517888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T01:04:52.933 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T01:04:52.943 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:52.944 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:52.957 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-19T01:04:52.972 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:52.970+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:53.073 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:04:53.070+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:04:53.120 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:53.120 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:53.134 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-04-19T01:04:53.173 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:53.174+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:53.253 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:53.253 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:53.269 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-04-19T01:04:53.275 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:53.274+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:53.288 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:04:53.289 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-04-19T01:04:53.308 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":99,"stamp":"2024-04-19T01:04:52.520684+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":66765344,"num_objects":71,"num_object_clones":21,"num_object_copies":487,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":7,"num_read":428,"num_read_kb":71717,"num_write":762,"num_write_kb":83620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":910,"ondisk_log_size":910,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":603148,"kb_used_data":129156,"kb_used_omap":24,"kb_used_meta":473703,"kb_avail":1509346292,"statfs":{"total":1546188226560,"available":1545570603008,"internally_reserved":0,"allocated":132255744,"data_stored":151561094,"data_compressed":1781331,"data_compressed_allocated":3710976,"data_compressed_original":8314880,"omap_allocated":25520,"internal_metadata":485071952},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8732672,"num_objects":7,"num_object_clones":7,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":14,"num_read_kb":7843,"num_write":24,"num_write_kb":8528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.908540"},"pg_stats":[{"pgid":"3.f","version":"79'47","reported_seq":289,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-19T01:04:50.662599+0000","last_change":"2024-04-19T01:04:49.630730+0000","last_active":"2024-04-19T01:04:50.662599+0000","last_peered":"2024-04-19T01:04:50.662599+0000","last_clean":"2024-04-19T01:04:50.662599+0000","last_became_active":"2024-04-19T01:04:03.238651+0000","last_became_peered":"2024-04-19T01:04:03.238651+0000","last_unstale":"2024-04-19T01:04:50.662599+0000","last_undegraded":"2024-04-19T01:04:50.662599+0000","last_fullsized":"2024-04-19T01:04:50.662599+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:24.895505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081676800000000005,"stat_sum":{"num_bytes":4472832,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":5288,"num_write":41,"num_write_kb":4368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,5,11,7,1,2],"acting":[0,13,5,11,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"1a","length":"1"}]},{"pgid":"3.e","version":"70'36","reported_seq":285,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-19T01:04:51.546153+0000","last_change":"2024-04-19T01:04:49.634440+0000","last_active":"2024-04-19T01:04:51.546153+0000","last_peered":"2024-04-19T01:04:51.546153+0000","last_clean":"2024-04-19T01:04:51.546153+0000","last_became_active":"2024-04-19T01:04:03.238386+0000","last_became_peered":"2024-04-19T01:04:03.238386+0000","last_unstale":"2024-04-19T01:04:51.546153+0000","last_undegraded":"2024-04-19T01:04:51.546153+0000","last_fullsized":"2024-04-19T01:04:51.546153+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:22.119764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018389929999999999,"stat_sum":{"num_bytes":1703936,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":19,"num_read_kb":8695,"num_write":14,"num_write_kb":2864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11,5,4,2,9,15],"acting":[13,11,5,4,2,9,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"3.d","version":"78'64","reported_seq":289,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-19T01:04:48.619389+0000","last_change":"2024-04-19T01:04:48.620146+0000","last_active":"2024-04-19T01:04:48.619389+0000","last_peered":"2024-04-19T01:04:48.619389+0000","last_clean":"2024-04-19T01:04:48.619389+0000","last_became_active":"2024-04-19T01:04:03.025363+0000","last_became_peered":"2024-04-19T01:04:03.025363+0000","last_unstale":"2024-04-19T01:04:48.619389+0000","last_undegraded":"2024-04-19T01:04:48.619389+0000","last_fullsized":"2024-04-19T01:04:48.619389+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:25.235325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0064189090000000004,"stat_sum":{"num_bytes":4751360,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":4327,"num_write":25,"num_write_kb":5728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,9,0,8,2,6],"acting":[12,4,9,0,8,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"13","length":"1"},{"start":"16","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"3.c","version":"79'49","reported_seq":283,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-19T01:04:50.662255+0000","last_change":"2024-04-19T01:04:49.633289+0000","last_active":"2024-04-19T01:04:50.662255+0000","last_peered":"2024-04-19T01:04:50.662255+0000","last_clean":"2024-04-19T01:04:50.662255+0000","last_became_active":"2024-04-19T01:04:03.025808+0000","last_became_peered":"2024-04-19T01:04:03.025808+0000","last_unstale":"2024-04-19T01:04:50.662255+0000","last_undegraded":"2024-04-19T01:04:50.662255+0000","last_fullsized":"2024-04-19T01:04:50.662255+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:14.427417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038213980000000002,"stat_sum":{"num_bytes":1425408,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":14,"num_read_kb":2453,"num_write":36,"num_write_kb":1904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,12,1,9,2,6],"acting":[0,10,12,1,9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"1a","length":"1"}]},{"pgid":"3.b","version":"74'42","reported_seq":270,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-19T01:04:47.734927+0000","last_change":"2024-04-19T01:04:44.576547+0000","last_active":"2024-04-19T01:04:47.734927+0000","last_peered":"2024-04-19T01:04:47.734927+0000","last_clean":"2024-04-19T01:04:47.734927+0000","last_became_active":"2024-04-19T01:04:02.781569+0000","last_became_peered":"2024-04-19T01:04:02.781569+0000","last_unstale":"2024-04-19T01:04:47.734927+0000","last_undegraded":"2024-04-19T01:04:47.734927+0000","last_fullsized":"2024-04-19T01:04:47.734927+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:18.208986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014763732,"stat_sum":{"num_bytes":5095424,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":5612,"num_write":37,"num_write_kb":4976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11,15,1,4,14],"acting":[10,6,11,15,1,4,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"13","length":"1"},{"start":"16","length":"1"}]},{"pgid":"3.a","version":"74'62","reported_seq":302,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-19T01:04:48.613735+0000","last_change":"2024-04-19T01:04:48.614256+0000","last_active":"2024-04-19T01:04:48.613735+0000","last_peered":"2024-04-19T01:04:48.613735+0000","last_clean":"2024-04-19T01:04:48.613735+0000","last_became_active":"2024-04-19T01:04:03.323177+0000","last_became_peered":"2024-04-19T01:04:03.323177+0000","last_unstale":"2024-04-19T01:04:48.613735+0000","last_undegraded":"2024-04-19T01:04:48.613735+0000","last_fullsized":"2024-04-19T01:04:48.613735+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:02.976107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070001000000000004,"stat_sum":{"num_bytes":10436608,"num_objects":9,"num_object_clones":5,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":30,"num_read_kb":5819,"num_write":26,"num_write_kb":11104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,7,3,13,1,8],"acting":[12,4,7,3,13,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"13","length":"1"},{"start":"16","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"3.9","version":"79'116","reported_seq":358,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-19T01:04:51.546040+0000","last_change":"2024-04-19T01:04:49.646289+0000","last_active":"2024-04-19T01:04:51.546040+0000","last_peered":"2024-04-19T01:04:51.546040+0000","last_clean":"2024-04-19T01:04:51.546040+0000","last_became_active":"2024-04-19T01:04:03.027372+0000","last_became_peered":"2024-04-19T01:04:03.027372+0000","last_unstale":"2024-04-19T01:04:51.546040+0000","last_undegraded":"2024-04-19T01:04:51.546040+0000","last_fullsized":"2024-04-19T01:04:51.546040+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:05.226287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013506852999999999,"stat_sum":{"num_bytes":7667712,"num_objects":7,"num_object_clones":1,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":28,"num_read_kb":638,"num_write":83,"num_write_kb":11712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11,14,9,4,10,6],"acting":[13,11,14,9,4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"3.8","version":"70'57","reported_seq":285,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-19T01:04:48.617351+0000","last_change":"2024-04-19T01:04:48.618160+0000","last_active":"2024-04-19T01:04:48.617351+0000","last_peered":"2024-04-19T01:04:48.617351+0000","last_clean":"2024-04-19T01:04:48.617351+0000","last_became_active":"2024-04-19T01:04:03.237653+0000","last_became_peered":"2024-04-19T01:04:03.237653+0000","last_unstale":"2024-04-19T01:04:48.617351+0000","last_undegraded":"2024-04-19T01:04:48.617351+0000","last_fullsized":"2024-04-19T01:04:48.617351+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:16.192850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013934150000000001,"stat_sum":{"num_bytes":5046272,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":14,"num_read_kb":1647,"num_write":41,"num_write_kb":5664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,1,15,14,0,5],"acting":[11,10,1,15,14,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"13","length":"1"},{"start":"16","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"1.0","version":"25'157","reported_seq":314,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-19T01:04:48.616694+0000","last_change":"2024-04-19T01:03:50.776344+0000","last_active":"2024-04-19T01:04:48.616694+0000","last_peered":"2024-04-19T01:04:48.616694+0000","last_clean":"2024-04-19T01:04:48.616694+0000","last_became_active":"2024-04-19T01:03:50.774078+0000","last_became_peered":"2024-04-19T01:03:50.774078+0000","last_unstale":"2024-04-19T01:04:48.616694+0000","last_undegraded":"2024-04-19T01:04:48.616694+0000","last_fullsized":"2024-04-19T01:04:48.616694+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:49.580851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:49.580851+0000","last_clean_scrub_stamp":"2024-04-19T01:03:49.580851+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:07.495976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":112,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-19T01:04:48.616879+0000","last_change":"2024-04-19T01:03:53.766587+0000","last_active":"2024-04-19T01:04:48.616879+0000","last_peered":"2024-04-19T01:04:48.616879+0000","last_clean":"2024-04-19T01:04:48.616879+0000","last_became_active":"2024-04-19T01:03:53.765967+0000","last_became_peered":"2024-04-19T01:03:53.765967+0000","last_unstale":"2024-04-19T01:04:48.616879+0000","last_undegraded":"2024-04-19T01:04:48.616879+0000","last_fullsized":"2024-04-19T01:04:48.616879+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:18.833542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"76'19","reported_seq":250,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-19T01:04:48.617584+0000","last_change":"2024-04-19T01:04:48.618589+0000","last_active":"2024-04-19T01:04:48.617584+0000","last_peered":"2024-04-19T01:04:48.617584+0000","last_clean":"2024-04-19T01:04:48.617584+0000","last_became_active":"2024-04-19T01:04:03.322437+0000","last_became_peered":"2024-04-19T01:04:03.322437+0000","last_unstale":"2024-04-19T01:04:48.617584+0000","last_undegraded":"2024-04-19T01:04:48.617584+0000","last_fullsized":"2024-04-19T01:04:48.617584+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:26.640927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001582647,"stat_sum":{"num_bytes":344064,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":9,"num_read_kb":531,"num_write":7,"num_write_kb":336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0,4,8,3,7],"acting":[11,5,0,4,8,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"13","length":"1"},{"start":"16","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":112,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-19T01:04:48.617918+0000","last_change":"2024-04-19T01:03:53.760075+0000","last_active":"2024-04-19T01:04:48.617918+0000","last_peered":"2024-04-19T01:04:48.617918+0000","last_clean":"2024-04-19T01:04:48.617918+0000","last_became_active":"2024-04-19T01:03:53.759326+0000","last_became_peered":"2024-04-19T01:03:53.759326+0000","last_unstale":"2024-04-19T01:04:48.617918+0000","last_undegraded":"2024-04-19T01:04:48.617918+0000","last_fullsized":"2024-04-19T01:04:48.617918+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:02.718012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"64'20","reported_seq":252,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-19T01:04:48.623495+0000","last_change":"2024-04-19T01:04:48.624421+0000","last_active":"2024-04-19T01:04:48.623495+0000","last_peered":"2024-04-19T01:04:48.623495+0000","last_clean":"2024-04-19T01:04:48.623495+0000","last_became_active":"2024-04-19T01:04:03.323727+0000","last_became_peered":"2024-04-19T01:04:03.323727+0000","last_unstale":"2024-04-19T01:04:48.623495+0000","last_undegraded":"2024-04-19T01:04:48.623495+0000","last_fullsized":"2024-04-19T01:04:48.623495+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:23.486341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001402599,"stat_sum":{"num_bytes":1392640,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":2020,"num_write":47,"num_write_kb":1488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,11,9,1,7],"acting":[6,4,3,11,9,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"13","length":"1"},{"start":"16","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":111,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-19T01:04:48.665566+0000","last_change":"2024-04-19T01:03:53.650434+0000","last_active":"2024-04-19T01:04:48.665566+0000","last_peered":"2024-04-19T01:04:48.665566+0000","last_clean":"2024-04-19T01:04:48.665566+0000","last_became_active":"2024-04-19T01:03:53.649606+0000","last_became_peered":"2024-04-19T01:03:53.649606+0000","last_unstale":"2024-04-19T01:04:48.665566+0000","last_undegraded":"2024-04-19T01:04:48.665566+0000","last_fullsized":"2024-04-19T01:04:48.665566+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:17.424729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"76'59","reported_seq":290,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-19T01:04:48.668356+0000","last_change":"2024-04-19T01:04:48.669367+0000","last_active":"2024-04-19T01:04:48.668356+0000","last_peered":"2024-04-19T01:04:48.668356+0000","last_clean":"2024-04-19T01:04:48.668356+0000","last_became_active":"2024-04-19T01:04:03.322503+0000","last_became_peered":"2024-04-19T01:04:03.322503+0000","last_unstale":"2024-04-19T01:04:48.668356+0000","last_undegraded":"2024-04-19T01:04:48.668356+0000","last_fullsized":"2024-04-19T01:04:48.668356+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:04.082408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015423360000000001,"stat_sum":{"num_bytes":5029888,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":19,"num_read_kb":10472,"num_write":24,"num_write_kb":6896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4,3,10,11,6],"acting":[1,9,4,3,10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"13","length":"1"},{"start":"16","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":112,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-19T01:04:48.618113+0000","last_change":"2024-04-19T01:03:54.012138+0000","last_active":"2024-04-19T01:04:48.618113+0000","last_peered":"2024-04-19T01:04:48.618113+0000","last_clean":"2024-04-19T01:04:48.618113+0000","last_became_active":"2024-04-19T01:03:54.011460+0000","last_became_peered":"2024-04-19T01:03:54.011460+0000","last_unstale":"2024-04-19T01:04:48.618113+0000","last_undegraded":"2024-04-19T01:04:48.618113+0000","last_fullsized":"2024-04-19T01:04:48.618113+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:04:58.336612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"57'49","reported_seq":279,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-19T01:04:48.666436+0000","last_change":"2024-04-19T01:04:48.666877+0000","last_active":"2024-04-19T01:04:48.666436+0000","last_peered":"2024-04-19T01:04:48.666436+0000","last_clean":"2024-04-19T01:04:48.666436+0000","last_became_active":"2024-04-19T01:04:02.835716+0000","last_became_peered":"2024-04-19T01:04:02.835716+0000","last_unstale":"2024-04-19T01:04:48.666436+0000","last_undegraded":"2024-04-19T01:04:48.666436+0000","last_fullsized":"2024-04-19T01:04:48.666436+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:17.532899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073056499999999997,"stat_sum":{"num_bytes":1064960,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":1879,"num_write":75,"num_write_kb":3744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2,0,1,4,12],"acting":[15,8,2,0,1,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"13","length":"1"},{"start":"16","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"3.5","version":"76'82","reported_seq":332,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-19T01:04:48.618760+0000","last_change":"2024-04-19T01:04:48.619113+0000","last_active":"2024-04-19T01:04:48.618760+0000","last_peered":"2024-04-19T01:04:48.618760+0000","last_clean":"2024-04-19T01:04:48.618760+0000","last_became_active":"2024-04-19T01:04:03.237886+0000","last_became_peered":"2024-04-19T01:04:03.237886+0000","last_unstale":"2024-04-19T01:04:48.618760+0000","last_undegraded":"2024-04-19T01:04:48.618760+0000","last_fullsized":"2024-04-19T01:04:48.618760+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:22.331915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081840700000000005,"stat_sum":{"num_bytes":10977280,"num_objects":12,"num_object_clones":5,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":51,"num_read_kb":11742,"num_write":80,"num_write_kb":12432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,14,11,10,2,5],"acting":[8,6,14,11,10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"13","length":"1"},{"start":"16","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":112,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-19T01:04:48.622873+0000","last_change":"2024-04-19T01:03:53.716606+0000","last_active":"2024-04-19T01:04:48.622873+0000","last_peered":"2024-04-19T01:04:48.622873+0000","last_clean":"2024-04-19T01:04:48.622873+0000","last_became_active":"2024-04-19T01:03:53.716040+0000","last_became_peered":"2024-04-19T01:03:53.716040+0000","last_unstale":"2024-04-19T01:04:48.622873+0000","last_undegraded":"2024-04-19T01:04:48.622873+0000","last_fullsized":"2024-04-19T01:04:48.622873+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:01.975515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"74'30","reported_seq":262,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-19T01:04:47.735144+0000","last_change":"2024-04-19T01:04:44.581680+0000","last_active":"2024-04-19T01:04:47.735144+0000","last_peered":"2024-04-19T01:04:47.735144+0000","last_clean":"2024-04-19T01:04:47.735144+0000","last_became_active":"2024-04-19T01:04:03.322815+0000","last_became_peered":"2024-04-19T01:04:03.322815+0000","last_unstale":"2024-04-19T01:04:47.735144+0000","last_undegraded":"2024-04-19T01:04:47.735144+0000","last_fullsized":"2024-04-19T01:04:47.735144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:31.159335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.019596985000000001,"stat_sum":{"num_bytes":3686400,"num_objects":6,"num_object_clones":2,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":24,"num_read_kb":1448,"num_write":14,"num_write_kb":3936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,13,14,3,15,1,5],"acting":[10,13,14,3,15,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"13","length":"1"},{"start":"16","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":115,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-19T01:04:50.733074+0000","last_change":"2024-04-19T01:03:53.772709+0000","last_active":"2024-04-19T01:04:50.733074+0000","last_peered":"2024-04-19T01:04:50.733074+0000","last_clean":"2024-04-19T01:04:50.733074+0000","last_became_active":"2024-04-19T01:03:53.771887+0000","last_became_peered":"2024-04-19T01:03:53.771887+0000","last_unstale":"2024-04-19T01:04:50.733074+0000","last_undegraded":"2024-04-19T01:04:50.733074+0000","last_fullsized":"2024-04-19T01:04:50.733074+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:17.479766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"67'21","reported_seq":266,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-19T01:04:51.542550+0000","last_change":"2024-04-19T01:04:49.639602+0000","last_active":"2024-04-19T01:04:51.542550+0000","last_peered":"2024-04-19T01:04:51.542550+0000","last_clean":"2024-04-19T01:04:51.542550+0000","last_became_active":"2024-04-19T01:04:03.238517+0000","last_became_peered":"2024-04-19T01:04:03.238517+0000","last_unstale":"2024-04-19T01:04:51.542550+0000","last_undegraded":"2024-04-19T01:04:51.542550+0000","last_fullsized":"2024-04-19T01:04:51.542550+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:11.944695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061145419999999997,"stat_sum":{"num_bytes":3080192,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":9028,"num_write":9,"num_write_kb":3008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,7,5,11,6,0],"acting":[14,13,7,5,11,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":118,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-19T01:04:51.541870+0000","last_change":"2024-04-19T01:03:53.736290+0000","last_active":"2024-04-19T01:04:51.541870+0000","last_peered":"2024-04-19T01:04:51.541870+0000","last_clean":"2024-04-19T01:04:51.541870+0000","last_became_active":"2024-04-19T01:03:53.735549+0000","last_became_peered":"2024-04-19T01:03:53.735549+0000","last_unstale":"2024-04-19T01:04:51.541870+0000","last_undegraded":"2024-04-19T01:04:51.541870+0000","last_fullsized":"2024-04-19T01:04:51.541870+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:07.640918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13],"acting":[14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":241,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-19T01:04:51.542400+0000","last_change":"2024-04-19T01:04:49.638631+0000","last_active":"2024-04-19T01:04:51.542400+0000","last_peered":"2024-04-19T01:04:51.542400+0000","last_clean":"2024-04-19T01:04:51.542400+0000","last_became_active":"2024-04-19T01:04:02.889974+0000","last_became_peered":"2024-04-19T01:04:02.889974+0000","last_unstale":"2024-04-19T01:04:51.542400+0000","last_undegraded":"2024-04-19T01:04:51.542400+0000","last_fullsized":"2024-04-19T01:04:51.542400+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:12.784715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004575717,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7,15,8,11,0,1],"acting":[14,7,15,8,11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":112,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-19T01:04:48.617927+0000","last_change":"2024-04-19T01:03:53.939441+0000","last_active":"2024-04-19T01:04:48.617927+0000","last_peered":"2024-04-19T01:04:48.617927+0000","last_clean":"2024-04-19T01:04:48.617927+0000","last_became_active":"2024-04-19T01:03:53.938774+0000","last_became_peered":"2024-04-19T01:03:53.938774+0000","last_unstale":"2024-04-19T01:04:48.617927+0000","last_undegraded":"2024-04-19T01:04:48.617927+0000","last_fullsized":"2024-04-19T01:04:48.617927+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:14.081709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":66174976,"num_objects":69,"num_object_clones":21,"num_object_copies":483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":7,"num_read":292,"num_read_kb":71599,"num_write":559,"num_write_kb":80160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":119201792,"data_stored":141426688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":753,"ondisk_log_size":753,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":13},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":15546,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":14,"up_from":24,"seq":103079215120,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45800,"kb_used_data":10456,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94326040,"statfs":{"total":96636764160,"available":96589864960,"internally_reserved":0,"allocated":10706944,"data_stored":12177328,"data_compressed":113117,"data_compressed_allocated":233472,"data_compressed_original":466944,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,5,6,7,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":13,"up_from":24,"seq":103079215121,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43868,"kb_used_data":8524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94327972,"statfs":{"total":96636764160,"available":96591843328,"internally_reserved":0,"allocated":8728576,"data_stored":9637808,"data_compressed":113117,"data_compressed_allocated":233472,"data_compressed_original":466944,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[2,4,5,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":24,"seq":103079215122,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41592,"kb_used_data":6248,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94330248,"statfs":{"total":96636764160,"available":96594173952,"internally_reserved":0,"allocated":6397952,"data_stored":6827952,"data_compressed":113117,"data_compressed_allocated":233472,"data_compressed_original":466944,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[1,2,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313232,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37360,"kb_used_data":10340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334480,"statfs":{"total":96636764160,"available":96598507520,"internally_reserved":0,"allocated":10588160,"data_stored":12422678,"data_compressed":105111,"data_compressed_allocated":217088,"data_compressed_original":434176,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,3,4,5,6,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280526,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37512,"kb_used_data":10484,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334328,"statfs":{"total":96636764160,"available":96598351872,"internally_reserved":0,"allocated":10735616,"data_stored":12574352,"data_compressed":109108,"data_compressed_allocated":225280,"data_compressed_original":450560,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280528,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37068,"kb_used_data":10052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334772,"statfs":{"total":96636764160,"available":96598806528,"internally_reserved":0,"allocated":10293248,"data_stored":11587216,"data_compressed":109108,"data_compressed_allocated":225280,"data_compressed_original":450560,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280527,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32988,"kb_used_data":5960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338852,"statfs":{"total":96636764160,"available":96602984448,"internally_reserved":0,"allocated":6103040,"data_stored":7028368,"data_compressed":109108,"data_compressed_allocated":225280,"data_compressed_original":450560,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[2,4,5,6,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280527,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37040,"kb_used_data":10012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334800,"statfs":{"total":96636764160,"available":96598835200,"internally_reserved":0,"allocated":10252288,"data_stored":11738768,"data_compressed":109108,"data_compressed_allocated":225280,"data_compressed_original":450560,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280527,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35096,"kb_used_data":8068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336744,"statfs":{"total":96636764160,"available":96600825856,"internally_reserved":0,"allocated":8261632,"data_stored":9523376,"data_compressed":116881,"data_compressed_allocated":258048,"data_compressed_original":962560,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,6,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280528,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34620,"kb_used_data":7592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337220,"statfs":{"total":96636764160,"available":96601313280,"internally_reserved":0,"allocated":7774208,"data_stored":9252496,"data_compressed":109108,"data_compressed_allocated":225280,"data_compressed_original":450560,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[2,5,6,7,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280526,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43040,"kb_used_data":11852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328800,"statfs":{"total":96636764160,"available":96592691200,"internally_reserved":0,"allocated":12136448,"data_stored":14442672,"data_compressed":116881,"data_compressed_allocated":258048,"data_compressed_original":962560,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,3,4,5,6,7,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280526,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32204,"kb_used_data":5176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339636,"statfs":{"total":96636764160,"available":96603787264,"internally_reserved":0,"allocated":5300224,"data_stored":5742224,"data_compressed":109108,"data_compressed_allocated":225280,"data_compressed_original":450560,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280527,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31928,"kb_used_data":4912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339912,"statfs":{"total":96636764160,"available":96604069888,"internally_reserved":0,"allocated":5029888,"data_stored":5148304,"data_compressed":109108,"data_compressed_allocated":225280,"data_compressed_original":450560,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,4,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":24,"seq":103079215120,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38924,"kb_used_data":7740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332916,"statfs":{"total":96636764160,"available":96596905984,"internally_reserved":0,"allocated":7925760,"data_stored":9519024,"data_compressed":113117,"data_compressed_allocated":233472,"data_compressed_original":466944,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215121,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37532,"kb_used_data":6348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334308,"statfs":{"total":96636764160,"available":96598331392,"internally_reserved":0,"allocated":6500352,"data_stored":6987696,"data_compressed":113117,"data_compressed_allocated":233472,"data_compressed_original":466944,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":24,"seq":103079215120,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36576,"kb_used_data":5392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335264,"statfs":{"total":96636764160,"available":96599310336,"internally_reserved":0,"allocated":5521408,"data_stored":6950832,"data_compressed":113117,"data_compressed_allocated":233472,"data_compressed_original":466944,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5595136,"data_stored":6336512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9412608,"data_stored":10956800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7020544,"data_stored":8867840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":5222400,"data_stored":6397952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":9371648,"data_stored":11108352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":7327744,"data_stored":8364032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":9854976,"data_stored":11943936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4616192,"data_stored":5529600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":6893568,"data_stored":8622080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":5492736,"data_stored":6176768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":9732096,"data_stored":11812864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":11202560,"data_stored":13283328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4419584,"data_stored":5111808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":8294400,"data_stored":9469952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":10596352,"data_stored":12926976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":4149248,"data_stored":4517888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T01:04:53.310 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-19T01:04:53.310 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-19T01:04:53.310 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-04-19T01:04:53.310 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-19T01:04:53.310 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-19T01:04:53.311 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3 2024-04-19T01:04:53.377 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:04:53.375+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:04:53.385 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:53.385 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:53.399 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-04-19T01:04:53.439 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:04:53.439 INFO:teuthology.orchestra.run.smithi110.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-19T01:04:53.439 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:04:53.478 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:04:53.476+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:04:53.518 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2024-04-19T01:04:53.527 INFO:tasks.rados.rados.0.smithi155.stdout:264: done (1 left) 2024-04-19T01:04:53.527 INFO:tasks.rados.rados.0.smithi155.stdout:265: done (0 left) 2024-04-19T01:04:53.527 INFO:tasks.rados.rados.0.smithi155.stdout:266: rollback oid 5 current snap is 28 2024-04-19T01:04:53.528 INFO:tasks.rados.rados.0.smithi155.stdout:rollback oid 5 to 18 2024-04-19T01:04:53.528 INFO:tasks.rados.rados.0.smithi155.stdout:267: snap_remove snap 16 2024-04-19T01:04:53.566 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:53.567 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:53.579 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:04:53.578+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:04:53.581 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-19T01:04:53.681 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:04:53.678+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:04:53.711 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:53.712 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:53.726 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-19T01:04:53.782 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:04:53.778+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:04:53.849 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:53.849 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:53.864 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-19T01:04:53.883 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:53.883+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:53.983 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:53.983 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:53.984 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:04:53.982+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:04:53.997 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-19T01:04:54.077 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-19T01:04:54.084 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:54.086+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:54.120 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:54.120 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:54.136 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-19T01:04:54.187 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:04:54.186+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:04:54.239 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-04-19T01:04:54.239 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T01:04:54.258 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:54.258 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:54.260 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-04-19T01:04:54.260 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T01:04:54.273 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-19T01:04:54.283 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-04-19T01:04:54.283 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T01:04:54.288 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:04:54.287+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:04:54.305 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-04-19T01:04:54.306 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T01:04:54.328 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-04-19T01:04:54.328 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T01:04:54.353 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-04-19T01:04:54.353 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T01:04:54.376 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-04-19T01:04:54.377 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T01:04:54.391 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:54.391 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:54.392 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:04:54.390+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:04:54.403 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-04-19T01:04:54.403 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T01:04:54.407 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-19T01:04:54.422 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2024-04-19T01:04:54.423 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T01:04:54.447 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2024-04-19T01:04:54.448 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T01:04:54.472 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2024-04-19T01:04:54.473 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T01:04:54.492 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:04:54.490+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:04:54.496 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2024-04-19T01:04:54.497 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T01:04:54.519 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2024-04-19T01:04:54.519 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-19T01:04:54.521 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:04:54.521 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:04:54.521 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-04-19T01:04:54.522 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:04:54.533 INFO:tasks.rados.rados.0.smithi155.stdout:267: done (1 left) 2024-04-19T01:04:54.533 INFO:tasks.rados.rados.0.smithi155.stdout:268: snap_remove snap 21 2024-04-19T01:04:54.536 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-19T01:04:54.552 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2024-04-19T01:04:54.552 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-19T01:04:54.595 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:04:54.595+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:04:54.653 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:04:54.653 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:04:54.653 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-04-19T01:04:54.653 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-04-19T01:04:54.653 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:04:54.665 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-19T01:04:54.696 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:04:54.694+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:04:54.780 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2024-04-19T01:04:54.781 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-19T01:04:54.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:04:54.782+0000 7f5daf59e640 0 -- 172.21.15.33:0/1814977150 >> [v2:172.21.15.155:6807/741132602,v1:172.21.15.155:6810/741132602] conn(0x7f5db0226c80 msgr2=0x7f5db0229070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-19T01:04:54.789 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:04:54.789 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-04-19T01:04:54.789 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-04-19T01:04:54.789 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-04-19T01:04:54.789 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.789 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4260.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-19T01:04:54.789 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:03:50.620310+0000", 2024-04-19T01:04:54.789 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 64.167404547999993, 2024-04-19T01:04:54.789 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.15782495899999999, 2024-04-19T01:04:54.789 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:54.790 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:54.790 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:54.790 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:54.790 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4260", 2024-04-19T01:04:54.790 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/1127060541", 2024-04-19T01:04:54.790 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1 2024-04-19T01:04:54.790 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.790 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:54.790 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.790 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:54.790 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.620310+0000", 2024-04-19T01:04:54.790 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:54.790 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.790 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.791 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:54.791 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.620310+0000", 2024-04-19T01:04:54.791 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:54.791 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.791 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.791 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:54.791 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.620313+0000", 2024-04-19T01:04:54.791 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.757e-06 2024-04-19T01:04:54.791 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.791 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.791 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:54.791 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.620327+0000", 2024-04-19T01:04:54.791 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4029000000000001e-05 2024-04-19T01:04:54.791 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.791 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.792 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:54.792 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.620362+0000", 2024-04-19T01:04:54.792 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5201999999999999e-05 2024-04-19T01:04:54.792 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.792 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.792 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:54.792 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.620383+0000", 2024-04-19T01:04:54.792 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1049000000000001e-05 2024-04-19T01:04:54.792 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.792 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.792 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:54.792 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.620512+0000", 2024-04-19T01:04:54.792 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012820900000000001 2024-04-19T01:04:54.792 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.792 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.793 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for peered", 2024-04-19T01:04:54.793 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.620542+0000", 2024-04-19T01:04:54.793 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0264999999999999e-05 2024-04-19T01:04:54.793 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.793 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.793 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:54.793 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.774580+0000", 2024-04-19T01:04:54.793 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.15403776799999999 2024-04-19T01:04:54.793 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.793 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.793 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:54.793 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.775052+0000", 2024-04-19T01:04:54.793 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00047277300000000001 2024-04-19T01:04:54.793 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.793 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.794 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 5", 2024-04-19T01:04:54.794 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.775495+0000", 2024-04-19T01:04:54.794 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00044209400000000003 2024-04-19T01:04:54.794 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.794 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.794 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-04-19T01:04:54.794 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.777608+0000", 2024-04-19T01:04:54.798 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0021133649999999999 2024-04-19T01:04:54.798 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.798 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.798 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-04-19T01:04:54.798 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.777957+0000", 2024-04-19T01:04:54.799 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00034961399999999999 2024-04-19T01:04:54.799 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.799 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.799 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:54.799 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.778062+0000", 2024-04-19T01:04:54.799 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000104576 2024-04-19T01:04:54.799 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.799 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.799 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:54.799 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.778135+0000", 2024-04-19T01:04:54.799 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3257999999999996e-05 2024-04-19T01:04:54.799 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:54.799 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:54.799 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:54.800 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.800 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.800 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs3 48/31 ECSubWrite(tid=76, reqid=client.4604.0:345, at_version=48'22, trim_to=0'0, roll_forward_to=47'19))", 2024-04-19T01:04:54.800 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.537801+0000", 2024-04-19T01:04:54.800 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 36.249913909999997, 2024-04-19T01:04:54.800 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.028349493, 2024-04-19T01:04:54.800 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:54.800 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:54.800 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:54.800 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:54.800 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.800 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:54.800 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.537801+0000", 2024-04-19T01:04:54.800 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:54.800 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.801 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.801 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:54.801 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.537801+0000", 2024-04-19T01:04:54.801 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:54.801 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.801 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.801 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:54.801 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.537803+0000", 2024-04-19T01:04:54.801 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7099999999999999e-06 2024-04-19T01:04:54.801 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.801 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.801 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:54.801 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.538304+0000", 2024-04-19T01:04:54.801 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00050171999999999999 2024-04-19T01:04:54.802 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.802 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.802 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:54.802 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.538333+0000", 2024-04-19T01:04:54.802 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8725999999999999e-05 2024-04-19T01:04:54.802 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.802 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.802 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:54.802 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.538354+0000", 2024-04-19T01:04:54.802 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.035e-05 2024-04-19T01:04:54.802 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.802 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.803 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:54.803 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.538557+0000", 2024-04-19T01:04:54.803 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000203766 2024-04-19T01:04:54.803 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.803 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.803 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:54.803 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.538579+0000", 2024-04-19T01:04:54.803 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1792000000000001e-05 2024-04-19T01:04:54.803 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.803 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.803 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:54.803 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.566117+0000", 2024-04-19T01:04:54.803 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.027538045000000001 2024-04-19T01:04:54.803 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.803 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.804 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:54.804 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.566150+0000", 2024-04-19T01:04:54.804 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3383999999999997e-05 2024-04-19T01:04:54.804 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:54.804 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:54.804 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:54.804 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.804 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.804 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:360 3.8s0 3:11aca45f:::smithi15528762-45:head [append 0~573440] snapc 9=[7] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-04-19T01:04:54.804 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:19.532000+0000", 2024-04-19T01:04:54.804 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 35.255714996999998, 2024-04-19T01:04:54.804 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.042676382999999998, 2024-04-19T01:04:54.804 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:54.805 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:54.805 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:54.805 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:54.805 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:04:54.805 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:54.805 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 360 2024-04-19T01:04:54.805 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.805 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:54.805 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.805 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:54.805 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.532000+0000", 2024-04-19T01:04:54.806 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:54.806 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.806 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.806 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:54.806 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.532000+0000", 2024-04-19T01:04:54.806 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:54.806 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.806 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.806 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:54.806 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.532003+0000", 2024-04-19T01:04:54.806 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3189999999999999e-06 2024-04-19T01:04:54.806 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.806 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.806 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:54.807 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.533430+0000", 2024-04-19T01:04:54.807 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001427069 2024-04-19T01:04:54.807 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.807 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.807 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:54.807 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.533464+0000", 2024-04-19T01:04:54.807 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3708999999999999e-05 2024-04-19T01:04:54.807 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.807 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.807 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:54.807 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.533485+0000", 2024-04-19T01:04:54.807 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0596000000000001e-05 2024-04-19T01:04:54.807 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.807 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.808 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:54.808 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.535443+0000", 2024-04-19T01:04:54.808 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0019587290000000002 2024-04-19T01:04:54.808 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.808 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.808 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:54.808 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.535618+0000", 2024-04-19T01:04:54.808 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000174518 2024-04-19T01:04:54.808 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.808 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.808 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:54.808 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.537401+0000", 2024-04-19T01:04:54.808 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0017834350000000001 2024-04-19T01:04:54.808 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.809 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.809 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:54.809 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.550350+0000", 2024-04-19T01:04:54.809 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.012948602 2024-04-19T01:04:54.809 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.809 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.809 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:54.809 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.574521+0000", 2024-04-19T01:04:54.810 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.024171392999999999 2024-04-19T01:04:54.810 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.810 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.810 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:54.810 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.574676+0000", 2024-04-19T01:04:54.810 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015501299999999999 2024-04-19T01:04:54.810 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:54.810 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:54.810 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:54.810 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.810 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.810 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:361 3.8s0 3:11aca45f:::smithi15528762-45:head [append 0~425984] snapc 9=[7] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-04-19T01:04:54.810 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:19.540243+0000", 2024-04-19T01:04:54.811 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 35.24747163, 2024-04-19T01:04:54.811 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.038864792000000002, 2024-04-19T01:04:54.811 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:54.811 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:54.811 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:54.811 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:54.811 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:04:54.811 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:54.811 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 361 2024-04-19T01:04:54.811 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.811 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:54.811 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.811 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:54.811 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540243+0000", 2024-04-19T01:04:54.811 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:54.812 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.812 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.812 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:54.812 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540243+0000", 2024-04-19T01:04:54.812 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:54.812 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.812 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.812 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:54.812 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540247+0000", 2024-04-19T01:04:54.812 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.506e-06 2024-04-19T01:04:54.812 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.812 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.812 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:54.812 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.541191+0000", 2024-04-19T01:04:54.812 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00094451099999999998 2024-04-19T01:04:54.813 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.813 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.813 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:54.813 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.541227+0000", 2024-04-19T01:04:54.813 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6053000000000002e-05 2024-04-19T01:04:54.813 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.813 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.813 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:54.813 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.541250+0000", 2024-04-19T01:04:54.813 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2543000000000001e-05 2024-04-19T01:04:54.813 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.813 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.813 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:54.813 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.550582+0000", 2024-04-19T01:04:54.813 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0093325990000000004 2024-04-19T01:04:54.814 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.814 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.814 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:54.814 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.550838+0000", 2024-04-19T01:04:54.814 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00025546199999999997 2024-04-19T01:04:54.814 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.814 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.814 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:54.814 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.552905+0000", 2024-04-19T01:04:54.814 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0020670699999999998 2024-04-19T01:04:54.814 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.814 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.814 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:54.814 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.574896+0000", 2024-04-19T01:04:54.815 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.021991301000000001 2024-04-19T01:04:54.815 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.815 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.815 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:54.815 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.578996+0000", 2024-04-19T01:04:54.815 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0040997849999999999 2024-04-19T01:04:54.815 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.815 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.815 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:54.816 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.579108+0000", 2024-04-19T01:04:54.816 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000111962 2024-04-19T01:04:54.816 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:54.816 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:54.816 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:54.816 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.816 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.816 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:362 3.8s0 3:11aca45f:::smithi15528762-45:head [append 0~344064] snapc 9=[7] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-04-19T01:04:54.816 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:19.547167+0000", 2024-04-19T01:04:54.816 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 35.240547626999998, 2024-04-19T01:04:54.816 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.032345455000000002, 2024-04-19T01:04:54.816 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:54.816 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:54.816 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:54.817 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:54.817 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:04:54.817 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:54.817 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 362 2024-04-19T01:04:54.817 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.817 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:54.817 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.817 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:54.817 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547167+0000", 2024-04-19T01:04:54.817 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:54.817 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.817 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.817 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:54.817 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547167+0000", 2024-04-19T01:04:54.817 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:54.818 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.818 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.818 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:54.818 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547171+0000", 2024-04-19T01:04:54.818 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.303e-06 2024-04-19T01:04:54.818 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.818 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.818 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:54.818 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547655+0000", 2024-04-19T01:04:54.818 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00048394099999999999 2024-04-19T01:04:54.818 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.818 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.818 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:54.818 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547684+0000", 2024-04-19T01:04:54.818 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9828999999999999e-05 2024-04-19T01:04:54.819 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.819 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.819 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:54.819 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547699+0000", 2024-04-19T01:04:54.819 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4739999999999999e-05 2024-04-19T01:04:54.819 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.819 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.819 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:54.819 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.564911+0000", 2024-04-19T01:04:54.819 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.017211820999999999 2024-04-19T01:04:54.819 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.819 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.819 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:54.819 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.565200+0000", 2024-04-19T01:04:54.819 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00028915700000000001 2024-04-19T01:04:54.820 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.820 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.820 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:54.820 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.567550+0000", 2024-04-19T01:04:54.820 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0023502869999999999 2024-04-19T01:04:54.820 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.820 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.820 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:54.820 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.574910+0000", 2024-04-19T01:04:54.820 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0073593690000000002 2024-04-19T01:04:54.820 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.820 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.820 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:54.820 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.579412+0000", 2024-04-19T01:04:54.821 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0045022100000000004 2024-04-19T01:04:54.821 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.821 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.821 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:54.821 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.579513+0000", 2024-04-19T01:04:54.821 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010079800000000001 2024-04-19T01:04:54.821 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:54.821 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:54.821 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:54.821 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.821 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.821 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:363 3.8s0 3:11aca45f:::smithi15528762-45:head [setxattr _header (56) in=7b] snapc 9=[7] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-04-19T01:04:54.821 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:19.547729+0000", 2024-04-19T01:04:54.821 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 35.239985761, 2024-04-19T01:04:54.821 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.033477475, 2024-04-19T01:04:54.822 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:54.822 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:54.822 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:54.822 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:54.822 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:04:54.822 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:54.822 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 363 2024-04-19T01:04:54.822 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.822 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:54.822 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.822 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:54.822 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547729+0000", 2024-04-19T01:04:54.822 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:54.822 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.822 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.823 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:54.823 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547729+0000", 2024-04-19T01:04:54.823 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:54.823 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.823 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.823 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:54.823 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547730+0000", 2024-04-19T01:04:54.823 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2160000000000001e-06 2024-04-19T01:04:54.823 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.823 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.823 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:54.823 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547749+0000", 2024-04-19T01:04:54.823 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8312999999999999e-05 2024-04-19T01:04:54.823 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.824 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.824 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:54.824 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547764+0000", 2024-04-19T01:04:54.824 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5279999999999999e-05 2024-04-19T01:04:54.824 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.824 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.824 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:54.824 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547773+0000", 2024-04-19T01:04:54.824 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3300000000000005e-06 2024-04-19T01:04:54.824 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.824 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.824 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:54.824 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.572682+0000", 2024-04-19T01:04:54.824 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.024908825999999998 2024-04-19T01:04:54.824 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.825 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.825 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:54.825 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.572817+0000", 2024-04-19T01:04:54.825 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000135026 2024-04-19T01:04:54.825 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.825 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.825 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:54.825 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.573699+0000", 2024-04-19T01:04:54.825 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00088153199999999998 2024-04-19T01:04:54.825 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.826 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.826 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:54.826 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.575580+0000", 2024-04-19T01:04:54.826 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0018816589999999999 2024-04-19T01:04:54.826 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.826 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.826 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:54.826 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.581051+0000", 2024-04-19T01:04:54.826 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0054708220000000002 2024-04-19T01:04:54.826 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.826 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.826 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:54.826 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.581207+0000", 2024-04-19T01:04:54.827 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015547099999999999 2024-04-19T01:04:54.827 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:54.827 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:54.827 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:54.827 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.827 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.827 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:364 3.8s0 3:11aca45f:::smithi15528762-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-04-19T01:04:54.827 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:19.547786+0000", 2024-04-19T01:04:54.827 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 35.239928493000001, 2024-04-19T01:04:54.827 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.037412188999999998, 2024-04-19T01:04:54.827 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:54.827 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:54.827 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-04-19T01:04:54.827 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:54.827 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:04:54.828 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:54.828 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 364 2024-04-19T01:04:54.828 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.828 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:54.828 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.828 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:54.828 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547786+0000", 2024-04-19T01:04:54.828 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:54.828 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.828 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.828 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:54.828 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547786+0000", 2024-04-19T01:04:54.828 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:54.828 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.829 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.829 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:54.829 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547787+0000", 2024-04-19T01:04:54.829 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.1399999999999996e-07 2024-04-19T01:04:54.829 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.829 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.829 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:54.829 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547791+0000", 2024-04-19T01:04:54.829 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8299999999999998e-06 2024-04-19T01:04:54.829 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.829 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.829 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:54.829 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547802+0000", 2024-04-19T01:04:54.829 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1372999999999999e-05 2024-04-19T01:04:54.829 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.830 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.830 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:54.830 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547811+0000", 2024-04-19T01:04:54.830 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.0469999999999999e-06 2024-04-19T01:04:54.830 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.830 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.830 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:54.830 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.575967+0000", 2024-04-19T01:04:54.830 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.028155099999999999 2024-04-19T01:04:54.830 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.830 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.830 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-04-19T01:04:54.830 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.576164+0000", 2024-04-19T01:04:54.830 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019718099999999999 2024-04-19T01:04:54.830 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.831 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.831 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:54.831 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.581873+0000", 2024-04-19T01:04:54.831 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0057093359999999998 2024-04-19T01:04:54.831 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.831 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.831 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:54.831 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.582089+0000", 2024-04-19T01:04:54.831 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021609099999999999 2024-04-19T01:04:54.831 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.831 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.831 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:54.831 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.585199+0000", 2024-04-19T01:04:54.831 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0031094170000000002 2024-04-19T01:04:54.831 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:54.832 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:54.832 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:54.832 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.832 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.832 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.5s3 59/31 ECSubWrite(tid=66, reqid=client.4604.0:423, at_version=59'51, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:04:54.832 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:29.545169+0000", 2024-04-19T01:04:54.832 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 25.242545960000001, 2024-04-19T01:04:54.832 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.030406544000000001, 2024-04-19T01:04:54.832 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:54.832 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:54.832 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:54.832 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:54.832 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.832 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:54.833 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.545169+0000", 2024-04-19T01:04:54.833 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:54.833 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.833 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.833 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:54.833 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.545169+0000", 2024-04-19T01:04:54.833 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:54.833 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.833 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.833 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:54.833 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.545169+0000", 2024-04-19T01:04:54.833 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7999999999999996e-07 2024-04-19T01:04:54.833 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.833 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.833 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:54.834 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.545506+0000", 2024-04-19T01:04:54.834 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00033656400000000003 2024-04-19T01:04:54.834 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.834 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.834 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:54.834 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.545535+0000", 2024-04-19T01:04:54.834 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9286e-05 2024-04-19T01:04:54.834 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.834 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.834 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:54.834 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.545558+0000", 2024-04-19T01:04:54.834 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2473e-05 2024-04-19T01:04:54.834 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.834 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.835 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:54.835 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.545886+0000", 2024-04-19T01:04:54.835 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00032803800000000001 2024-04-19T01:04:54.835 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.835 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.835 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:54.835 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.545912+0000", 2024-04-19T01:04:54.835 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6486e-05 2024-04-19T01:04:54.835 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.835 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.835 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:54.835 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.575499+0000", 2024-04-19T01:04:54.835 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.029586642 2024-04-19T01:04:54.835 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.835 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.836 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:54.836 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.575575+0000", 2024-04-19T01:04:54.836 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6575000000000005e-05 2024-04-19T01:04:54.836 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:54.836 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:54.836 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:54.836 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.836 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.837 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:433 3.8s0 3:11aca45f:::smithi15528762-45:head [append 0~671744] snapc f=[f,d] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-19T01:04:54.837 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:30.566921+0000", 2024-04-19T01:04:54.837 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 24.220794175000002, 2024-04-19T01:04:54.837 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.030628952000000001, 2024-04-19T01:04:54.837 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:54.837 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:54.837 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:54.837 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:54.837 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:04:54.837 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:54.837 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 433 2024-04-19T01:04:54.837 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.837 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:54.837 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.837 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:54.838 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.566921+0000", 2024-04-19T01:04:54.838 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:54.838 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.838 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.838 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:54.838 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.566921+0000", 2024-04-19T01:04:54.838 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:54.838 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.838 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.838 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:54.838 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.566924+0000", 2024-04-19T01:04:54.838 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.213e-06 2024-04-19T01:04:54.838 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.838 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.839 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:54.839 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.567622+0000", 2024-04-19T01:04:54.839 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000698293 2024-04-19T01:04:54.839 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.839 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.839 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:54.839 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.567661+0000", 2024-04-19T01:04:54.839 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9122000000000003e-05 2024-04-19T01:04:54.839 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.839 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.839 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:54.839 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.567680+0000", 2024-04-19T01:04:54.839 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8326999999999999e-05 2024-04-19T01:04:54.839 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.839 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.840 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:54.840 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.567773+0000", 2024-04-19T01:04:54.840 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3372e-05 2024-04-19T01:04:54.840 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.840 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.840 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:54.840 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.567902+0000", 2024-04-19T01:04:54.840 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000128987 2024-04-19T01:04:54.840 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.840 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.840 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:54.840 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.569319+0000", 2024-04-19T01:04:54.840 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0014167559999999999 2024-04-19T01:04:54.840 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.840 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.841 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:54.841 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.577387+0000", 2024-04-19T01:04:54.841 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.008068492 2024-04-19T01:04:54.841 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.841 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.841 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:54.841 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.597375+0000", 2024-04-19T01:04:54.841 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.019987900999999999 2024-04-19T01:04:54.841 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.841 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.841 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:54.841 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.597550+0000", 2024-04-19T01:04:54.841 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000174489 2024-04-19T01:04:54.841 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:54.841 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:54.842 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:54.842 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.842 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.842 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.5s3 65/31 ECSubWrite(tid=75, reqid=client.4604.0:486, at_version=65'56, trim_to=0'0, roll_forward_to=59'54))", 2024-04-19T01:04:54.842 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:35.660587+0000", 2024-04-19T01:04:54.842 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 19.127127937000001, 2024-04-19T01:04:54.842 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.030868289, 2024-04-19T01:04:54.842 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:54.842 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:54.842 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:54.842 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:54.842 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.842 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:54.842 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.660587+0000", 2024-04-19T01:04:54.842 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:54.843 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.843 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.843 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:54.843 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.660587+0000", 2024-04-19T01:04:54.843 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:54.843 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.843 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.843 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:54.843 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.660589+0000", 2024-04-19T01:04:54.843 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.407e-06 2024-04-19T01:04:54.843 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.843 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.843 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:54.843 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.661477+0000", 2024-04-19T01:04:54.843 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00088810699999999996 2024-04-19T01:04:54.844 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.844 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.844 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:54.844 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.661515+0000", 2024-04-19T01:04:54.844 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7265999999999999e-05 2024-04-19T01:04:54.844 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.844 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.844 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:54.844 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.661533+0000", 2024-04-19T01:04:54.844 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8056000000000001e-05 2024-04-19T01:04:54.844 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.844 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.844 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:54.844 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.661700+0000", 2024-04-19T01:04:54.844 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000167548 2024-04-19T01:04:54.845 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.845 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.845 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:54.845 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.661733+0000", 2024-04-19T01:04:54.845 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2258999999999997e-05 2024-04-19T01:04:54.845 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.845 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.845 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:54.845 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.691386+0000", 2024-04-19T01:04:54.845 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.029653834 2024-04-19T01:04:54.845 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.845 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.845 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:54.845 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.691455+0000", 2024-04-19T01:04:54.845 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.8812000000000005e-05 2024-04-19T01:04:54.846 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:54.846 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:54.846 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:54.846 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.846 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.846 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.5s3 74/31 ECSubWrite(tid=87, reqid=client.4604.0:543, at_version=74'67, trim_to=0'0, roll_forward_to=72'65))", 2024-04-19T01:04:54.846 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:44.553593+0000", 2024-04-19T01:04:54.846 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.23412182, 2024-04-19T01:04:54.846 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.044047740000000002, 2024-04-19T01:04:54.847 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:54.847 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:54.847 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:54.847 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:54.847 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.847 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:54.847 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.553593+0000", 2024-04-19T01:04:54.847 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:54.847 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.847 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.847 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:54.847 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.553593+0000", 2024-04-19T01:04:54.848 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:54.848 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.848 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.848 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:54.848 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.553594+0000", 2024-04-19T01:04:54.848 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.327e-06 2024-04-19T01:04:54.848 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.848 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.848 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:54.848 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.554148+0000", 2024-04-19T01:04:54.848 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00055343600000000003 2024-04-19T01:04:54.848 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.848 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.848 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:54.849 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.554184+0000", 2024-04-19T01:04:54.849 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5865999999999998e-05 2024-04-19T01:04:54.849 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.849 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.849 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:54.849 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.554211+0000", 2024-04-19T01:04:54.849 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7203999999999999e-05 2024-04-19T01:04:54.849 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.849 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.849 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:54.849 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.554385+0000", 2024-04-19T01:04:54.849 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000174387 2024-04-19T01:04:54.849 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.849 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.850 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:54.850 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.554418+0000", 2024-04-19T01:04:54.850 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2502999999999997e-05 2024-04-19T01:04:54.850 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.850 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.850 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:54.850 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.597567+0000", 2024-04-19T01:04:54.850 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.043149383999999999 2024-04-19T01:04:54.850 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.850 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.850 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:54.850 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.597641+0000", 2024-04-19T01:04:54.850 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3633000000000005e-05 2024-04-19T01:04:54.850 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:54.850 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:54.851 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:54.851 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.851 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.851 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.5s3 74/31 ECSubWrite(tid=88, reqid=osd.8.0:88, at_version=74'69, trim_to=0'0, roll_forward_to=72'65))", 2024-04-19T01:04:54.851 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:44.559673+0000", 2024-04-19T01:04:54.851 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.228042393999999, 2024-04-19T01:04:54.851 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.038043423, 2024-04-19T01:04:54.851 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:54.851 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:54.851 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:54.851 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:54.851 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.851 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:54.851 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.559673+0000", 2024-04-19T01:04:54.852 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:54.852 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.852 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.852 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:54.852 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.559673+0000", 2024-04-19T01:04:54.852 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:54.852 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.852 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.852 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:54.852 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.559674+0000", 2024-04-19T01:04:54.852 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.77e-06 2024-04-19T01:04:54.852 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.852 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.852 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:54.852 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.559727+0000", 2024-04-19T01:04:54.853 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2552000000000001e-05 2024-04-19T01:04:54.853 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.853 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.853 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:54.853 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.559756+0000", 2024-04-19T01:04:54.853 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9254000000000001e-05 2024-04-19T01:04:54.853 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.853 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.853 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:54.853 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.559780+0000", 2024-04-19T01:04:54.853 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3856e-05 2024-04-19T01:04:54.853 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.853 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.853 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:54.853 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.571632+0000", 2024-04-19T01:04:54.854 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.011851733 2024-04-19T01:04:54.854 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.854 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.854 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:54.854 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.571658+0000", 2024-04-19T01:04:54.855 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6568999999999999e-05 2024-04-19T01:04:54.855 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.856 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.856 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:54.856 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.597662+0000", 2024-04-19T01:04:54.856 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.02600421 2024-04-19T01:04:54.856 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.856 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.856 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:54.856 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.597716+0000", 2024-04-19T01:04:54.856 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3479000000000001e-05 2024-04-19T01:04:54.856 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:54.856 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:54.857 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:54.857 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.857 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.857 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.5s3 74/31 ECSubWrite(tid=89, reqid=client.4604.0:544, at_version=74'70, trim_to=0'0, roll_forward_to=72'65))", 2024-04-19T01:04:54.857 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:44.565551+0000", 2024-04-19T01:04:54.857 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.222163547999999, 2024-04-19T01:04:54.857 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.032242209000000001, 2024-04-19T01:04:54.857 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:54.857 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:54.857 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:54.857 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:54.857 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.858 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:54.858 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.565551+0000", 2024-04-19T01:04:54.858 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:54.858 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.858 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.858 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:54.858 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.565551+0000", 2024-04-19T01:04:54.858 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:54.858 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.858 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.858 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:54.858 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.565553+0000", 2024-04-19T01:04:54.858 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.136e-06 2024-04-19T01:04:54.858 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.859 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.859 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:54.859 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.566058+0000", 2024-04-19T01:04:54.859 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00050426699999999997 2024-04-19T01:04:54.859 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.859 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.859 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:54.859 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.566105+0000", 2024-04-19T01:04:54.859 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7636000000000003e-05 2024-04-19T01:04:54.859 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.859 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.859 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:54.859 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.566129+0000", 2024-04-19T01:04:54.859 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3433000000000001e-05 2024-04-19T01:04:54.859 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.860 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.860 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:54.860 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.575213+0000", 2024-04-19T01:04:54.860 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0090842939999999997 2024-04-19T01:04:54.860 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.860 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.860 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:54.860 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.575244+0000", 2024-04-19T01:04:54.860 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1350000000000003e-05 2024-04-19T01:04:54.860 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.860 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.860 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:54.860 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.597722+0000", 2024-04-19T01:04:54.860 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.022477924 2024-04-19T01:04:54.860 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.861 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.861 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:54.861 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.597794+0000", 2024-04-19T01:04:54.861 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.1168999999999996e-05 2024-04-19T01:04:54.861 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:54.861 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:54.861 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:54.861 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.861 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.861 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.5s3 79/31 ECSubWrite(tid=115, reqid=client.4604.0:591, at_version=79'90, trim_to=0'0, roll_forward_to=79'86))", 2024-04-19T01:04:54.861 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:49.660197+0000", 2024-04-19T01:04:54.861 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.1275180139999996, 2024-04-19T01:04:54.861 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.037045658000000002, 2024-04-19T01:04:54.861 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:54.861 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:54.862 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:54.862 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:54.862 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.862 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:54.862 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.660197+0000", 2024-04-19T01:04:54.862 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:54.862 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.862 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.862 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:54.862 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.660197+0000", 2024-04-19T01:04:54.862 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:54.862 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.862 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.862 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:54.862 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.660198+0000", 2024-04-19T01:04:54.863 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4270000000000001e-06 2024-04-19T01:04:54.863 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.863 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.863 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:54.863 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.660609+0000", 2024-04-19T01:04:54.863 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00041111799999999998 2024-04-19T01:04:54.863 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.863 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.863 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:54.863 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.660635+0000", 2024-04-19T01:04:54.863 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5939e-05 2024-04-19T01:04:54.863 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.863 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.863 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:54.863 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.660649+0000", 2024-04-19T01:04:54.864 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3737000000000001e-05 2024-04-19T01:04:54.864 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.864 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.864 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:54.864 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.660788+0000", 2024-04-19T01:04:54.864 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013921199999999999 2024-04-19T01:04:54.864 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.864 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.864 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:54.864 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.660808+0000", 2024-04-19T01:04:54.864 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9202999999999999e-05 2024-04-19T01:04:54.864 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.864 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.864 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:54.865 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.697159+0000", 2024-04-19T01:04:54.865 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.036351151999999998 2024-04-19T01:04:54.865 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.865 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.865 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:54.865 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.697243+0000", 2024-04-19T01:04:54.865 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.3869999999999995e-05 2024-04-19T01:04:54.865 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:54.865 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:54.865 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:54.865 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.865 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.865 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.5s3 79/31 ECSubWrite(tid=116, reqid=client.4604.0:592, at_version=79'91, trim_to=0'0, roll_forward_to=79'88))", 2024-04-19T01:04:54.865 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:49.665284+0000", 2024-04-19T01:04:54.865 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.1224312919999999, 2024-04-19T01:04:54.866 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.032029100999999997, 2024-04-19T01:04:54.866 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:54.866 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:54.866 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:54.866 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:54.866 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.866 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:54.866 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.665284+0000", 2024-04-19T01:04:54.866 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:54.866 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.866 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.866 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:54.866 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.665284+0000", 2024-04-19T01:04:54.866 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:54.867 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.867 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.867 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:54.867 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.665285+0000", 2024-04-19T01:04:54.867 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.843e-06 2024-04-19T01:04:54.867 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.867 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.867 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:54.867 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.665682+0000", 2024-04-19T01:04:54.868 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00039639900000000002 2024-04-19T01:04:54.868 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.868 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.868 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:54.868 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.665717+0000", 2024-04-19T01:04:54.868 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5342999999999997e-05 2024-04-19T01:04:54.868 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.868 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.868 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:54.868 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.665740+0000", 2024-04-19T01:04:54.868 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2731999999999999e-05 2024-04-19T01:04:54.868 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.868 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.868 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:54.869 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.675712+0000", 2024-04-19T01:04:54.869 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0099725370000000001 2024-04-19T01:04:54.869 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.869 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.869 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:54.869 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.678072+0000", 2024-04-19T01:04:54.869 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0023595130000000001 2024-04-19T01:04:54.869 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.869 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.870 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:54.870 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.697256+0000", 2024-04-19T01:04:54.870 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.019184063000000001 2024-04-19T01:04:54.870 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.870 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.870 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:54.870 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.697313+0000", 2024-04-19T01:04:54.870 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6671e-05 2024-04-19T01:04:54.870 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:54.870 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:54.870 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:54.870 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.871 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.871 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.5s3 81/31 ECSubWrite(tid=122, reqid=client.4604.0:626, at_version=81'96, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:04:54.871 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:51.593541+0000", 2024-04-19T01:04:54.871 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 3.194174034, 2024-04-19T01:04:54.871 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.061054634000000003, 2024-04-19T01:04:54.871 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:54.871 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:54.871 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:54.871 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:54.871 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.871 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:54.871 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.593541+0000", 2024-04-19T01:04:54.871 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:54.871 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.872 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.872 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:54.872 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.593541+0000", 2024-04-19T01:04:54.872 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:54.872 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.872 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.872 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:54.872 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.593541+0000", 2024-04-19T01:04:54.872 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9699999999999996e-07 2024-04-19T01:04:54.872 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.872 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.872 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:54.872 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.594429+0000", 2024-04-19T01:04:54.872 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000887318 2024-04-19T01:04:54.872 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.873 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.873 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:54.873 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.594456+0000", 2024-04-19T01:04:54.873 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7509e-05 2024-04-19T01:04:54.873 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.873 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.873 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:54.873 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.594472+0000", 2024-04-19T01:04:54.873 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6107e-05 2024-04-19T01:04:54.873 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.873 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.873 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:54.873 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.594609+0000", 2024-04-19T01:04:54.873 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013671899999999999 2024-04-19T01:04:54.873 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.874 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.874 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:54.874 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.594628+0000", 2024-04-19T01:04:54.874 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9123e-05 2024-04-19T01:04:54.874 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.874 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.874 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:54.874 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.654485+0000", 2024-04-19T01:04:54.874 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.059857150999999997 2024-04-19T01:04:54.874 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.874 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.874 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:54.874 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.654596+0000", 2024-04-19T01:04:54.874 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011021 2024-04-19T01:04:54.874 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:54.875 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:54.875 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:54.875 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.875 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.875 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.5s3 81/31 ECSubWrite(tid=123, reqid=client.4604.0:627, at_version=81'97, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:04:54.875 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:51.600033+0000", 2024-04-19T01:04:54.875 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 3.1876815000000001, 2024-04-19T01:04:54.875 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.054639177999999997, 2024-04-19T01:04:54.875 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:54.875 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:54.875 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:54.875 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:54.875 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.875 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:54.875 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.600033+0000", 2024-04-19T01:04:54.876 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:54.876 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.876 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.876 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:54.876 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.600033+0000", 2024-04-19T01:04:54.876 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:54.876 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.876 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.876 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:54.876 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.600036+0000", 2024-04-19T01:04:54.876 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.419e-06 2024-04-19T01:04:54.876 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.876 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.876 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:54.876 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.600416+0000", 2024-04-19T01:04:54.877 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00038017299999999998 2024-04-19T01:04:54.877 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.877 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.877 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:54.877 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.600457+0000", 2024-04-19T01:04:54.877 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1075999999999998e-05 2024-04-19T01:04:54.877 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.877 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.877 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:54.878 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.600480+0000", 2024-04-19T01:04:54.878 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2976000000000002e-05 2024-04-19T01:04:54.878 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.878 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.878 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:54.878 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.609097+0000", 2024-04-19T01:04:54.878 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.008617408 2024-04-19T01:04:54.878 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.878 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.878 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:54.878 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.609118+0000", 2024-04-19T01:04:54.878 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0136000000000002e-05 2024-04-19T01:04:54.878 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.878 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.879 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:54.879 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.654605+0000", 2024-04-19T01:04:54.879 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.045486985000000001 2024-04-19T01:04:54.879 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.879 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.879 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:54.879 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.654673+0000", 2024-04-19T01:04:54.879 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.8004999999999997e-05 2024-04-19T01:04:54.879 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:54.879 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:54.879 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:54.879 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.879 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.879 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.5s3 81/31 ECSubWrite(tid=124, reqid=client.4604.0:628, at_version=81'98, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:04:54.880 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:51.605750+0000", 2024-04-19T01:04:54.880 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 3.1819644660000002, 2024-04-19T01:04:54.880 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.048993015000000001, 2024-04-19T01:04:54.880 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:54.880 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:54.880 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:54.880 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:54.880 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.880 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:54.880 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.605750+0000", 2024-04-19T01:04:54.880 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:54.880 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.880 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.881 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:54.881 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.605750+0000", 2024-04-19T01:04:54.881 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:54.882 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.882 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.882 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:54.882 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.605752+0000", 2024-04-19T01:04:54.882 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9870000000000002e-06 2024-04-19T01:04:54.882 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.882 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.882 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:54.882 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.606125+0000", 2024-04-19T01:04:54.882 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00037219999999999999 2024-04-19T01:04:54.882 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.882 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.883 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:54.883 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.606160+0000", 2024-04-19T01:04:54.883 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5534999999999998e-05 2024-04-19T01:04:54.883 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.883 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.883 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:54.883 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.606182+0000", 2024-04-19T01:04:54.883 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1999000000000001e-05 2024-04-19T01:04:54.883 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.883 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.883 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:54.883 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.618898+0000", 2024-04-19T01:04:54.883 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.012715888 2024-04-19T01:04:54.883 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.884 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.884 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:54.884 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.618930+0000", 2024-04-19T01:04:54.884 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1712999999999999e-05 2024-04-19T01:04:54.884 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.884 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.884 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:54.884 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.654683+0000", 2024-04-19T01:04:54.884 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.035753339000000002 2024-04-19T01:04:54.884 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.884 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.884 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:54.884 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.654743+0000", 2024-04-19T01:04:54.884 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0353999999999998e-05 2024-04-19T01:04:54.884 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:54.885 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:54.885 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:54.885 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.885 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.885 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.5s3 81/31 ECSubWrite(tid=125, reqid=client.4604.0:629, at_version=81'99, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:04:54.885 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:51.609931+0000", 2024-04-19T01:04:54.885 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 3.177783544, 2024-04-19T01:04:54.885 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.044873181999999998, 2024-04-19T01:04:54.885 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:54.885 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:54.885 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:54.885 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:54.885 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.885 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:54.886 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.609931+0000", 2024-04-19T01:04:54.886 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:54.886 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.886 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.886 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:54.886 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.609931+0000", 2024-04-19T01:04:54.886 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:54.886 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.886 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.886 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:54.886 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.609933+0000", 2024-04-19T01:04:54.886 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.753e-06 2024-04-19T01:04:54.886 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.886 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.887 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:54.887 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.609982+0000", 2024-04-19T01:04:54.887 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9330999999999999e-05 2024-04-19T01:04:54.887 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.887 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.887 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:54.887 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.610031+0000", 2024-04-19T01:04:54.887 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8718999999999999e-05 2024-04-19T01:04:54.887 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.887 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.887 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:54.887 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.610048+0000", 2024-04-19T01:04:54.887 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6923e-05 2024-04-19T01:04:54.887 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.887 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.888 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:54.888 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.629621+0000", 2024-04-19T01:04:54.888 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.019572523000000001 2024-04-19T01:04:54.888 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.888 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.888 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:54.888 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.629650+0000", 2024-04-19T01:04:54.888 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9104000000000001e-05 2024-04-19T01:04:54.889 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.889 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.889 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:54.889 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.654749+0000", 2024-04-19T01:04:54.889 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.025098874 2024-04-19T01:04:54.889 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.889 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.889 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:54.889 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.654805+0000", 2024-04-19T01:04:54.889 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.5955000000000002e-05 2024-04-19T01:04:54.889 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:54.889 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:54.889 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:54.889 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.890 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.890 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.5s3 81/31 ECSubWrite(tid=126, reqid=client.4604.0:631, at_version=81'101, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:04:54.890 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:51.618408+0000", 2024-04-19T01:04:54.890 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 3.1693070360000002, 2024-04-19T01:04:54.890 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.038972772000000003, 2024-04-19T01:04:54.890 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:54.890 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:54.890 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:54.890 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:54.890 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.890 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:54.890 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.618408+0000", 2024-04-19T01:04:54.890 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:54.891 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.891 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.891 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:54.891 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.618408+0000", 2024-04-19T01:04:54.891 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:54.891 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.891 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.891 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:54.891 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.618410+0000", 2024-04-19T01:04:54.891 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.863e-06 2024-04-19T01:04:54.891 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.891 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.891 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:54.891 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.619228+0000", 2024-04-19T01:04:54.892 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00081862099999999995 2024-04-19T01:04:54.892 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.892 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.892 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:54.892 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.619258+0000", 2024-04-19T01:04:54.892 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9805e-05 2024-04-19T01:04:54.892 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.892 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.892 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:54.892 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.619280+0000", 2024-04-19T01:04:54.892 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.158e-05 2024-04-19T01:04:54.892 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.892 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.892 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:54.892 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.630187+0000", 2024-04-19T01:04:54.893 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.010906911 2024-04-19T01:04:54.893 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.893 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.893 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:54.893 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.630202+0000", 2024-04-19T01:04:54.893 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.56e-05 2024-04-19T01:04:54.893 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.893 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.893 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:54.893 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.657322+0000", 2024-04-19T01:04:54.893 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.027119353999999998 2024-04-19T01:04:54.893 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:54.893 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:54.893 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:54.894 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.657381+0000", 2024-04-19T01:04:54.894 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9038e-05 2024-04-19T01:04:54.894 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:54.894 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:54.894 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:54.894 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:54.894 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:54.894 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:04:54.895 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2024-04-19T01:04:54.896 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-04-19T01:04:54.897 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-19T01:04:54.898 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:04:54.895+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:04:54.996 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:04:54.995+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:04:55.022 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:04:55.022 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:04:55.022 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-04-19T01:04:55.022 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:04:55.035 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-04-19T01:04:55.098 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:04:55.098+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:04:55.155 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:04:55.155 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:04:55.155 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-04-19T01:04:55.156 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-04-19T01:04:55.156 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:04:55.169 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-04-19T01:04:55.199 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:04:55.197+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:04:55.291 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:04:55.291 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-04-19T01:04:55.292 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-04-19T01:04:55.292 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-04-19T01:04:55.292 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.292 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:173 3.3s0 3:c34529b5:::smithi15528762-31:head [append 0~507904] snapc 5=[5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-19T01:04:55.292 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:08.816172+0000", 2024-04-19T01:04:55.292 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 46.473571599000003, 2024-04-19T01:04:55.292 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.024750307999999999, 2024-04-19T01:04:55.292 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:55.292 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:55.292 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:55.292 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:55.292 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:04:55.292 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:55.292 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 173 2024-04-19T01:04:55.293 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.293 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:55.293 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.293 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:55.293 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816172+0000", 2024-04-19T01:04:55.293 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:55.293 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.293 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.293 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:55.293 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816172+0000", 2024-04-19T01:04:55.293 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:55.293 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.293 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.293 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:55.293 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816175+0000", 2024-04-19T01:04:55.294 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9589999999999997e-06 2024-04-19T01:04:55.294 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.294 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.294 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:55.294 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816766+0000", 2024-04-19T01:04:55.294 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00059020999999999998 2024-04-19T01:04:55.294 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.294 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.294 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:55.294 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816802+0000", 2024-04-19T01:04:55.294 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6140999999999998e-05 2024-04-19T01:04:55.294 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.294 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.295 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:55.295 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816820+0000", 2024-04-19T01:04:55.295 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7771999999999999e-05 2024-04-19T01:04:55.295 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.295 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.295 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:55.295 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816990+0000", 2024-04-19T01:04:55.295 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016993299999999999 2024-04-19T01:04:55.295 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.295 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.295 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op must wait for map", 2024-04-19T01:04:55.295 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.817016+0000", 2024-04-19T01:04:55.295 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6857000000000001e-05 2024-04-19T01:04:55.295 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.296 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.296 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:55.296 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.821660+0000", 2024-04-19T01:04:55.296 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0046440520000000001 2024-04-19T01:04:55.296 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.296 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.296 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:55.296 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.821835+0000", 2024-04-19T01:04:55.296 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000174112 2024-04-19T01:04:55.296 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.296 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.296 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:55.296 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.824020+0000", 2024-04-19T01:04:55.296 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.002185523 2024-04-19T01:04:55.296 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.297 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.297 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:55.297 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.839423+0000", 2024-04-19T01:04:55.297 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.015403261999999999 2024-04-19T01:04:55.297 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.297 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.297 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:55.297 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.840758+0000", 2024-04-19T01:04:55.297 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001335145 2024-04-19T01:04:55.297 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.297 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.297 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:55.297 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.840922+0000", 2024-04-19T01:04:55.297 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000163342 2024-04-19T01:04:55.298 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:55.298 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:55.298 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:55.298 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.298 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.310 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:174 3.3s0 3:c34529b5:::smithi15528762-31:head [append 0~376832] snapc 5=[5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-19T01:04:55.310 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:08.821669+0000", 2024-04-19T01:04:55.310 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 46.468074350999999, 2024-04-19T01:04:55.310 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.019779560000000002, 2024-04-19T01:04:55.311 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:55.311 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:55.311 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:55.311 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:55.311 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:04:55.311 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:55.311 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 174 2024-04-19T01:04:55.311 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.311 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:55.311 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.311 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:55.312 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.821669+0000", 2024-04-19T01:04:55.312 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:55.312 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.312 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.312 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:55.312 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.821669+0000", 2024-04-19T01:04:55.312 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:55.312 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.312 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.312 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:55.312 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.821671+0000", 2024-04-19T01:04:55.313 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2539999999999999e-06 2024-04-19T01:04:55.313 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.313 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.313 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:55.313 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822078+0000", 2024-04-19T01:04:55.313 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00040720600000000002 2024-04-19T01:04:55.313 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.313 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.313 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:55.313 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822097+0000", 2024-04-19T01:04:55.313 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9011000000000001e-05 2024-04-19T01:04:55.314 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.314 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.314 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:55.314 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822111+0000", 2024-04-19T01:04:55.314 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4188e-05 2024-04-19T01:04:55.314 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.314 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.314 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:55.314 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.830924+0000", 2024-04-19T01:04:55.314 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0088129609999999994 2024-04-19T01:04:55.314 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.314 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.314 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:55.314 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.831084+0000", 2024-04-19T01:04:55.315 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000159889 2024-04-19T01:04:55.315 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.315 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.315 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:55.315 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.832503+0000", 2024-04-19T01:04:55.315 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001418655 2024-04-19T01:04:55.328 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.328 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.328 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:55.328 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.841099+0000", 2024-04-19T01:04:55.329 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0085957340000000007 2024-04-19T01:04:55.329 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.329 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.329 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:55.329 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.841345+0000", 2024-04-19T01:04:55.329 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00024665800000000002 2024-04-19T01:04:55.329 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.329 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.329 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:55.329 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.841448+0000", 2024-04-19T01:04:55.329 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010300400000000001 2024-04-19T01:04:55.329 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:55.329 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:55.330 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:55.330 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.330 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.330 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:175 3.3s0 3:c34529b5:::smithi15528762-31:head [setxattr _header (56) in=7b] snapc 5=[5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-19T01:04:55.330 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:08.822142+0000", 2024-04-19T01:04:55.331 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 46.467601350999999, 2024-04-19T01:04:55.331 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.020224723999999999, 2024-04-19T01:04:55.331 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:55.331 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:55.331 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:55.331 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:55.331 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:04:55.331 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:55.335 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 175 2024-04-19T01:04:55.335 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.335 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:55.335 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.336 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:55.336 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822142+0000", 2024-04-19T01:04:55.336 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:55.336 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.336 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.336 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:55.336 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822142+0000", 2024-04-19T01:04:55.336 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:55.336 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.336 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.336 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:55.336 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822143+0000", 2024-04-19T01:04:55.336 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2699999999999999e-06 2024-04-19T01:04:55.337 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.337 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.337 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:55.337 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822151+0000", 2024-04-19T01:04:55.337 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4760000000000001e-06 2024-04-19T01:04:55.337 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.337 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.337 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:55.337 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822165+0000", 2024-04-19T01:04:55.337 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4138000000000001e-05 2024-04-19T01:04:55.337 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.337 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.337 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:55.337 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822178+0000", 2024-04-19T01:04:55.338 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3185e-05 2024-04-19T01:04:55.338 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.338 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.338 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:55.338 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.839463+0000", 2024-04-19T01:04:55.338 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.017285038999999999 2024-04-19T01:04:55.338 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.338 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.338 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:55.338 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.839580+0000", 2024-04-19T01:04:55.338 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000117601 2024-04-19T01:04:55.338 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.338 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.338 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:55.339 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.840161+0000", 2024-04-19T01:04:55.339 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00058043199999999997 2024-04-19T01:04:55.339 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.339 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.339 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:55.339 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.841469+0000", 2024-04-19T01:04:55.339 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001308065 2024-04-19T01:04:55.339 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.339 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.339 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:55.342 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.842264+0000", 2024-04-19T01:04:55.342 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00079508400000000005 2024-04-19T01:04:55.342 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.342 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.342 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:55.342 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.842366+0000", 2024-04-19T01:04:55.342 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000102434 2024-04-19T01:04:55.343 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:55.343 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:55.343 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:55.343 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.343 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.343 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:176 3.3s0 3:c34529b5:::smithi15528762-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-19T01:04:55.343 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:08.822193+0000", 2024-04-19T01:04:55.343 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 46.467550136, 2024-04-19T01:04:55.343 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.023273202, 2024-04-19T01:04:55.343 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:55.343 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:55.343 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-04-19T01:04:55.343 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:55.343 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:04:55.344 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:55.344 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 176 2024-04-19T01:04:55.344 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.344 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:55.344 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.344 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:55.344 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822193+0000", 2024-04-19T01:04:55.344 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:55.344 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.344 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.344 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:55.344 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822193+0000", 2024-04-19T01:04:55.344 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:55.344 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.344 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.345 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:55.345 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822194+0000", 2024-04-19T01:04:55.345 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4200000000000004e-07 2024-04-19T01:04:55.345 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.345 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.345 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:55.345 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822198+0000", 2024-04-19T01:04:55.345 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.206e-06 2024-04-19T01:04:55.345 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.345 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.345 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:55.345 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822210+0000", 2024-04-19T01:04:55.345 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.217e-05 2024-04-19T01:04:55.345 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.346 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.346 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:55.346 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822220+0000", 2024-04-19T01:04:55.346 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0059e-05 2024-04-19T01:04:55.346 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.353 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.353 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:55.353 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.842930+0000", 2024-04-19T01:04:55.353 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.020709701 2024-04-19T01:04:55.353 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.353 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.353 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:55.353 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.843170+0000", 2024-04-19T01:04:55.354 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00024026700000000001 2024-04-19T01:04:55.354 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.354 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.354 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:55.354 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.845466+0000", 2024-04-19T01:04:55.354 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0022958570000000001 2024-04-19T01:04:55.354 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:55.354 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:55.354 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:55.354 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.355 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.355 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:343 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [copy-from ver 14] snapc 9=[8,7] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:04:55.355 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.511298+0000", 2024-04-19T01:04:55.355 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 36.778445138000002, 2024-04-19T01:04:55.355 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.043267551000000001, 2024-04-19T01:04:55.355 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:55.355 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:55.355 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:55.355 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:55.355 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:04:55.355 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:55.355 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 343 2024-04-19T01:04:55.356 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.356 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:55.356 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.356 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:55.356 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511298+0000", 2024-04-19T01:04:55.356 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:55.356 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.356 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.356 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:55.356 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511298+0000", 2024-04-19T01:04:55.356 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:55.356 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.356 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.357 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:55.357 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511302+0000", 2024-04-19T01:04:55.357 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1030000000000004e-06 2024-04-19T01:04:55.357 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.357 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.357 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:55.357 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511319+0000", 2024-04-19T01:04:55.357 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7070000000000001e-05 2024-04-19T01:04:55.357 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.357 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.357 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:55.357 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511357+0000", 2024-04-19T01:04:55.357 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8056000000000003e-05 2024-04-19T01:04:55.358 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.358 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.358 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:55.358 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511390+0000", 2024-04-19T01:04:55.358 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3213000000000002e-05 2024-04-19T01:04:55.358 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.358 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.358 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:55.358 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.516539+0000", 2024-04-19T01:04:55.358 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0051490729999999997 2024-04-19T01:04:55.358 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.358 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.358 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:55.359 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.516748+0000", 2024-04-19T01:04:55.359 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000208099 2024-04-19T01:04:55.359 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.359 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.359 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:55.359 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.541650+0000", 2024-04-19T01:04:55.359 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.024902837000000001 2024-04-19T01:04:55.359 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.359 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.359 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:55.359 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.551197+0000", 2024-04-19T01:04:55.359 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0095463240000000001 2024-04-19T01:04:55.359 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.359 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.360 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:55.360 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.554383+0000", 2024-04-19T01:04:55.360 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0031863830000000001 2024-04-19T01:04:55.360 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.360 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.360 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:55.360 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.554566+0000", 2024-04-19T01:04:55.360 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00018239299999999999 2024-04-19T01:04:55.360 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:55.360 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:55.360 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:55.360 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.360 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.360 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:344 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:04:55.361 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.511452+0000", 2024-04-19T01:04:55.361 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 36.778290888000001, 2024-04-19T01:04:55.361 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.038223957000000003, 2024-04-19T01:04:55.361 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:55.361 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:55.361 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-04-19T01:04:55.361 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:55.361 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:04:55.361 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:55.361 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 344 2024-04-19T01:04:55.361 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.361 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:55.361 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.362 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:55.362 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511452+0000", 2024-04-19T01:04:55.362 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:55.362 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.362 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.362 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:55.362 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511452+0000", 2024-04-19T01:04:55.362 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:55.362 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.362 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.363 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:55.363 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511454+0000", 2024-04-19T01:04:55.363 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.39e-06 2024-04-19T01:04:55.363 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.363 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.363 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:55.363 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511460+0000", 2024-04-19T01:04:55.363 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.46e-06 2024-04-19T01:04:55.363 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.363 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.364 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:55.364 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511476+0000", 2024-04-19T01:04:55.364 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6180000000000001e-05 2024-04-19T01:04:55.364 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.364 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.364 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:55.364 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511487+0000", 2024-04-19T01:04:55.364 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.049e-05 2024-04-19T01:04:55.364 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.365 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.365 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:55.365 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.517203+0000", 2024-04-19T01:04:55.365 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00571595 2024-04-19T01:04:55.368 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.368 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.368 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for blocked object", 2024-04-19T01:04:55.368 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.517341+0000", 2024-04-19T01:04:55.368 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013835099999999999 2024-04-19T01:04:55.368 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.368 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.368 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:55.368 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.549188+0000", 2024-04-19T01:04:55.368 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.031846589000000002 2024-04-19T01:04:55.369 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.369 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.369 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:55.369 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.549399+0000", 2024-04-19T01:04:55.369 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021149600000000001 2024-04-19T01:04:55.369 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.369 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.369 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:55.369 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.549676+0000", 2024-04-19T01:04:55.370 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000277051 2024-04-19T01:04:55.370 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:55.370 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:55.370 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:55.370 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.370 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.370 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.8s3 49/31 ECSubWrite(tid=203, reqid=client.4604.0:360, at_version=49'25, trim_to=0'0, roll_forward_to=49'24))", 2024-04-19T01:04:55.370 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:19.539927+0000", 2024-04-19T01:04:55.370 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 35.749815705000003, 2024-04-19T01:04:55.370 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.027280209, 2024-04-19T01:04:55.370 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:55.370 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:55.370 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:55.371 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:55.371 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.371 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:55.371 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.539927+0000", 2024-04-19T01:04:55.371 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:55.371 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.371 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.371 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:55.371 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.539927+0000", 2024-04-19T01:04:55.371 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:55.371 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.371 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.371 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:55.371 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.539929+0000", 2024-04-19T01:04:55.371 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3829999999999999e-06 2024-04-19T01:04:55.372 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.372 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.372 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:55.372 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540190+0000", 2024-04-19T01:04:55.372 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00026120499999999998 2024-04-19T01:04:55.372 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.372 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.372 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:55.372 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540216+0000", 2024-04-19T01:04:55.372 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5573000000000001e-05 2024-04-19T01:04:55.372 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.372 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.372 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:55.372 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540242+0000", 2024-04-19T01:04:55.373 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5939999999999999e-05 2024-04-19T01:04:55.373 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.373 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.373 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:55.373 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540387+0000", 2024-04-19T01:04:55.373 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000145983 2024-04-19T01:04:55.373 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.373 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.373 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:55.373 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540412+0000", 2024-04-19T01:04:55.373 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.442e-05 2024-04-19T01:04:55.373 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.373 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.373 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:55.373 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.567114+0000", 2024-04-19T01:04:55.374 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.026701643000000001 2024-04-19T01:04:55.374 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.374 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.374 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:55.374 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.567208+0000", 2024-04-19T01:04:55.374 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4061999999999995e-05 2024-04-19T01:04:55.374 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:55.374 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:55.374 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:55.374 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.374 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.374 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:419 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [read 0~262144,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e59)", 2024-04-19T01:04:55.374 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:29.527509+0000", 2024-04-19T01:04:55.374 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 25.762233688999999, 2024-04-19T01:04:55.375 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.015116167, 2024-04-19T01:04:55.375 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:55.375 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:55.375 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-04-19T01:04:55.375 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:55.375 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:04:55.375 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:55.375 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 419 2024-04-19T01:04:55.375 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.375 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:55.375 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.375 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:55.375 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527509+0000", 2024-04-19T01:04:55.375 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:55.375 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.376 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.376 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:55.376 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527509+0000", 2024-04-19T01:04:55.376 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:55.376 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.376 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.376 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:55.376 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527511+0000", 2024-04-19T01:04:55.376 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4869999999999999e-06 2024-04-19T01:04:55.376 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.376 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.376 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:55.376 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527520+0000", 2024-04-19T01:04:55.376 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3689999999999996e-06 2024-04-19T01:04:55.377 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.377 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.377 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:55.377 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527552+0000", 2024-04-19T01:04:55.377 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1316999999999998e-05 2024-04-19T01:04:55.377 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.377 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.377 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:55.377 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527566+0000", 2024-04-19T01:04:55.377 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4893e-05 2024-04-19T01:04:55.377 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.377 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.377 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:55.377 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527787+0000", 2024-04-19T01:04:55.377 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00022044999999999999 2024-04-19T01:04:55.378 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.378 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.378 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:04:55.378 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527803+0000", 2024-04-19T01:04:55.378 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6076e-05 2024-04-19T01:04:55.378 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.378 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.378 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:55.378 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.534575+0000", 2024-04-19T01:04:55.378 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0067724559999999996 2024-04-19T01:04:55.378 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.378 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.378 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:55.378 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.534811+0000", 2024-04-19T01:04:55.379 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00023533000000000001 2024-04-19T01:04:55.379 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.379 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.379 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:55.379 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.542626+0000", 2024-04-19T01:04:55.379 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0078147890000000008 2024-04-19T01:04:55.379 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:55.379 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:55.379 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:55.379 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.379 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.379 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:420 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [read 0~262144,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e59)", 2024-04-19T01:04:55.379 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:29.527610+0000", 2024-04-19T01:04:55.379 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 25.762133124000002, 2024-04-19T01:04:55.380 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.018555769999999999, 2024-04-19T01:04:55.380 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:55.380 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:55.380 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-04-19T01:04:55.380 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:55.380 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:04:55.380 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:55.380 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 420 2024-04-19T01:04:55.380 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.380 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:55.380 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.380 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:55.380 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527610+0000", 2024-04-19T01:04:55.380 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:55.381 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.381 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.381 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:55.381 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527610+0000", 2024-04-19T01:04:55.381 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:55.381 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.381 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.381 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:55.381 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527612+0000", 2024-04-19T01:04:55.381 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.967e-06 2024-04-19T01:04:55.381 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.381 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.381 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:55.381 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527642+0000", 2024-04-19T01:04:55.381 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9589000000000001e-05 2024-04-19T01:04:55.382 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.382 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.382 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:55.382 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527664+0000", 2024-04-19T01:04:55.382 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2143000000000001e-05 2024-04-19T01:04:55.382 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.382 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.382 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:55.382 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527679+0000", 2024-04-19T01:04:55.382 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5e-05 2024-04-19T01:04:55.382 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.382 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.382 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:55.382 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527873+0000", 2024-04-19T01:04:55.383 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000194537 2024-04-19T01:04:55.383 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.383 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.383 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:04:55.383 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527889+0000", 2024-04-19T01:04:55.383 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5603e-05 2024-04-19T01:04:55.383 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.383 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.383 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:55.383 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.535626+0000", 2024-04-19T01:04:55.383 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0077369099999999996 2024-04-19T01:04:55.383 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.383 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.383 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:55.384 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.535845+0000", 2024-04-19T01:04:55.384 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000218827 2024-04-19T01:04:55.384 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.384 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.384 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:55.384 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.546166+0000", 2024-04-19T01:04:55.384 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.010321194000000001 2024-04-19T01:04:55.384 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:55.384 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:55.384 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:55.384 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.384 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.384 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.8s3 60/31 ECSubWrite(tid=225, reqid=client.4604.0:433, at_version=60'43, trim_to=0'0, roll_forward_to=59'41))", 2024-04-19T01:04:55.384 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:30.569755+0000", 2024-04-19T01:04:55.385 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 24.719988503, 2024-04-19T01:04:55.385 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.026016885999999999, 2024-04-19T01:04:55.385 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:55.385 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:55.385 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:55.385 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:55.385 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.385 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:55.385 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.569755+0000", 2024-04-19T01:04:55.385 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:55.385 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.385 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.385 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:55.385 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.569755+0000", 2024-04-19T01:04:55.385 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:55.386 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.386 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.386 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:55.386 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.569755+0000", 2024-04-19T01:04:55.386 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6599999999999996e-07 2024-04-19T01:04:55.387 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.387 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.387 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:55.387 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.570112+0000", 2024-04-19T01:04:55.387 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00035699100000000002 2024-04-19T01:04:55.387 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.387 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.387 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:55.387 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.570159+0000", 2024-04-19T01:04:55.387 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7312999999999999e-05 2024-04-19T01:04:55.387 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.388 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.388 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:55.388 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.570184+0000", 2024-04-19T01:04:55.388 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4593e-05 2024-04-19T01:04:55.388 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.388 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.388 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:55.388 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.570362+0000", 2024-04-19T01:04:55.388 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017814400000000001 2024-04-19T01:04:55.388 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.388 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.388 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:55.388 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.570381+0000", 2024-04-19T01:04:55.388 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8369999999999999e-05 2024-04-19T01:04:55.389 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.389 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.389 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:55.389 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.595729+0000", 2024-04-19T01:04:55.389 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.025348186000000002 2024-04-19T01:04:55.389 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.389 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.389 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:55.389 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.595771+0000", 2024-04-19T01:04:55.389 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2722999999999997e-05 2024-04-19T01:04:55.389 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:55.389 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:55.389 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:55.389 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.390 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.390 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.4s4 71/31 ECSubWrite(tid=89, reqid=client.4604.0:518, at_version=71'22, trim_to=0'0, roll_forward_to=52'20))", 2024-04-19T01:04:55.390 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:41.542817+0000", 2024-04-19T01:04:55.390 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 13.746926283000001, 2024-04-19T01:04:55.390 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.068188129, 2024-04-19T01:04:55.391 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:55.391 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:55.391 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:55.391 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:55.391 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.391 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:55.391 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.542817+0000", 2024-04-19T01:04:55.391 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:55.391 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.391 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.391 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:55.391 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.542817+0000", 2024-04-19T01:04:55.391 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:55.392 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.392 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.395 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:55.395 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.542819+0000", 2024-04-19T01:04:55.395 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.283e-06 2024-04-19T01:04:55.395 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.395 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.395 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:55.395 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.545764+0000", 2024-04-19T01:04:55.395 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0029444530000000001 2024-04-19T01:04:55.395 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.396 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.396 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:55.396 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.545797+0000", 2024-04-19T01:04:55.396 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3203e-05 2024-04-19T01:04:55.396 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.396 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.396 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:55.396 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.545821+0000", 2024-04-19T01:04:55.396 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4436000000000001e-05 2024-04-19T01:04:55.396 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.396 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.396 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:55.396 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.545998+0000", 2024-04-19T01:04:55.396 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017682099999999999 2024-04-19T01:04:55.397 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.397 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.397 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:55.397 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.546048+0000", 2024-04-19T01:04:55.397 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0423000000000002e-05 2024-04-19T01:04:55.397 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.397 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.397 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:55.397 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.610960+0000", 2024-04-19T01:04:55.398 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.064912005999999994 2024-04-19T01:04:55.398 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.398 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.398 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:55.398 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.611005+0000", 2024-04-19T01:04:55.398 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4503999999999999e-05 2024-04-19T01:04:55.398 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:55.398 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:55.398 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:55.398 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.398 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.398 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:612 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [append 0~704512] snapc 1b=[1b,19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-04-19T01:04:55.398 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:50.652703+0000", 2024-04-19T01:04:55.399 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 4.6370400700000003, 2024-04-19T01:04:55.399 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.035173242, 2024-04-19T01:04:55.399 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:55.399 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:55.399 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:55.399 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:55.399 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:04:55.399 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:55.399 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 612 2024-04-19T01:04:55.399 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.399 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:55.399 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.399 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:55.399 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.652703+0000", 2024-04-19T01:04:55.400 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:55.400 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.400 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.400 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:55.400 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.652703+0000", 2024-04-19T01:04:55.400 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:55.400 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.400 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.400 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:55.400 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.652706+0000", 2024-04-19T01:04:55.400 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4070000000000002e-06 2024-04-19T01:04:55.401 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.401 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.401 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:55.401 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.653472+0000", 2024-04-19T01:04:55.401 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00076519800000000003 2024-04-19T01:04:55.401 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.401 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.401 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:55.401 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.653507+0000", 2024-04-19T01:04:55.401 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5525999999999998e-05 2024-04-19T01:04:55.401 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.402 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.402 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:55.402 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.653525+0000", 2024-04-19T01:04:55.402 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7329e-05 2024-04-19T01:04:55.402 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.402 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.402 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:55.402 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.653662+0000", 2024-04-19T01:04:55.402 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013767900000000001 2024-04-19T01:04:55.402 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.402 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.402 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op must wait for map", 2024-04-19T01:04:55.402 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.653675+0000", 2024-04-19T01:04:55.403 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2723e-05 2024-04-19T01:04:55.403 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.403 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.403 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:55.403 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.658036+0000", 2024-04-19T01:04:55.403 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0043615820000000001 2024-04-19T01:04:55.403 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.403 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.403 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:55.403 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.658195+0000", 2024-04-19T01:04:55.403 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000158448 2024-04-19T01:04:55.403 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.403 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.403 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:55.404 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.660614+0000", 2024-04-19T01:04:55.404 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0024192749999999998 2024-04-19T01:04:55.404 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.404 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.404 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:55.404 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.687106+0000", 2024-04-19T01:04:55.404 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.026491879999999999 2024-04-19T01:04:55.404 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.404 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.404 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:55.404 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.687703+0000", 2024-04-19T01:04:55.404 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00059716299999999995 2024-04-19T01:04:55.404 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.404 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.404 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:55.405 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.687876+0000", 2024-04-19T01:04:55.405 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017303199999999999 2024-04-19T01:04:55.405 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:55.405 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:55.405 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:55.405 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.405 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.405 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:613 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [append 0~655360] snapc 1b=[1b,19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-04-19T01:04:55.405 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:50.661555+0000", 2024-04-19T01:04:55.405 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 4.6281878340000002, 2024-04-19T01:04:55.406 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.038292557999999997, 2024-04-19T01:04:55.406 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:55.406 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:55.406 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:55.406 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:55.406 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:04:55.406 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:55.406 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 613 2024-04-19T01:04:55.406 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.406 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:55.406 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.410 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:55.410 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.661555+0000", 2024-04-19T01:04:55.410 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:55.410 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.410 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.411 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:55.411 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.661555+0000", 2024-04-19T01:04:55.411 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:55.411 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.411 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.411 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:55.411 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.661559+0000", 2024-04-19T01:04:55.412 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.337e-06 2024-04-19T01:04:55.412 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.412 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.412 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:55.412 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.662402+0000", 2024-04-19T01:04:55.412 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00084358700000000003 2024-04-19T01:04:55.412 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.412 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.412 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:55.412 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.662434+0000", 2024-04-19T01:04:55.412 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2223e-05 2024-04-19T01:04:55.412 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.412 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.412 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:55.413 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.662458+0000", 2024-04-19T01:04:55.413 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3598999999999999e-05 2024-04-19T01:04:55.413 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.413 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.413 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:55.413 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.673646+0000", 2024-04-19T01:04:55.413 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.011187852999999999 2024-04-19T01:04:55.413 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.413 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.413 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:55.413 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.673820+0000", 2024-04-19T01:04:55.413 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017450199999999999 2024-04-19T01:04:55.413 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.413 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.413 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:55.414 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.675719+0000", 2024-04-19T01:04:55.414 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001898757 2024-04-19T01:04:55.414 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.414 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.414 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:55.414 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.687984+0000", 2024-04-19T01:04:55.414 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.012265009 2024-04-19T01:04:55.414 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.414 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.414 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:55.414 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.699734+0000", 2024-04-19T01:04:55.414 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.011749426 2024-04-19T01:04:55.414 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.414 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.415 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:55.415 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.699848+0000", 2024-04-19T01:04:55.415 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000114265 2024-04-19T01:04:55.415 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:55.415 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:55.415 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:55.415 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.415 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.415 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 80/31 ECSubWriteReply(tid=63, last_complete=80'51, committed=1, applied=1))", 2024-04-19T01:04:55.415 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:50.667974+0000", 2024-04-19T01:04:55.415 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 4.6217690449999997, 2024-04-19T01:04:55.415 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.018952044000000001, 2024-04-19T01:04:55.415 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:55.415 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:55.416 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:55.416 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:55.416 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.416 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:55.416 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.667974+0000", 2024-04-19T01:04:55.416 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:55.416 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.416 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.416 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:55.416 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.667974+0000", 2024-04-19T01:04:55.416 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:55.416 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.416 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.416 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:55.417 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.667976+0000", 2024-04-19T01:04:55.417 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8479999999999999e-06 2024-04-19T01:04:55.417 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.417 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.417 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:55.417 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.667990+0000", 2024-04-19T01:04:55.417 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.403e-05 2024-04-19T01:04:55.417 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.417 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.417 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:55.417 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.668020+0000", 2024-04-19T01:04:55.417 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0222e-05 2024-04-19T01:04:55.417 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.418 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.418 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:55.418 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.668037+0000", 2024-04-19T01:04:55.418 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6424999999999999e-05 2024-04-19T01:04:55.418 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.418 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.418 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:55.418 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.686874+0000", 2024-04-19T01:04:55.418 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.018837596000000002 2024-04-19T01:04:55.418 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.418 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.418 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:55.418 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.686926+0000", 2024-04-19T01:04:55.418 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.1922999999999997e-05 2024-04-19T01:04:55.419 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:55.419 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:55.419 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:55.419 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.419 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.419 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:614 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [append 0~507904] snapc 1b=[1b,19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-04-19T01:04:55.419 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:50.668341+0000", 2024-04-19T01:04:55.419 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 4.6214025989999996, 2024-04-19T01:04:55.419 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.032677876000000002, 2024-04-19T01:04:55.419 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:55.419 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:55.419 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:55.419 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:55.419 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:04:55.420 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:55.420 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 614 2024-04-19T01:04:55.420 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.420 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:55.420 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.420 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:55.420 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.668341+0000", 2024-04-19T01:04:55.420 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:55.421 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.421 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.421 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:55.421 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.668341+0000", 2024-04-19T01:04:55.421 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:55.421 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.421 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.421 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:55.421 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.668343+0000", 2024-04-19T01:04:55.422 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2759999999999999e-06 2024-04-19T01:04:55.422 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.422 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.422 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:55.422 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.668923+0000", 2024-04-19T01:04:55.422 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00058019699999999996 2024-04-19T01:04:55.422 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.422 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.422 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:55.422 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.668956+0000", 2024-04-19T01:04:55.422 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2539000000000001e-05 2024-04-19T01:04:55.422 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.423 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.423 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:55.423 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.668977+0000", 2024-04-19T01:04:55.423 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1903e-05 2024-04-19T01:04:55.423 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.423 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.423 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:55.423 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.688297+0000", 2024-04-19T01:04:55.423 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.019319106999999999 2024-04-19T01:04:55.423 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.423 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.423 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:55.423 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.688460+0000", 2024-04-19T01:04:55.423 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016338799999999999 2024-04-19T01:04:55.423 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.424 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.424 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:55.424 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.689733+0000", 2024-04-19T01:04:55.424 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001273499 2024-04-19T01:04:55.424 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.424 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.424 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:55.424 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.699996+0000", 2024-04-19T01:04:55.424 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.010263042 2024-04-19T01:04:55.424 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.424 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.424 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:55.424 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.700905+0000", 2024-04-19T01:04:55.424 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00090875999999999999 2024-04-19T01:04:55.425 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.425 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.425 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:55.425 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.701018+0000", 2024-04-19T01:04:55.425 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000113165 2024-04-19T01:04:55.425 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:55.425 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:55.425 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:55.425 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.425 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.425 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:615 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [setxattr _header (57) in=7b] snapc 1b=[1b,19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-04-19T01:04:55.425 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:50.669014+0000", 2024-04-19T01:04:55.425 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 4.6207291210000001, 2024-04-19T01:04:55.425 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.032983169999999999, 2024-04-19T01:04:55.425 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:55.426 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:55.426 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:55.426 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:55.426 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:04:55.426 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:55.426 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 615 2024-04-19T01:04:55.426 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.426 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:55.426 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.426 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:55.426 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669014+0000", 2024-04-19T01:04:55.426 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:55.426 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.426 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.427 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:55.427 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669014+0000", 2024-04-19T01:04:55.427 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:55.427 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.427 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.427 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:55.427 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669016+0000", 2024-04-19T01:04:55.427 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3439999999999999e-06 2024-04-19T01:04:55.427 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.427 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.427 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:55.427 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669039+0000", 2024-04-19T01:04:55.427 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3008999999999999e-05 2024-04-19T01:04:55.427 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.428 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.428 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:55.428 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669057+0000", 2024-04-19T01:04:55.428 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.749e-05 2024-04-19T01:04:55.428 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.428 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.428 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:55.428 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669073+0000", 2024-04-19T01:04:55.428 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6492999999999999e-05 2024-04-19T01:04:55.428 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.428 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.428 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:55.428 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.698263+0000", 2024-04-19T01:04:55.428 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.029189397999999998 2024-04-19T01:04:55.428 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.429 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.429 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:55.429 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.698412+0000", 2024-04-19T01:04:55.429 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014930499999999999 2024-04-19T01:04:55.429 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.429 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.429 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:55.429 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.699113+0000", 2024-04-19T01:04:55.429 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00070141799999999999 2024-04-19T01:04:55.429 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.429 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.429 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:55.429 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.701045+0000", 2024-04-19T01:04:55.429 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001931667 2024-04-19T01:04:55.430 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.430 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.430 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:04:55.430 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.701847+0000", 2024-04-19T01:04:55.430 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00080155099999999996 2024-04-19T01:04:55.430 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.430 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.430 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:55.430 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.701997+0000", 2024-04-19T01:04:55.430 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000150495 2024-04-19T01:04:55.430 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:55.430 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:55.430 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:55.430 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.431 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.431 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:616 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e80)", 2024-04-19T01:04:55.431 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:50.669097+0000", 2024-04-19T01:04:55.431 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 4.6206465850000003, 2024-04-19T01:04:55.431 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.038907733999999999, 2024-04-19T01:04:55.431 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:55.431 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:55.432 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-04-19T01:04:55.432 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:04:55.432 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:04:55.432 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:55.432 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 616 2024-04-19T01:04:55.433 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.433 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:55.433 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.433 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:55.433 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669097+0000", 2024-04-19T01:04:55.433 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:55.433 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.433 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.433 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:55.433 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669097+0000", 2024-04-19T01:04:55.433 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:55.433 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.433 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.433 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:55.434 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669098+0000", 2024-04-19T01:04:55.434 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.587e-06 2024-04-19T01:04:55.434 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.434 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.434 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:55.434 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669105+0000", 2024-04-19T01:04:55.434 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3660000000000003e-06 2024-04-19T01:04:55.434 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.434 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.434 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:55.434 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669130+0000", 2024-04-19T01:04:55.434 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4377000000000001e-05 2024-04-19T01:04:55.434 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.434 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.435 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:55.435 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669146+0000", 2024-04-19T01:04:55.435 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6399e-05 2024-04-19T01:04:55.435 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.435 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.435 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:55.435 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.702705+0000", 2024-04-19T01:04:55.435 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.033558534000000001 2024-04-19T01:04:55.435 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.435 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.435 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:04:55.435 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.702962+0000", 2024-04-19T01:04:55.435 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00025750700000000003 2024-04-19T01:04:55.435 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.436 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.436 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:55.436 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.708004+0000", 2024-04-19T01:04:55.436 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0050419640000000003 2024-04-19T01:04:55.436 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:55.436 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:55.436 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:55.436 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.436 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.436 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 80/31 ECSubWriteReply(tid=63, last_complete=80'51, committed=1, applied=1))", 2024-04-19T01:04:55.436 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:50.671957+0000", 2024-04-19T01:04:55.436 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 4.6177863930000003, 2024-04-19T01:04:55.436 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.015086721000000001, 2024-04-19T01:04:55.437 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:55.437 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:55.437 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:55.437 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:55.437 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.437 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:55.437 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.671957+0000", 2024-04-19T01:04:55.437 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:55.437 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.437 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.437 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:55.437 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.671957+0000", 2024-04-19T01:04:55.437 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:55.437 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.437 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.438 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:55.438 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.671958+0000", 2024-04-19T01:04:55.438 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2669999999999999e-06 2024-04-19T01:04:55.438 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.438 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.438 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:55.438 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.671969+0000", 2024-04-19T01:04:55.438 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1272e-05 2024-04-19T01:04:55.438 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.438 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.438 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:55.438 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.671990+0000", 2024-04-19T01:04:55.438 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0245e-05 2024-04-19T01:04:55.439 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.439 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.439 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:55.439 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.672000+0000", 2024-04-19T01:04:55.439 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.029e-05 2024-04-19T01:04:55.439 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.439 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.439 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:55.439 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.686989+0000", 2024-04-19T01:04:55.439 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.014989077 2024-04-19T01:04:55.439 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.439 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.439 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:55.439 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.687043+0000", 2024-04-19T01:04:55.439 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4570000000000001e-05 2024-04-19T01:04:55.440 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:55.440 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:55.440 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:55.440 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.440 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.440 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 80/31 ECSubWriteReply(tid=63, last_complete=80'51, committed=1, applied=1))", 2024-04-19T01:04:55.440 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:50.672096+0000", 2024-04-19T01:04:55.440 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 4.6176473869999999, 2024-04-19T01:04:55.440 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.015013018, 2024-04-19T01:04:55.440 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:55.440 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:55.440 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:55.440 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:55.440 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.441 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:55.441 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.672096+0000", 2024-04-19T01:04:55.441 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:55.441 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.441 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.441 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:55.441 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.672096+0000", 2024-04-19T01:04:55.441 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:55.441 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.441 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.441 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:55.441 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.672098+0000", 2024-04-19T01:04:55.442 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0099999999999998e-06 2024-04-19T01:04:55.442 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.442 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.442 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:55.442 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.672108+0000", 2024-04-19T01:04:55.442 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9539999999999999e-06 2024-04-19T01:04:55.442 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.442 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.443 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:55.443 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.672136+0000", 2024-04-19T01:04:55.443 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8762e-05 2024-04-19T01:04:55.443 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.443 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.443 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:55.443 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.672151+0000", 2024-04-19T01:04:55.443 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4562999999999999e-05 2024-04-19T01:04:55.443 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.443 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.443 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:55.443 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.687079+0000", 2024-04-19T01:04:55.443 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.014928182 2024-04-19T01:04:55.443 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.444 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.444 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:55.444 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.687109+0000", 2024-04-19T01:04:55.444 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9547e-05 2024-04-19T01:04:55.444 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:55.444 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:55.444 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:55.444 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.444 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.444 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.4s4 81/31 ECSubWrite(tid=117, reqid=client.4604.0:618, at_version=81'41, trim_to=0'0, roll_forward_to=79'39))", 2024-04-19T01:04:55.444 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:51.536707+0000", 2024-04-19T01:04:55.444 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 3.7530360329999999, 2024-04-19T01:04:55.444 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.019218485, 2024-04-19T01:04:55.444 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:04:55.445 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:04:55.445 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:04:55.445 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:04:55.445 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.445 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:04:55.445 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.536707+0000", 2024-04-19T01:04:55.445 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:55.445 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.445 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.445 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:04:55.445 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.536707+0000", 2024-04-19T01:04:55.445 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:04:55.445 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.445 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.445 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:04:55.446 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.536720+0000", 2024-04-19T01:04:55.446 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3267e-05 2024-04-19T01:04:55.446 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.446 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.446 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:04:55.446 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.537717+0000", 2024-04-19T01:04:55.446 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00099714199999999991 2024-04-19T01:04:55.446 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.446 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.446 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:04:55.446 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.537750+0000", 2024-04-19T01:04:55.446 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2558999999999998e-05 2024-04-19T01:04:55.446 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.446 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.447 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:04:55.447 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.537773+0000", 2024-04-19T01:04:55.447 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3013000000000001e-05 2024-04-19T01:04:55.447 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.447 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.447 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:04:55.447 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.537958+0000", 2024-04-19T01:04:55.447 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00018457400000000001 2024-04-19T01:04:55.447 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.447 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.447 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:04:55.447 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.537976+0000", 2024-04-19T01:04:55.447 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8734000000000001e-05 2024-04-19T01:04:55.447 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.448 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.448 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:04:55.448 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.555848+0000", 2024-04-19T01:04:55.448 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.017871617999999999 2024-04-19T01:04:55.448 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:04:55.448 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:04:55.448 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:04:55.448 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.555926+0000", 2024-04-19T01:04:55.448 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7577999999999995e-05 2024-04-19T01:04:55.448 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:55.448 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:55.448 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:55.448 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:04:55.448 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:04:55.449 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:04:55.450 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:04:55.448+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:04:55.450 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-19T01:04:55.534 INFO:tasks.rados.rados.0.smithi155.stdout:268: done (1 left) 2024-04-19T01:04:55.534 INFO:tasks.rados.rados.0.smithi155.stdout:269: snap_remove snap 23 2024-04-19T01:04:55.549 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-19T01:04:55.552 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:04:55.550+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:04:55.602 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:55.603 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:04:55.603 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2024-04-19T01:04:55.603 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:55.618 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-19T01:04:55.652 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:04:55.651+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:04:55.742 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:55.742 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:04:55.742 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2024-04-19T01:04:55.742 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2024-04-19T01:04:55.742 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:55.758 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-19T01:04:55.759 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:04:55.753+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:04:55.859 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:04:55.856+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:04:55.885 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:55.885 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2024-04-19T01:04:55.886 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2024-04-19T01:04:55.886 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-04-19T01:04:55.886 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.886 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:172 3.cs0 3:356739ec:::smithi15528762-4:head [create,setxattr aSrcO`NnS[qadSjpQT[PSmfh\\PfYagpctmbQ_kXi (1753) in=40b,setxattr hratdsb^VNONN^^acUkPeXNZohcdVkmXXhdTeofs (2805) in=40b,setxattr abOo``aaqamjUn`hc`Qde]XU\\lqYVrmVdcimNs_a (35) in=40b,setxattr ZheopalZpQPnebTbena`gaWXtmaSTdQkji`aZYOi (34) in=40b,setxattr katXYockjtYtieO]`jgpSrZZ\\mq`gOYplPOVal`Y (842) in=40b,setxattr [^]lbsa\\UX[na]akS\\hYcdqU^]aWe_[U[OlaQXUY (1894) in=40b,setxattr We][_Xheab\\Ylg`q^hsi`XTX_tQS`Ng\\]aqZTnV[ (1473) in=40b,setxattr TTRSo^W`XoqVWpSjRgc[adm`ROSSVYUNlsilh\\kS (911) in=40b,setxattr bqbUQROpngcT`ndhrZQP`iraoUgs[ZZatacnZoc_ (2876) in=40b,setxattr qPPSibdg_e\\Y_V`kTQO^\\`_jr^jt\\aiadkepqYUj (3576) in=40b,setxattr q`N\\bPOqejbpq`QZVQfWgjScjfahlgXWodnRReWS (1017) in=40b,setxattr `pdT]PSaeeZ^jhWYc]iO`^tRXWcjedgQdke^adpf (7928) in=40b,setxattr matSmfe^casVRfWbNqkt_mZNWbhhh`\\Waf\\djeca (7157) in=40b,setxattr ]SWX\\hrhracj[TeSRYP[bZ[[lSSRghmfarTYpp]U (735) in=40b,setxattr WrnQrVa (13964) in=7b,setxattr PcP_]XU]rhapTs\\X`ojNmm_Qsjkkjecf`VgrdTrR (1542) in=40b,setxattr o^RO^oeZlji]\\\\UhVsnb]t_WXomaRr^higjSUShQ (1016) in=40b,setxattr Z_ofVVcfZdUV[UaRnTV_Ni^_rRNl`\\kt`]aaaoed (701) in=40b,setxattr X[m^[`bVlbgi``^mWZ]cl[S_nn[Ok\\ldS\\_h]]qk (1999) in=40b,setxattr fpmQVVnVPSmZmt`OeaT`adhoNjhpOb[`\\aPWQ^Rd (700) in=40b] snapc 5=[5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-19T01:04:55.886 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:08.804031+0000", 2024-04-19T01:04:55.886 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 47.080729122999998, 2024-04-19T01:04:55.886 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014732528999999999, 2024-04-19T01:04:55.886 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:55.886 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:55.886 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:55.886 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:55.886 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:55.886 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:55.887 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 172 2024-04-19T01:04:55.887 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.887 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:55.887 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.887 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:55.887 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.804031+0000", 2024-04-19T01:04:55.887 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:55.887 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.887 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.887 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:55.887 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.804031+0000", 2024-04-19T01:04:55.887 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:55.887 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.888 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.888 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:55.888 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.804034+0000", 2024-04-19T01:04:55.888 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5389999999999999e-06 2024-04-19T01:04:55.888 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.888 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.888 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:55.888 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.804179+0000", 2024-04-19T01:04:55.888 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014485600000000001 2024-04-19T01:04:55.888 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.888 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.888 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:55.888 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.804208+0000", 2024-04-19T01:04:55.889 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8615999999999998e-05 2024-04-19T01:04:55.889 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.889 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.889 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:55.889 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.804223+0000", 2024-04-19T01:04:55.889 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5244e-05 2024-04-19T01:04:55.889 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.889 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.889 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:55.889 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.804362+0000", 2024-04-19T01:04:55.889 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013886000000000001 2024-04-19T01:04:55.889 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.889 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.889 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op must wait for map", 2024-04-19T01:04:55.890 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.804377+0000", 2024-04-19T01:04:55.890 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4832e-05 2024-04-19T01:04:55.890 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.890 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.890 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:55.890 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.809596+0000", 2024-04-19T01:04:55.890 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0052189250000000001 2024-04-19T01:04:55.890 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.890 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.890 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:55.890 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.809925+0000", 2024-04-19T01:04:55.890 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00032908900000000002 2024-04-19T01:04:55.890 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.891 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.891 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:55.891 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.813383+0000", 2024-04-19T01:04:55.891 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0034579929999999999 2024-04-19T01:04:55.891 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.891 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.891 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:55.891 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.815667+0000", 2024-04-19T01:04:55.891 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.002284632 2024-04-19T01:04:55.891 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.891 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.891 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:55.892 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.818639+0000", 2024-04-19T01:04:55.892 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.002972003 2024-04-19T01:04:55.892 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.892 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.892 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:55.892 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:08.818763+0000", 2024-04-19T01:04:55.892 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012394000000000001 2024-04-19T01:04:55.892 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:55.892 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:55.892 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:55.892 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.892 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.892 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:232 3.cs0 3:3233f066:::smithi15528762-50:2 [read 0~835584,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:55.892 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.838358+0000", 2024-04-19T01:04:55.893 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 42.046402413999999, 2024-04-19T01:04:55.893 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011277769, 2024-04-19T01:04:55.893 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:55.893 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:55.893 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:55.893 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:55.893 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:55.893 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:55.893 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 232 2024-04-19T01:04:55.893 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.893 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:55.893 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.893 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:55.894 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838358+0000", 2024-04-19T01:04:55.894 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:55.894 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.894 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.894 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:55.894 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838358+0000", 2024-04-19T01:04:55.894 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:55.894 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.894 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.894 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:55.894 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838360+0000", 2024-04-19T01:04:55.894 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8770000000000002e-06 2024-04-19T01:04:55.894 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.894 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.895 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:55.895 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838388+0000", 2024-04-19T01:04:55.895 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7282000000000001e-05 2024-04-19T01:04:55.895 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.895 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.895 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:55.895 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838427+0000", 2024-04-19T01:04:55.895 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9242999999999997e-05 2024-04-19T01:04:55.895 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.895 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.895 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:55.896 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838444+0000", 2024-04-19T01:04:55.896 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7127000000000001e-05 2024-04-19T01:04:55.896 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.896 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.896 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:55.896 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838641+0000", 2024-04-19T01:04:55.896 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00019704499999999999 2024-04-19T01:04:55.896 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.896 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.896 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op must wait for map", 2024-04-19T01:04:55.896 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838653+0000", 2024-04-19T01:04:55.896 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2187999999999999e-05 2024-04-19T01:04:55.896 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.896 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.897 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:55.897 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.842741+0000", 2024-04-19T01:04:55.897 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0040879879999999999 2024-04-19T01:04:55.897 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.897 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.897 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:55.897 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.842898+0000", 2024-04-19T01:04:55.901 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015647299999999999 2024-04-19T01:04:55.901 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.901 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.901 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:55.901 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.849635+0000", 2024-04-19T01:04:55.901 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0067375459999999996 2024-04-19T01:04:55.901 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:55.901 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:55.901 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:55.901 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.901 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.902 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:233 3.cs0 3:3233f066:::smithi15528762-50:2 [read 0~835584,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:55.902 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.838487+0000", 2024-04-19T01:04:55.902 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 42.046273399999997, 2024-04-19T01:04:55.902 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013873264, 2024-04-19T01:04:55.902 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:55.902 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:55.902 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:55.902 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:55.902 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:55.902 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:55.902 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 233 2024-04-19T01:04:55.902 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.903 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:55.903 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.903 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:55.903 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838487+0000", 2024-04-19T01:04:55.903 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:55.903 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.903 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.903 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:55.903 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838487+0000", 2024-04-19T01:04:55.903 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:55.903 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.911 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.911 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:55.911 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838488+0000", 2024-04-19T01:04:55.911 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0380000000000001e-06 2024-04-19T01:04:55.911 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.911 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.911 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:55.911 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838495+0000", 2024-04-19T01:04:55.912 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.8299999999999996e-06 2024-04-19T01:04:55.912 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.912 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.912 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:55.912 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838509+0000", 2024-04-19T01:04:55.912 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3954000000000001e-05 2024-04-19T01:04:55.912 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.912 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.912 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:55.912 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838519+0000", 2024-04-19T01:04:55.912 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.9729999999999992e-06 2024-04-19T01:04:55.912 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.912 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.912 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:55.913 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838755+0000", 2024-04-19T01:04:55.913 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00023589999999999999 2024-04-19T01:04:55.913 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.913 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.913 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:04:55.913 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838764+0000", 2024-04-19T01:04:55.913 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.0429999999999996e-06 2024-04-19T01:04:55.913 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.913 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.913 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:55.913 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.844563+0000", 2024-04-19T01:04:55.913 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0057983749999999997 2024-04-19T01:04:55.914 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.914 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.914 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:55.914 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.844897+0000", 2024-04-19T01:04:55.914 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00033450199999999998 2024-04-19T01:04:55.914 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.914 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.914 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:55.914 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.852360+0000", 2024-04-19T01:04:55.914 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.007462649 2024-04-19T01:04:55.914 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:55.914 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:55.915 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:55.915 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.915 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.915 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:234 3.cs0 3:3233f066:::smithi15528762-50:2 [read 0~835584,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:55.915 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.838536+0000", 2024-04-19T01:04:55.915 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 42.046223914000002, 2024-04-19T01:04:55.915 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.016682111999999999, 2024-04-19T01:04:55.915 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:55.915 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:55.915 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:55.915 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:55.915 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:55.915 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:55.915 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 234 2024-04-19T01:04:55.916 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.916 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:55.916 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.916 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:55.916 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838536+0000", 2024-04-19T01:04:55.916 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:55.916 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.916 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.917 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:55.917 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838536+0000", 2024-04-19T01:04:55.917 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:55.917 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.917 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.917 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:55.917 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838537+0000", 2024-04-19T01:04:55.917 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1200000000000002e-07 2024-04-19T01:04:55.917 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.917 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.917 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:55.917 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838542+0000", 2024-04-19T01:04:55.917 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3789999999999997e-06 2024-04-19T01:04:55.917 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.918 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.918 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:55.918 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838553+0000", 2024-04-19T01:04:55.918 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0986e-05 2024-04-19T01:04:55.918 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.918 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.918 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:55.918 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.838564+0000", 2024-04-19T01:04:55.918 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0895e-05 2024-04-19T01:04:55.918 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.918 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.918 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:55.918 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.840056+0000", 2024-04-19T01:04:55.918 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014917229999999999 2024-04-19T01:04:55.919 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.919 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.919 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:04:55.919 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.840066+0000", 2024-04-19T01:04:55.919 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.9329999999999996e-06 2024-04-19T01:04:55.919 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.919 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.919 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:55.919 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.846974+0000", 2024-04-19T01:04:55.919 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0069077770000000004 2024-04-19T01:04:55.919 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.919 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.919 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:55.919 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.847132+0000", 2024-04-19T01:04:55.920 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015876400000000001 2024-04-19T01:04:55.920 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.920 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.920 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:55.920 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.855218+0000", 2024-04-19T01:04:55.920 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0080858430000000005 2024-04-19T01:04:55.920 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:55.920 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:55.920 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:55.920 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.920 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.920 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:263 3.fs0 3:f8a60714:::smithi15528762-14:head [append 0~720896] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:55.920 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.990535+0000", 2024-04-19T01:04:55.920 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 41.894224628000003, 2024-04-19T01:04:55.920 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.043978092000000003, 2024-04-19T01:04:55.921 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:55.921 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:55.921 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:55.921 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:55.921 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:55.921 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:55.921 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 263 2024-04-19T01:04:55.921 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.921 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:55.921 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.921 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:55.921 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.990535+0000", 2024-04-19T01:04:55.921 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:55.921 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.922 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.922 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:55.922 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.990535+0000", 2024-04-19T01:04:55.922 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:55.922 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.922 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.922 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:55.922 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.990537+0000", 2024-04-19T01:04:55.922 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4160000000000001e-06 2024-04-19T01:04:55.922 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.922 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.922 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:55.922 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991425+0000", 2024-04-19T01:04:55.922 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00088775099999999999 2024-04-19T01:04:55.923 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.923 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.923 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:55.923 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991440+0000", 2024-04-19T01:04:55.923 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5305999999999999e-05 2024-04-19T01:04:55.923 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.923 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.923 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:55.923 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991447+0000", 2024-04-19T01:04:55.923 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.4760000000000001e-06 2024-04-19T01:04:55.923 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.923 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.923 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:55.923 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991516+0000", 2024-04-19T01:04:55.923 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.9041999999999994e-05 2024-04-19T01:04:55.924 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.924 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.924 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:55.924 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991620+0000", 2024-04-19T01:04:55.924 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000103282 2024-04-19T01:04:55.924 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.924 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.924 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:55.924 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993018+0000", 2024-04-19T01:04:55.924 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001397976 2024-04-19T01:04:55.924 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.924 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.924 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:55.924 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.004440+0000", 2024-04-19T01:04:55.925 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011422261 2024-04-19T01:04:55.925 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.925 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.925 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:55.925 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.034454+0000", 2024-04-19T01:04:55.925 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.030013782999999999 2024-04-19T01:04:55.925 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.925 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.925 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:55.925 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.034513+0000", 2024-04-19T01:04:55.925 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.9799000000000001e-05 2024-04-19T01:04:55.925 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:55.925 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:55.925 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:55.926 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.926 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.926 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:264 3.fs0 3:f8a60714:::smithi15528762-14:head [append 0~376832] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:55.926 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.992666+0000", 2024-04-19T01:04:55.926 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 41.892093856000002, 2024-04-19T01:04:55.926 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.041981560000000001, 2024-04-19T01:04:55.926 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:55.926 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:55.926 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:55.926 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:55.926 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:55.926 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:55.926 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 264 2024-04-19T01:04:55.926 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.927 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:55.927 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.927 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:55.927 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.992666+0000", 2024-04-19T01:04:55.927 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:55.927 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.927 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.927 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:55.927 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.992666+0000", 2024-04-19T01:04:55.927 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:55.927 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.927 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.927 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:55.927 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.992668+0000", 2024-04-19T01:04:55.928 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.483e-06 2024-04-19T01:04:55.928 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.928 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.928 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:55.928 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993021+0000", 2024-04-19T01:04:55.928 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000352958 2024-04-19T01:04:55.928 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.928 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.928 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:55.928 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993031+0000", 2024-04-19T01:04:55.928 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0723000000000001e-05 2024-04-19T01:04:55.928 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.928 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.928 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:55.929 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993040+0000", 2024-04-19T01:04:55.929 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.507e-06 2024-04-19T01:04:55.929 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.929 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.929 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:55.929 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.000363+0000", 2024-04-19T01:04:55.929 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0073236819999999998 2024-04-19T01:04:55.929 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.929 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.929 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:55.929 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.000482+0000", 2024-04-19T01:04:55.929 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011817299999999999 2024-04-19T01:04:55.929 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.930 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.930 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:55.930 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.001332+0000", 2024-04-19T01:04:55.930 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000849886 2024-04-19T01:04:55.930 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.930 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.930 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:55.930 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.006292+0000", 2024-04-19T01:04:55.930 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0049605090000000001 2024-04-19T01:04:55.930 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.930 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.930 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:55.930 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.034602+0000", 2024-04-19T01:04:55.930 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.028310076999999999 2024-04-19T01:04:55.931 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.931 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.931 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:55.931 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.034648+0000", 2024-04-19T01:04:55.931 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5562000000000002e-05 2024-04-19T01:04:55.931 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:55.931 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:55.931 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:55.931 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.931 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.931 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:265 3.fs0 3:f8a60714:::smithi15528762-14:head [setxattr _header (56) in=7b] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:55.931 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.993067+0000", 2024-04-19T01:04:55.931 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 41.891692618, 2024-04-19T01:04:55.931 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.041773988999999997, 2024-04-19T01:04:55.932 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:55.932 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:55.932 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:55.932 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:55.932 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:55.932 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:55.932 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 265 2024-04-19T01:04:55.932 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.932 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:55.932 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.932 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:55.932 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993067+0000", 2024-04-19T01:04:55.932 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:55.932 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.933 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.933 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:55.933 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993067+0000", 2024-04-19T01:04:55.933 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:55.933 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.933 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.933 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:55.933 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993068+0000", 2024-04-19T01:04:55.933 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2699999999999999e-07 2024-04-19T01:04:55.933 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.933 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.933 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:55.933 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993072+0000", 2024-04-19T01:04:55.933 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0539999999999996e-06 2024-04-19T01:04:55.934 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.934 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.934 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:55.934 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993080+0000", 2024-04-19T01:04:55.934 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6140000000000003e-06 2024-04-19T01:04:55.934 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.934 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.934 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:55.934 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993119+0000", 2024-04-19T01:04:55.934 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8754000000000002e-05 2024-04-19T01:04:55.935 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.935 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.935 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:55.935 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.004460+0000", 2024-04-19T01:04:55.935 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011341188 2024-04-19T01:04:55.935 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.935 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.935 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:55.935 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.004518+0000", 2024-04-19T01:04:55.935 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8306e-05 2024-04-19T01:04:55.935 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.935 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.935 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:55.935 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.004774+0000", 2024-04-19T01:04:55.936 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00025565400000000002 2024-04-19T01:04:55.936 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.936 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.936 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:55.936 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.006318+0000", 2024-04-19T01:04:55.936 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0015444409999999999 2024-04-19T01:04:55.936 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.936 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.936 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:55.936 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.034787+0000", 2024-04-19T01:04:55.936 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.028468566000000001 2024-04-19T01:04:55.936 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.936 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.936 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:55.936 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.034841+0000", 2024-04-19T01:04:55.937 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4685000000000002e-05 2024-04-19T01:04:55.937 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:55.937 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:55.937 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:55.937 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.937 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.937 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:266 3.fs0 3:f8a60714:::smithi15528762-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:04:55.937 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.993144+0000", 2024-04-19T01:04:55.937 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 41.891615565000002, 2024-04-19T01:04:55.937 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.042235719999999997, 2024-04-19T01:04:55.937 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:55.937 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:55.937 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:55.938 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:55.938 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:55.938 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:55.938 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 266 2024-04-19T01:04:55.938 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.938 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:55.938 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.938 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:55.938 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993144+0000", 2024-04-19T01:04:55.938 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:55.938 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.938 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.938 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:55.939 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993144+0000", 2024-04-19T01:04:55.939 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:55.939 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.939 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.939 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:55.939 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993145+0000", 2024-04-19T01:04:55.939 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5099999999999999e-07 2024-04-19T01:04:55.939 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.939 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.939 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:55.939 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993149+0000", 2024-04-19T01:04:55.939 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4539999999999999e-06 2024-04-19T01:04:55.939 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.939 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.940 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:55.940 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993157+0000", 2024-04-19T01:04:55.940 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.6349999999999995e-06 2024-04-19T01:04:55.940 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.940 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.940 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:55.940 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993166+0000", 2024-04-19T01:04:55.940 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2709999999999998e-06 2024-04-19T01:04:55.940 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.940 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.940 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:55.940 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.004924+0000", 2024-04-19T01:04:55.940 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011757834 2024-04-19T01:04:55.941 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.941 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.941 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:04:55.941 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.004971+0000", 2024-04-19T01:04:55.941 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6947000000000003e-05 2024-04-19T01:04:55.941 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.941 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.941 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:55.941 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.035366+0000", 2024-04-19T01:04:55.941 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.03039501 2024-04-19T01:04:55.941 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.941 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.941 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:55.942 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.035380+0000", 2024-04-19T01:04:55.942 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3918e-05 2024-04-19T01:04:55.942 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:55.942 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:55.942 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:55.942 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.942 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.942 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:317 3.fs0 3:f23900a8:::smithi15528762-6:head [append 0~475136] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:04:55.942 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.601883+0000", 2024-04-19T01:04:55.942 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 38.282876586999997, 2024-04-19T01:04:55.942 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.029134103000000001, 2024-04-19T01:04:55.942 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:55.942 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:55.943 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:55.943 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:55.943 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:55.943 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:55.943 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 317 2024-04-19T01:04:55.943 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.943 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:55.943 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.943 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:55.943 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.601883+0000", 2024-04-19T01:04:55.943 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:55.943 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.943 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.943 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:55.944 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.601883+0000", 2024-04-19T01:04:55.944 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:55.944 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.944 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.944 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:55.944 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.601886+0000", 2024-04-19T01:04:55.944 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.2709999999999999e-06 2024-04-19T01:04:55.944 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.944 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.944 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:55.944 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.602696+0000", 2024-04-19T01:04:55.944 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00080995700000000004 2024-04-19T01:04:55.944 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.944 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.945 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:55.945 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.602714+0000", 2024-04-19T01:04:55.945 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8168999999999998e-05 2024-04-19T01:04:55.945 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.945 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.945 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:55.945 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.602721+0000", 2024-04-19T01:04:55.945 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.554e-06 2024-04-19T01:04:55.945 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.945 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.945 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:55.953 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.602799+0000", 2024-04-19T01:04:55.953 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7804000000000002e-05 2024-04-19T01:04:55.953 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.965 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.965 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:55.972 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.602920+0000", 2024-04-19T01:04:55.972 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012105299999999999 2024-04-19T01:04:55.972 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.972 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.972 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:55.972 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.604215+0000", 2024-04-19T01:04:55.972 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0012944129999999999 2024-04-19T01:04:55.972 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.972 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.972 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:55.972 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.609630+0000", 2024-04-19T01:04:55.973 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0054153789999999997 2024-04-19T01:04:55.973 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.973 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.973 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:55.973 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.630924+0000", 2024-04-19T01:04:55.973 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021294306999999998 2024-04-19T01:04:55.973 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.973 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.973 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:55.973 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.631018+0000", 2024-04-19T01:04:55.973 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.3195999999999993e-05 2024-04-19T01:04:55.973 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:55.973 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:55.973 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:55.973 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.974 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.974 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:318 3.fs0 3:f23900a8:::smithi15528762-6:head [append 0~622592] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:04:55.974 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.609539+0000", 2024-04-19T01:04:55.974 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 38.275220992999998, 2024-04-19T01:04:55.974 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021603679000000001, 2024-04-19T01:04:55.974 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:55.974 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:55.974 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:55.974 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:55.974 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:55.974 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:55.974 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 318 2024-04-19T01:04:55.974 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.975 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:55.976 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.976 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:55.976 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.609539+0000", 2024-04-19T01:04:55.977 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:55.977 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.977 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.977 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:55.977 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.609539+0000", 2024-04-19T01:04:55.977 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:55.977 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.977 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.977 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:55.977 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.609541+0000", 2024-04-19T01:04:55.977 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.855e-06 2024-04-19T01:04:55.977 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.977 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.977 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:55.978 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610203+0000", 2024-04-19T01:04:55.978 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00066190200000000002 2024-04-19T01:04:55.978 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.978 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.978 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:55.978 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610215+0000", 2024-04-19T01:04:55.978 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2143e-05 2024-04-19T01:04:55.978 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.978 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.978 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:55.978 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610221+0000", 2024-04-19T01:04:55.978 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4590000000000003e-06 2024-04-19T01:04:55.978 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.979 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.979 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:55.979 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610283+0000", 2024-04-19T01:04:55.979 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1322999999999995e-05 2024-04-19T01:04:55.979 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.979 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.979 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:55.979 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610394+0000", 2024-04-19T01:04:55.979 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000111044 2024-04-19T01:04:55.979 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.979 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.979 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:55.979 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611327+0000", 2024-04-19T01:04:55.979 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00093369799999999995 2024-04-19T01:04:55.979 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.980 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.980 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:55.980 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.623095+0000", 2024-04-19T01:04:55.980 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011767454 2024-04-19T01:04:55.980 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.980 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.980 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:55.980 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.631101+0000", 2024-04-19T01:04:55.981 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0080059889999999998 2024-04-19T01:04:55.981 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.981 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.981 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:55.981 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.631143+0000", 2024-04-19T01:04:55.981 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1811999999999999e-05 2024-04-19T01:04:55.981 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:55.981 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:55.981 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:55.981 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.982 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.982 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:319 3.fs0 3:f23900a8:::smithi15528762-6:head [append 0~147456] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:04:55.982 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.611402+0000", 2024-04-19T01:04:55.982 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 38.273358029999997, 2024-04-19T01:04:55.982 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.020301099, 2024-04-19T01:04:55.982 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:55.987 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:55.987 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:55.987 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:55.987 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:55.987 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:55.987 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 319 2024-04-19T01:04:55.987 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.987 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:55.987 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.988 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:55.988 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611402+0000", 2024-04-19T01:04:55.988 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:55.988 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.988 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.988 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:55.988 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611402+0000", 2024-04-19T01:04:55.988 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:55.988 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.988 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.988 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:55.988 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611404+0000", 2024-04-19T01:04:55.988 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.52e-06 2024-04-19T01:04:55.988 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.988 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.989 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:55.989 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611632+0000", 2024-04-19T01:04:55.989 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000227526 2024-04-19T01:04:55.989 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.989 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.989 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:55.989 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611660+0000", 2024-04-19T01:04:55.989 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7841999999999999e-05 2024-04-19T01:04:55.989 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.989 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.989 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:55.989 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611678+0000", 2024-04-19T01:04:55.989 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8479e-05 2024-04-19T01:04:55.990 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.990 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.990 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:55.990 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.620224+0000", 2024-04-19T01:04:55.990 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0085460999999999992 2024-04-19T01:04:55.990 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:55.990 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:55.997 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:55.998 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.620334+0000", 2024-04-19T01:04:56.004 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010964 2024-04-19T01:04:56.004 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.004 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.004 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:56.004 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.620982+0000", 2024-04-19T01:04:56.004 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00064788699999999999 2024-04-19T01:04:56.004 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.004 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.004 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:56.004 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.623230+0000", 2024-04-19T01:04:56.004 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0022481950000000001 2024-04-19T01:04:56.004 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.004 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.005 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:56.005 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.631664+0000", 2024-04-19T01:04:56.005 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0084339810000000001 2024-04-19T01:04:56.005 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.005 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.005 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:56.005 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.631703+0000", 2024-04-19T01:04:56.005 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8928999999999999e-05 2024-04-19T01:04:56.005 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.005 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:56.005 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.005 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.005 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.005 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:320 3.fs0 3:f23900a8:::smithi15528762-6:head [setxattr _header (55) in=7b] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:04:56.005 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.611709+0000", 2024-04-19T01:04:56.006 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 38.273050536, 2024-04-19T01:04:56.006 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.020427773, 2024-04-19T01:04:56.006 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:56.006 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:56.006 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:56.006 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:56.006 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:56.006 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:56.006 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 320 2024-04-19T01:04:56.006 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.006 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:56.006 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.007 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:56.007 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611709+0000", 2024-04-19T01:04:56.007 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.007 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.007 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.007 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:56.007 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611709+0000", 2024-04-19T01:04:56.007 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.007 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.007 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.007 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:56.007 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611711+0000", 2024-04-19T01:04:56.007 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.451e-06 2024-04-19T01:04:56.007 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.008 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.008 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:56.008 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611718+0000", 2024-04-19T01:04:56.008 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5800000000000003e-06 2024-04-19T01:04:56.008 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.008 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.008 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:56.008 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611736+0000", 2024-04-19T01:04:56.008 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7339000000000001e-05 2024-04-19T01:04:56.008 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.008 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.008 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:56.008 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611749+0000", 2024-04-19T01:04:56.008 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3557000000000001e-05 2024-04-19T01:04:56.009 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.009 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.009 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:56.009 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.622598+0000", 2024-04-19T01:04:56.009 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010848315000000001 2024-04-19T01:04:56.016 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.016 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.016 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:56.016 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.622654+0000", 2024-04-19T01:04:56.016 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6311999999999999e-05 2024-04-19T01:04:56.016 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.017 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.017 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:56.017 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.622895+0000", 2024-04-19T01:04:56.017 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00024134300000000001 2024-04-19T01:04:56.017 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.017 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.017 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:56.017 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.623782+0000", 2024-04-19T01:04:56.017 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00088635099999999996 2024-04-19T01:04:56.017 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.017 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.017 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:56.018 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.632093+0000", 2024-04-19T01:04:56.018 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.008311466 2024-04-19T01:04:56.018 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.018 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.018 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:56.018 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.632137+0000", 2024-04-19T01:04:56.018 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4058999999999999e-05 2024-04-19T01:04:56.018 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.018 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:56.018 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.018 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.018 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.018 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:321 3.fs0 3:f23900a8:::smithi15528762-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:04:56.018 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.611769+0000", 2024-04-19T01:04:56.019 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 38.272990999000001, 2024-04-19T01:04:56.019 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021684730999999999, 2024-04-19T01:04:56.019 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:56.019 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:56.019 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:56.019 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:56.019 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:56.019 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:56.019 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 321 2024-04-19T01:04:56.019 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.019 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:56.019 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.019 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:56.019 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611769+0000", 2024-04-19T01:04:56.019 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.020 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.020 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.020 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:56.020 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611769+0000", 2024-04-19T01:04:56.020 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.020 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.020 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.020 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:56.020 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611770+0000", 2024-04-19T01:04:56.020 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.04e-06 2024-04-19T01:04:56.020 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.020 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.020 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:56.020 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611775+0000", 2024-04-19T01:04:56.021 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3310000000000003e-06 2024-04-19T01:04:56.021 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.021 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.021 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:56.021 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611793+0000", 2024-04-19T01:04:56.021 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7408e-05 2024-04-19T01:04:56.021 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.021 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.021 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:56.021 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611808+0000", 2024-04-19T01:04:56.021 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5259000000000001e-05 2024-04-19T01:04:56.021 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.021 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.021 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:56.021 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.623039+0000", 2024-04-19T01:04:56.022 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011230756 2024-04-19T01:04:56.022 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.022 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.022 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:04:56.022 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.623089+0000", 2024-04-19T01:04:56.032 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0078999999999999e-05 2024-04-19T01:04:56.032 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.032 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.032 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:56.032 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.632385+0000", 2024-04-19T01:04:56.032 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0092956649999999998 2024-04-19T01:04:56.033 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.033 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.033 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:56.033 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.632471+0000", 2024-04-19T01:04:56.033 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.6693999999999994e-05 2024-04-19T01:04:56.033 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.033 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.034 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:56.034 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.633454+0000", 2024-04-19T01:04:56.034 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00098249899999999996 2024-04-19T01:04:56.034 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.034 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:56.034 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.034 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.034 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.034 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:345 3.fs0 3:f23900a8:::smithi15528762-6:head [append 0~557056] snapc 9=[8,7] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:04:56.034 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.534632+0000", 2024-04-19T01:04:56.034 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 37.350128175000002, 2024-04-19T01:04:56.035 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.044819185999999997, 2024-04-19T01:04:56.035 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:56.035 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:56.035 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:56.035 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:56.035 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:56.035 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:56.035 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 345 2024-04-19T01:04:56.035 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.035 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:56.035 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.035 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:56.035 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.534632+0000", 2024-04-19T01:04:56.035 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.035 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.036 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.036 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:56.036 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.534632+0000", 2024-04-19T01:04:56.036 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.036 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.036 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.036 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:56.036 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.534634+0000", 2024-04-19T01:04:56.036 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4099999999999998e-06 2024-04-19T01:04:56.037 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.037 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.037 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:56.037 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.535690+0000", 2024-04-19T01:04:56.037 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010557260000000001 2024-04-19T01:04:56.037 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.037 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.037 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:56.037 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.535710+0000", 2024-04-19T01:04:56.037 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0339e-05 2024-04-19T01:04:56.037 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.038 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.038 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:56.038 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.535720+0000", 2024-04-19T01:04:56.038 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7599999999999997e-06 2024-04-19T01:04:56.038 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.038 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.038 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:56.038 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.535817+0000", 2024-04-19T01:04:56.038 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7218000000000003e-05 2024-04-19T01:04:56.038 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.038 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.038 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:56.038 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.536037+0000", 2024-04-19T01:04:56.039 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00021956699999999999 2024-04-19T01:04:56.039 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.039 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.039 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:56.039 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.537439+0000", 2024-04-19T01:04:56.039 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014022240000000001 2024-04-19T01:04:56.039 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.039 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.039 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:56.039 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.557535+0000", 2024-04-19T01:04:56.039 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.020095874999999999 2024-04-19T01:04:56.039 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.039 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.039 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:56.040 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.579388+0000", 2024-04-19T01:04:56.040 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021852772999999999 2024-04-19T01:04:56.040 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.040 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.040 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:56.040 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.579451+0000", 2024-04-19T01:04:56.040 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3293999999999994e-05 2024-04-19T01:04:56.040 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.040 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:56.040 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.040 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.040 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.040 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:346 3.fs0 3:f23900a8:::smithi15528762-6:head [append 0~671744] snapc 9=[8,7] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:04:56.041 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.540969+0000", 2024-04-19T01:04:56.041 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 37.343790646999999, 2024-04-19T01:04:56.041 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.039266644000000003, 2024-04-19T01:04:56.041 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:56.041 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:56.041 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:56.041 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:56.041 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:56.041 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:56.041 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 346 2024-04-19T01:04:56.041 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.041 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:56.041 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.041 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:56.041 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.540969+0000", 2024-04-19T01:04:56.042 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.042 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.042 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.042 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:56.042 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.540969+0000", 2024-04-19T01:04:56.042 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.042 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.042 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.042 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:56.042 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.540971+0000", 2024-04-19T01:04:56.042 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9039999999999999e-06 2024-04-19T01:04:56.042 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.042 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.043 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:56.043 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542173+0000", 2024-04-19T01:04:56.044 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00120192 2024-04-19T01:04:56.044 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.045 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.045 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:56.045 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542196+0000", 2024-04-19T01:04:56.045 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3159e-05 2024-04-19T01:04:56.045 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.045 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.045 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:56.045 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542210+0000", 2024-04-19T01:04:56.046 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3899e-05 2024-04-19T01:04:56.046 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.046 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.046 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:56.046 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.546462+0000", 2024-04-19T01:04:56.046 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0042514629999999996 2024-04-19T01:04:56.046 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.046 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.046 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:56.046 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.546596+0000", 2024-04-19T01:04:56.046 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013463900000000001 2024-04-19T01:04:56.046 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.046 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.047 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:56.047 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.547898+0000", 2024-04-19T01:04:56.047 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0013016239999999999 2024-04-19T01:04:56.047 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.047 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.047 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:56.047 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.558500+0000", 2024-04-19T01:04:56.047 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010601928 2024-04-19T01:04:56.047 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.047 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.047 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:56.047 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.580189+0000", 2024-04-19T01:04:56.047 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021688796 2024-04-19T01:04:56.048 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.048 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.048 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:56.048 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.580236+0000", 2024-04-19T01:04:56.048 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7311999999999997e-05 2024-04-19T01:04:56.048 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.048 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:56.052 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.052 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.052 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.052 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:347 3.fs0 3:f23900a8:::smithi15528762-6:head [append 0~98304] snapc 9=[8,7] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:04:56.052 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.542237+0000", 2024-04-19T01:04:56.052 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 37.34252309, 2024-04-19T01:04:56.052 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.038983274999999998, 2024-04-19T01:04:56.052 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:56.052 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:56.052 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:56.052 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:56.053 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:56.053 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:56.053 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 347 2024-04-19T01:04:56.053 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.053 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:56.053 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.053 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:56.053 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542237+0000", 2024-04-19T01:04:56.053 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.053 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.053 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.053 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:56.053 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542237+0000", 2024-04-19T01:04:56.054 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.054 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.054 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.054 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:56.054 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542238+0000", 2024-04-19T01:04:56.054 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3659999999999999e-06 2024-04-19T01:04:56.054 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.054 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.054 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:56.054 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542318+0000", 2024-04-19T01:04:56.055 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.0158000000000001e-05 2024-04-19T01:04:56.055 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.055 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.055 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:56.055 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542336+0000", 2024-04-19T01:04:56.055 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7836999999999999e-05 2024-04-19T01:04:56.055 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.055 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.056 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:56.056 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542348+0000", 2024-04-19T01:04:56.056 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1646e-05 2024-04-19T01:04:56.056 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.056 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.056 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:56.056 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.553869+0000", 2024-04-19T01:04:56.056 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011521164 2024-04-19T01:04:56.056 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.056 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.056 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:56.056 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.554020+0000", 2024-04-19T01:04:56.057 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015064200000000001 2024-04-19T01:04:56.057 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.057 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.057 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:56.057 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.554913+0000", 2024-04-19T01:04:56.057 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00089314099999999997 2024-04-19T01:04:56.057 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.057 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.057 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:56.057 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.558556+0000", 2024-04-19T01:04:56.058 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003642803 2024-04-19T01:04:56.058 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.058 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.058 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:56.058 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.581179+0000", 2024-04-19T01:04:56.058 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.022623306999999999 2024-04-19T01:04:56.058 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.058 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.058 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:56.058 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.581220+0000", 2024-04-19T01:04:56.059 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1211000000000002e-05 2024-04-19T01:04:56.059 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.059 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:56.059 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.059 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.059 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.059 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:348 3.fs0 3:f23900a8:::smithi15528762-6:head [setxattr _header (55) in=7b] snapc 9=[8,7] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:04:56.059 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.542369+0000", 2024-04-19T01:04:56.059 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 37.342391229999997, 2024-04-19T01:04:56.059 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.039646576000000003, 2024-04-19T01:04:56.059 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:56.059 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:56.060 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:56.060 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:56.060 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:56.060 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:56.060 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 348 2024-04-19T01:04:56.060 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.060 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:56.060 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.060 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:56.060 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542369+0000", 2024-04-19T01:04:56.060 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.060 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.060 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.060 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:56.061 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542369+0000", 2024-04-19T01:04:56.061 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.061 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.061 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.061 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:56.061 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542370+0000", 2024-04-19T01:04:56.061 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.384e-06 2024-04-19T01:04:56.061 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.061 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.061 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:56.061 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542376+0000", 2024-04-19T01:04:56.061 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.806e-06 2024-04-19T01:04:56.061 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.061 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.061 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:56.062 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542400+0000", 2024-04-19T01:04:56.062 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4218000000000001e-05 2024-04-19T01:04:56.062 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.062 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.062 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:56.062 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542411+0000", 2024-04-19T01:04:56.062 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0801999999999999e-05 2024-04-19T01:04:56.062 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.062 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.062 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:56.062 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.557611+0000", 2024-04-19T01:04:56.062 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015199724 2024-04-19T01:04:56.062 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.062 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.063 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:56.063 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.557731+0000", 2024-04-19T01:04:56.063 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012045900000000001 2024-04-19T01:04:56.063 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.063 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.063 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:56.063 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.558229+0000", 2024-04-19T01:04:56.063 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00049771099999999997 2024-04-19T01:04:56.063 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.063 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.063 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:56.063 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.559080+0000", 2024-04-19T01:04:56.063 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00085117400000000001 2024-04-19T01:04:56.063 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.064 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.064 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:56.064 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.581912+0000", 2024-04-19T01:04:56.064 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.022832086000000001 2024-04-19T01:04:56.064 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.064 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.064 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:56.064 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.582015+0000", 2024-04-19T01:04:56.064 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000103212 2024-04-19T01:04:56.064 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.064 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:56.064 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.064 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.064 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.064 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:349 3.fs0 3:f23900a8:::smithi15528762-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:04:56.065 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.542427+0000", 2024-04-19T01:04:56.065 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 37.342333058999998, 2024-04-19T01:04:56.065 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.041128310000000001, 2024-04-19T01:04:56.065 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:56.065 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:56.065 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:56.066 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:56.066 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:56.066 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:56.066 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 349 2024-04-19T01:04:56.066 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.066 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:56.066 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.066 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:56.066 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542427+0000", 2024-04-19T01:04:56.067 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.067 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.067 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.067 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:56.067 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542427+0000", 2024-04-19T01:04:56.067 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.067 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.067 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.067 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:56.067 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542428+0000", 2024-04-19T01:04:56.067 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.4e-07 2024-04-19T01:04:56.067 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.067 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.067 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:56.068 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542432+0000", 2024-04-19T01:04:56.068 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5499999999999996e-06 2024-04-19T01:04:56.068 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.068 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.068 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:56.068 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542446+0000", 2024-04-19T01:04:56.068 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3365e-05 2024-04-19T01:04:56.068 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.068 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.068 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:56.068 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542456+0000", 2024-04-19T01:04:56.068 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.029e-05 2024-04-19T01:04:56.069 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.069 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.069 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:56.069 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.558587+0000", 2024-04-19T01:04:56.069 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.016131106999999999 2024-04-19T01:04:56.069 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.069 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.069 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:04:56.069 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.558676+0000", 2024-04-19T01:04:56.069 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.8823e-05 2024-04-19T01:04:56.070 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.070 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.070 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:56.070 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.582441+0000", 2024-04-19T01:04:56.070 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.023764864 2024-04-19T01:04:56.070 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.070 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.070 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:56.070 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.582493+0000", 2024-04-19T01:04:56.070 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2130999999999999e-05 2024-04-19T01:04:56.071 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.071 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.071 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:56.071 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.583555+0000", 2024-04-19T01:04:56.071 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010623399999999999 2024-04-19T01:04:56.071 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.071 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:56.071 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.071 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.071 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.071 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:590 3.cs0 3:3233f066:::smithi15528762-50:head [stat,delete] snapc 1a=[19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-19T01:04:56.071 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:49.626683+0000", 2024-04-19T01:04:56.071 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 6.258076698, 2024-04-19T01:04:56.072 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013899593, 2024-04-19T01:04:56.072 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:56.072 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:56.072 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:56.072 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:56.072 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:56.072 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:56.072 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 590 2024-04-19T01:04:56.072 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.072 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:56.072 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.073 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:56.073 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.626683+0000", 2024-04-19T01:04:56.073 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.073 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.073 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.073 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:56.073 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.626683+0000", 2024-04-19T01:04:56.073 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.073 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.073 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.073 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:56.073 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.626686+0000", 2024-04-19T01:04:56.074 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9579999999999999e-06 2024-04-19T01:04:56.074 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.074 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.074 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:56.074 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.626700+0000", 2024-04-19T01:04:56.074 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3365e-05 2024-04-19T01:04:56.074 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.074 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.074 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:56.074 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.626731+0000", 2024-04-19T01:04:56.074 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.1809e-05 2024-04-19T01:04:56.074 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.074 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.075 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:56.076 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.626745+0000", 2024-04-19T01:04:56.076 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3703000000000001e-05 2024-04-19T01:04:56.076 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.076 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.076 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:56.076 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.626843+0000", 2024-04-19T01:04:56.077 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7767000000000001e-05 2024-04-19T01:04:56.077 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.077 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.077 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op must wait for map", 2024-04-19T01:04:56.077 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.626851+0000", 2024-04-19T01:04:56.077 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7889999999999995e-06 2024-04-19T01:04:56.077 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.077 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.077 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:56.077 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.629688+0000", 2024-04-19T01:04:56.077 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0028371899999999999 2024-04-19T01:04:56.077 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.077 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.078 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:56.078 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.629782+0000", 2024-04-19T01:04:56.078 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4178999999999994e-05 2024-04-19T01:04:56.078 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.078 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.078 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:56.078 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.631268+0000", 2024-04-19T01:04:56.078 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014857939999999999 2024-04-19T01:04:56.078 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.078 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.078 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:56.078 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.634108+0000", 2024-04-19T01:04:56.078 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0028405029999999999 2024-04-19T01:04:56.079 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.079 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.079 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:56.079 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.640465+0000", 2024-04-19T01:04:56.079 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0063570019999999996 2024-04-19T01:04:56.079 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.079 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.079 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:56.079 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.640583+0000", 2024-04-19T01:04:56.079 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000117534 2024-04-19T01:04:56.079 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.079 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:56.079 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.079 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.080 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.080 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:599 3.fs0 3:f23900a8:::smithi15528762-6:head [append 0~0] snapc 1a=[19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-19T01:04:56.080 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:49.663368+0000", 2024-04-19T01:04:56.080 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 6.2213919019999997, 2024-04-19T01:04:56.080 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010336621000000001, 2024-04-19T01:04:56.080 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:56.080 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:56.080 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:56.080 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:56.080 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:56.080 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:56.080 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 599 2024-04-19T01:04:56.080 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.080 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:56.081 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.081 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:56.081 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663368+0000", 2024-04-19T01:04:56.081 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.081 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.081 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.081 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:56.081 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663368+0000", 2024-04-19T01:04:56.081 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.081 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.081 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.081 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:56.081 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663369+0000", 2024-04-19T01:04:56.081 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.3099999999999996e-07 2024-04-19T01:04:56.082 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.082 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.082 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:56.082 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663373+0000", 2024-04-19T01:04:56.082 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9570000000000004e-06 2024-04-19T01:04:56.082 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.082 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.082 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:56.082 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663383+0000", 2024-04-19T01:04:56.082 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0288999999999999e-05 2024-04-19T01:04:56.082 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.082 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.082 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:56.082 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663391+0000", 2024-04-19T01:04:56.083 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.0339999999999997e-06 2024-04-19T01:04:56.083 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.083 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.083 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:56.083 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.666947+0000", 2024-04-19T01:04:56.083 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003555683 2024-04-19T01:04:56.083 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.083 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.083 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:56.083 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.667010+0000", 2024-04-19T01:04:56.083 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2732999999999997e-05 2024-04-19T01:04:56.083 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.083 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.083 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:56.083 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.667364+0000", 2024-04-19T01:04:56.084 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00035485999999999997 2024-04-19T01:04:56.084 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.084 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.084 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:56.084 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.669976+0000", 2024-04-19T01:04:56.084 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0026119730000000001 2024-04-19T01:04:56.084 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.084 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.084 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:56.084 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.673641+0000", 2024-04-19T01:04:56.084 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0036649859999999999 2024-04-19T01:04:56.084 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.084 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.085 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:56.085 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.673705+0000", 2024-04-19T01:04:56.085 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3275000000000007e-05 2024-04-19T01:04:56.085 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.085 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:56.085 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.085 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.085 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:56.085 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:56.087 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-19T01:04:56.088 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:04:56.087+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:04:56.187 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:04:56.188+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:04:56.221 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:56.221 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:04:56.221 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2024-04-19T01:04:56.221 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:56.236 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-19T01:04:56.290 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:56.287+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:56.365 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:56.365 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:04:56.365 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2024-04-19T01:04:56.365 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2024-04-19T01:04:56.365 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:56.377 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-19T01:04:56.389 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:04:56.388+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:04:56.491 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:04:56.492+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:04:56.508 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:56.509 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2024-04-19T01:04:56.509 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2024-04-19T01:04:56.509 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-04-19T01:04:56.509 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.509 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds1 43/31 ECSubWrite(tid=38, reqid=client.4604.0:253, at_version=43'14, trim_to=0'0, roll_forward_to=43'12))", 2024-04-19T01:04:56.509 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.986285+0000", 2024-04-19T01:04:56.509 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 42.520049583000002, 2024-04-19T01:04:56.509 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0090548699999999996, 2024-04-19T01:04:56.509 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:56.509 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:56.509 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:56.509 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:56.509 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.509 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:56.510 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.986285+0000", 2024-04-19T01:04:56.510 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.510 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.510 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.510 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:56.510 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.986285+0000", 2024-04-19T01:04:56.510 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.510 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.510 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.510 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:56.510 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.986287+0000", 2024-04-19T01:04:56.510 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.421e-06 2024-04-19T01:04:56.510 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.510 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.511 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:56.511 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.986389+0000", 2024-04-19T01:04:56.511 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010215100000000001 2024-04-19T01:04:56.511 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.511 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.511 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:56.511 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.986414+0000", 2024-04-19T01:04:56.511 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4447000000000002e-05 2024-04-19T01:04:56.511 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.511 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.511 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:56.511 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.986435+0000", 2024-04-19T01:04:56.511 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1069000000000001e-05 2024-04-19T01:04:56.511 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.512 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.512 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:56.512 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.986566+0000", 2024-04-19T01:04:56.512 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013169899999999999 2024-04-19T01:04:56.512 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.512 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.512 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:56.512 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.986604+0000", 2024-04-19T01:04:56.512 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7857e-05 2024-04-19T01:04:56.512 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.512 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.512 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:56.512 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.995257+0000", 2024-04-19T01:04:56.512 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0086525209999999998 2024-04-19T01:04:56.512 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.513 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.513 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:56.513 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.995340+0000", 2024-04-19T01:04:56.513 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.3704999999999999e-05 2024-04-19T01:04:56.513 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.513 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:56.513 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.513 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.513 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.513 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.9s4 44/31 ECSubWrite(tid=82, reqid=client.4604.0:280, at_version=44'50, trim_to=0'0, roll_forward_to=43'48))", 2024-04-19T01:04:56.513 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:14.880621+0000", 2024-04-19T01:04:56.513 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 41.625714375999998, 2024-04-19T01:04:56.513 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.009454835, 2024-04-19T01:04:56.513 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:56.513 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:56.514 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:56.514 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:56.514 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.514 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:56.514 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.880621+0000", 2024-04-19T01:04:56.514 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.514 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.514 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.514 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:56.514 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.880621+0000", 2024-04-19T01:04:56.514 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.514 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.514 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.515 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:56.515 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.880622+0000", 2024-04-19T01:04:56.515 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.6200000000000006e-07 2024-04-19T01:04:56.515 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.515 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.515 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:56.515 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.881098+0000", 2024-04-19T01:04:56.515 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00047665400000000002 2024-04-19T01:04:56.515 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.515 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.516 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:56.516 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.881115+0000", 2024-04-19T01:04:56.516 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6612999999999998e-05 2024-04-19T01:04:56.516 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.517 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.517 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:56.517 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.881124+0000", 2024-04-19T01:04:56.517 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.3959999999999995e-06 2024-04-19T01:04:56.517 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.517 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.517 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:56.517 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.881176+0000", 2024-04-19T01:04:56.517 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.1965000000000001e-05 2024-04-19T01:04:56.517 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.517 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.517 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:56.517 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.881182+0000", 2024-04-19T01:04:56.517 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.806e-06 2024-04-19T01:04:56.518 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.518 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.518 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:56.518 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.890038+0000", 2024-04-19T01:04:56.518 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0088557170000000008 2024-04-19T01:04:56.518 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.518 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.518 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:56.518 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.890076+0000", 2024-04-19T01:04:56.518 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7722000000000002e-05 2024-04-19T01:04:56.518 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.518 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:56.518 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.519 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.519 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.519 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.0s2 44/31 ECSubWrite(tid=25, reqid=client.4604.0:284, at_version=44'22, trim_to=0'0, roll_forward_to=44'20))", 2024-04-19T01:04:56.519 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:14.920303+0000", 2024-04-19T01:04:56.519 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 41.586032103000001, 2024-04-19T01:04:56.519 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011517777999999999, 2024-04-19T01:04:56.519 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:56.519 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:56.519 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:56.519 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:56.519 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.519 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:56.519 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.920303+0000", 2024-04-19T01:04:56.519 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.519 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.520 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.520 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:56.520 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.920303+0000", 2024-04-19T01:04:56.520 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.520 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.520 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.520 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:56.520 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.920304+0000", 2024-04-19T01:04:56.520 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2900000000000003e-07 2024-04-19T01:04:56.520 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.520 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.520 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:56.520 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.921382+0000", 2024-04-19T01:04:56.520 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010785510000000001 2024-04-19T01:04:56.521 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.521 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.521 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:56.521 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.921394+0000", 2024-04-19T01:04:56.521 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2183e-05 2024-04-19T01:04:56.521 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.521 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.521 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:56.521 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.921403+0000", 2024-04-19T01:04:56.521 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9330000000000007e-06 2024-04-19T01:04:56.521 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.521 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.521 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:56.521 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.921468+0000", 2024-04-19T01:04:56.521 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4191999999999998e-05 2024-04-19T01:04:56.522 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.522 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.522 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:56.522 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.921479+0000", 2024-04-19T01:04:56.522 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1763000000000001e-05 2024-04-19T01:04:56.522 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.522 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.522 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:56.522 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.931782+0000", 2024-04-19T01:04:56.522 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010302891999999999 2024-04-19T01:04:56.522 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.522 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.522 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:56.522 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.931821+0000", 2024-04-19T01:04:56.523 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8535000000000003e-05 2024-04-19T01:04:56.523 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.523 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:56.523 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.523 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.523 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.523 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds1 47/31 ECSubWrite(tid=61, reqid=client.4604.0:330, at_version=47'32, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:04:56.523 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.676507+0000", 2024-04-19T01:04:56.523 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 38.829827993000002, 2024-04-19T01:04:56.523 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010458914, 2024-04-19T01:04:56.523 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:56.523 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:56.523 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:56.523 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:56.524 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.524 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:56.524 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.676507+0000", 2024-04-19T01:04:56.524 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.524 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.524 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.524 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:56.524 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.676507+0000", 2024-04-19T01:04:56.524 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.525 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.525 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.525 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:56.525 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.676508+0000", 2024-04-19T01:04:56.525 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2399999999999996e-07 2024-04-19T01:04:56.525 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.525 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.525 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:56.525 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.676672+0000", 2024-04-19T01:04:56.525 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000164258 2024-04-19T01:04:56.525 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.526 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.526 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:56.526 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.676688+0000", 2024-04-19T01:04:56.527 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5443000000000001e-05 2024-04-19T01:04:56.527 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.527 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.527 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:56.527 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.676696+0000", 2024-04-19T01:04:56.527 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1550000000000007e-06 2024-04-19T01:04:56.527 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.527 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.527 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:56.527 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.676799+0000", 2024-04-19T01:04:56.527 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000102786 2024-04-19T01:04:56.527 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.527 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.527 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:56.528 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.676814+0000", 2024-04-19T01:04:56.528 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4951e-05 2024-04-19T01:04:56.528 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.528 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.528 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:56.528 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.686911+0000", 2024-04-19T01:04:56.528 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010097415 2024-04-19T01:04:56.528 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.528 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.528 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:56.528 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.686966+0000", 2024-04-19T01:04:56.528 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4982000000000003e-05 2024-04-19T01:04:56.528 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.528 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:56.528 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.529 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.529 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.529 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.3s5 48/31 ECSubWrite(tid=46, reqid=client.4604.0:343, at_version=48'36, trim_to=0'0, roll_forward_to=48'34))", 2024-04-19T01:04:56.529 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.541349+0000", 2024-04-19T01:04:56.529 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 37.964985581999997, 2024-04-19T01:04:56.529 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010550992, 2024-04-19T01:04:56.529 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:56.529 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:56.529 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:56.529 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:56.529 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.529 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:56.529 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.541349+0000", 2024-04-19T01:04:56.529 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.530 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.530 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.530 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:56.530 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.541349+0000", 2024-04-19T01:04:56.530 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.530 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.530 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.530 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:56.530 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.541351+0000", 2024-04-19T01:04:56.530 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.248e-06 2024-04-19T01:04:56.530 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.530 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.530 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:56.530 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.541675+0000", 2024-04-19T01:04:56.531 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00032395500000000001 2024-04-19T01:04:56.531 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.531 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.531 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:56.531 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.541707+0000", 2024-04-19T01:04:56.531 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2042000000000003e-05 2024-04-19T01:04:56.531 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.531 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.531 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:56.531 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.541720+0000", 2024-04-19T01:04:56.531 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3101999999999999e-05 2024-04-19T01:04:56.531 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.531 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.531 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:56.531 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.541873+0000", 2024-04-19T01:04:56.532 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015282199999999999 2024-04-19T01:04:56.532 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.532 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.532 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:56.532 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.541899+0000", 2024-04-19T01:04:56.532 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6381999999999999e-05 2024-04-19T01:04:56.532 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.532 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.532 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:56.532 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.551828+0000", 2024-04-19T01:04:56.532 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0099288589999999999 2024-04-19T01:04:56.532 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.532 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.532 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:56.533 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.551900+0000", 2024-04-19T01:04:56.533 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2582000000000004e-05 2024-04-19T01:04:56.533 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.533 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:56.533 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.533 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.533 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.533 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.as1 48/31 ECSubWrite(tid=71, reqid=client.4604.0:354, at_version=48'10, trim_to=0'0, roll_forward_to=34'8))", 2024-04-19T01:04:56.533 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.556705+0000", 2024-04-19T01:04:56.533 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 37.949629807999997, 2024-04-19T01:04:56.533 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013008369000000001, 2024-04-19T01:04:56.533 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:56.533 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:56.533 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:56.533 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:56.534 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.534 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:56.534 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.556705+0000", 2024-04-19T01:04:56.534 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.534 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.534 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.534 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:56.534 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.556705+0000", 2024-04-19T01:04:56.534 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.534 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.534 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.534 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:56.534 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.556706+0000", 2024-04-19T01:04:56.534 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2800000000000005e-07 2024-04-19T01:04:56.535 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.535 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.535 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:56.535 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.557214+0000", 2024-04-19T01:04:56.535 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00050754200000000004 2024-04-19T01:04:56.535 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.535 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.535 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:56.535 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.557239+0000", 2024-04-19T01:04:56.535 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.567e-05 2024-04-19T01:04:56.535 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.535 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.535 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:56.535 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.557268+0000", 2024-04-19T01:04:56.536 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8496e-05 2024-04-19T01:04:56.536 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.536 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.536 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:56.536 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.557438+0000", 2024-04-19T01:04:56.536 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000169963 2024-04-19T01:04:56.536 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.536 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.536 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:56.536 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.557491+0000", 2024-04-19T01:04:56.536 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2689999999999999e-05 2024-04-19T01:04:56.536 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.536 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.536 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:56.537 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.569674+0000", 2024-04-19T01:04:56.537 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012183727 2024-04-19T01:04:56.537 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.537 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.537 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:56.537 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.569714+0000", 2024-04-19T01:04:56.537 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9353000000000001e-05 2024-04-19T01:04:56.537 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.537 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:56.537 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.537 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.537 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.537 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.3s5 49/31 ECSubWrite(tid=48, reqid=osd.15.0:48, at_version=49'38, trim_to=0'0, roll_forward_to=48'36))", 2024-04-19T01:04:56.537 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:19.533518+0000", 2024-04-19T01:04:56.537 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 36.972816823000002, 2024-04-19T01:04:56.538 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011056513, 2024-04-19T01:04:56.538 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:56.538 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:56.538 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:56.538 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:56.538 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.538 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:56.538 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.533518+0000", 2024-04-19T01:04:56.538 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.538 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.538 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.538 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:56.538 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.533518+0000", 2024-04-19T01:04:56.539 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.539 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.539 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.539 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:56.539 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.533519+0000", 2024-04-19T01:04:56.539 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4000000000000002e-07 2024-04-19T01:04:56.539 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.539 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.539 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:56.539 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.533543+0000", 2024-04-19T01:04:56.539 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4494999999999998e-05 2024-04-19T01:04:56.539 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.539 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.539 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:56.539 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.533556+0000", 2024-04-19T01:04:56.540 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3145e-05 2024-04-19T01:04:56.540 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.540 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.540 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:56.540 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.533565+0000", 2024-04-19T01:04:56.540 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.4330000000000004e-06 2024-04-19T01:04:56.540 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.540 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.540 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:56.540 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.533697+0000", 2024-04-19T01:04:56.540 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013253399999999999 2024-04-19T01:04:56.540 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.540 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.540 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:56.541 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.533714+0000", 2024-04-19T01:04:56.541 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7163000000000002e-05 2024-04-19T01:04:56.541 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.541 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.541 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:56.541 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.544518+0000", 2024-04-19T01:04:56.541 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010803063999999999 2024-04-19T01:04:56.541 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.541 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.541 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:56.541 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.544575+0000", 2024-04-19T01:04:56.541 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.7238999999999999e-05 2024-04-19T01:04:56.541 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.541 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:56.541 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.542 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.542 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.542 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.3s5 49/31 ECSubWrite(tid=49, reqid=osd.15.0:49, at_version=49'40, trim_to=0'0, roll_forward_to=48'36))", 2024-04-19T01:04:56.542 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:19.535950+0000", 2024-04-19T01:04:56.542 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 36.970384922000001, 2024-04-19T01:04:56.542 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0096398579999999994, 2024-04-19T01:04:56.542 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:56.542 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:56.542 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:56.542 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:56.542 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.543 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:56.543 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.535950+0000", 2024-04-19T01:04:56.543 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.543 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.543 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.543 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:56.543 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.535950+0000", 2024-04-19T01:04:56.543 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.543 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.543 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.543 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:56.543 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.535952+0000", 2024-04-19T01:04:56.543 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4249999999999999e-06 2024-04-19T01:04:56.543 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.544 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.544 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:56.544 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.535984+0000", 2024-04-19T01:04:56.544 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2076000000000003e-05 2024-04-19T01:04:56.544 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.544 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.544 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:56.544 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.536002+0000", 2024-04-19T01:04:56.544 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8252000000000001e-05 2024-04-19T01:04:56.544 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.544 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.544 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:56.544 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.536015+0000", 2024-04-19T01:04:56.544 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3358e-05 2024-04-19T01:04:56.544 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.545 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.545 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:56.545 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.536116+0000", 2024-04-19T01:04:56.545 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000101105 2024-04-19T01:04:56.545 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.545 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.545 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:56.545 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.536132+0000", 2024-04-19T01:04:56.545 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5495999999999999e-05 2024-04-19T01:04:56.545 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.545 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.545 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:56.545 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.545543+0000", 2024-04-19T01:04:56.545 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0094109139999999994 2024-04-19T01:04:56.545 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.546 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.546 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:56.546 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.545590+0000", 2024-04-19T01:04:56.546 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7231999999999998e-05 2024-04-19T01:04:56.546 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.546 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:56.546 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.546 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.546 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.546 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.es3 49/31 ECSubWrite(tid=125, reqid=osd.13.0:125, at_version=49'28, trim_to=0'0, roll_forward_to=48'26))", 2024-04-19T01:04:56.546 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:19.538199+0000", 2024-04-19T01:04:56.546 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 36.968135627000002, 2024-04-19T01:04:56.546 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0081672089999999999, 2024-04-19T01:04:56.546 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:56.547 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:56.547 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:56.547 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:56.547 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.547 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:56.547 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538199+0000", 2024-04-19T01:04:56.547 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.547 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.547 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.547 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:56.547 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538199+0000", 2024-04-19T01:04:56.547 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.547 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.547 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.547 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:56.548 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538200+0000", 2024-04-19T01:04:56.548 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8799999999999998e-07 2024-04-19T01:04:56.548 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.548 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.548 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:56.548 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538236+0000", 2024-04-19T01:04:56.548 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.6575999999999997e-05 2024-04-19T01:04:56.548 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.548 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.548 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:56.548 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538257+0000", 2024-04-19T01:04:56.548 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0874999999999999e-05 2024-04-19T01:04:56.548 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.548 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.548 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:56.549 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538275+0000", 2024-04-19T01:04:56.549 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7745999999999999e-05 2024-04-19T01:04:56.549 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.549 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.549 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:56.549 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538356+0000", 2024-04-19T01:04:56.549 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1361999999999998e-05 2024-04-19T01:04:56.549 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.549 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.549 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:56.549 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538366+0000", 2024-04-19T01:04:56.549 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.9809999999999998e-06 2024-04-19T01:04:56.549 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.549 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.549 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:56.550 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.546323+0000", 2024-04-19T01:04:56.550 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0079562420000000005 2024-04-19T01:04:56.550 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.550 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.550 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:56.550 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.546367+0000", 2024-04-19T01:04:56.550 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4039000000000003e-05 2024-04-19T01:04:56.550 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.550 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:56.550 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.550 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.550 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.550 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.9s4 49/31 ECSubWrite(tid=124, reqid=osd.13.0:124, at_version=49'75, trim_to=0'0, roll_forward_to=48'73))", 2024-04-19T01:04:56.550 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:19.538317+0000", 2024-04-19T01:04:56.551 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 36.968018360999999, 2024-04-19T01:04:56.551 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0082727589999999993, 2024-04-19T01:04:56.551 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:56.551 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:56.551 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:56.551 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:56.551 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.551 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:56.551 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538317+0000", 2024-04-19T01:04:56.551 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.551 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.551 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.551 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:56.551 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538317+0000", 2024-04-19T01:04:56.551 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.552 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.552 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.552 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:56.552 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538317+0000", 2024-04-19T01:04:56.552 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1200000000000003e-07 2024-04-19T01:04:56.552 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.552 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.552 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:56.552 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538344+0000", 2024-04-19T01:04:56.552 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6293000000000001e-05 2024-04-19T01:04:56.552 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.552 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.552 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:56.552 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538360+0000", 2024-04-19T01:04:56.553 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6761000000000001e-05 2024-04-19T01:04:56.553 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.553 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.553 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:56.553 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538374+0000", 2024-04-19T01:04:56.553 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4017e-05 2024-04-19T01:04:56.553 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.553 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.553 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:56.553 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538500+0000", 2024-04-19T01:04:56.553 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000125587 2024-04-19T01:04:56.553 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.553 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.553 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:56.554 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.538510+0000", 2024-04-19T01:04:56.554 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.046e-05 2024-04-19T01:04:56.554 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.554 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.554 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:56.554 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.546545+0000", 2024-04-19T01:04:56.554 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0080348729999999997 2024-04-19T01:04:56.554 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.554 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.554 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:56.554 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.546589+0000", 2024-04-19T01:04:56.554 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4156000000000002e-05 2024-04-19T01:04:56.554 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.554 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:56.554 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.555 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.555 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.555 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.9s4 63/31 ECSubWrite(tid=152, reqid=client.4604.0:465, at_version=63'100, trim_to=0'0, roll_forward_to=62'98))", 2024-04-19T01:04:56.555 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:33.565180+0000", 2024-04-19T01:04:56.555 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 22.941155325, 2024-04-19T01:04:56.555 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018638968999999998, 2024-04-19T01:04:56.555 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:56.555 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:56.555 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:56.555 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:56.555 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.555 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:56.555 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:33.565180+0000", 2024-04-19T01:04:56.555 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.556 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.556 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.556 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:56.556 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:33.565180+0000", 2024-04-19T01:04:56.556 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.556 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.556 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.556 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:56.556 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:33.565181+0000", 2024-04-19T01:04:56.556 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7700000000000004e-07 2024-04-19T01:04:56.556 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.556 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.556 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:56.556 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:33.565938+0000", 2024-04-19T01:04:56.556 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00075768199999999997 2024-04-19T01:04:56.557 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.557 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.557 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:56.557 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:33.565973+0000", 2024-04-19T01:04:56.557 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5021999999999998e-05 2024-04-19T01:04:56.557 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.557 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.557 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:56.557 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:33.566001+0000", 2024-04-19T01:04:56.557 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8133e-05 2024-04-19T01:04:56.557 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.557 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.557 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:56.557 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:33.566147+0000", 2024-04-19T01:04:56.557 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000145539 2024-04-19T01:04:56.558 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.558 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.558 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:56.558 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:33.566165+0000", 2024-04-19T01:04:56.558 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7788999999999999e-05 2024-04-19T01:04:56.558 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.558 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.558 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:56.558 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:33.583741+0000", 2024-04-19T01:04:56.558 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017576789999999998 2024-04-19T01:04:56.559 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.559 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.559 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:56.559 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:33.583819+0000", 2024-04-19T01:04:56.559 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7237000000000006e-05 2024-04-19T01:04:56.559 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.560 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:56.560 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.560 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.560 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.560 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.as1 65/31 ECSubWrite(tid=107, reqid=client.4604.0:478, at_version=65'26, trim_to=0'0, roll_forward_to=64'24))", 2024-04-19T01:04:56.560 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:35.639127+0000", 2024-04-19T01:04:56.560 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 20.867207896, 2024-04-19T01:04:56.560 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011274793, 2024-04-19T01:04:56.560 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:56.560 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:56.560 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:56.560 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:56.560 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.560 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:56.561 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639127+0000", 2024-04-19T01:04:56.561 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.561 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.561 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.561 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:56.561 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639127+0000", 2024-04-19T01:04:56.561 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.561 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.561 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.561 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:56.561 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639128+0000", 2024-04-19T01:04:56.561 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6000000000000004e-07 2024-04-19T01:04:56.561 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.561 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.562 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:56.562 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639372+0000", 2024-04-19T01:04:56.562 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00024466199999999998 2024-04-19T01:04:56.562 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.562 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.562 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:56.562 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639401+0000", 2024-04-19T01:04:56.562 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8282000000000001e-05 2024-04-19T01:04:56.562 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.562 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.562 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:56.562 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639446+0000", 2024-04-19T01:04:56.564 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5513000000000003e-05 2024-04-19T01:04:56.564 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.564 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.564 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:56.564 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639612+0000", 2024-04-19T01:04:56.564 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000166069 2024-04-19T01:04:56.564 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.564 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.564 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:56.564 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639626+0000", 2024-04-19T01:04:56.564 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3811999999999999e-05 2024-04-19T01:04:56.564 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.564 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.564 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:56.565 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.650359+0000", 2024-04-19T01:04:56.565 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010732858 2024-04-19T01:04:56.565 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.565 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.565 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:56.565 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.650402+0000", 2024-04-19T01:04:56.565 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3037000000000001e-05 2024-04-19T01:04:56.565 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.565 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:56.565 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.565 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.565 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.565 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.9s4 74/31 ECSubWrite(tid=174, reqid=client.4604.0:548, at_version=74'110, trim_to=0'0, roll_forward_to=70'108))", 2024-04-19T01:04:56.565 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:44.594011+0000", 2024-04-19T01:04:56.565 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 11.912323759, 2024-04-19T01:04:56.566 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0081016119999999994, 2024-04-19T01:04:56.566 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:56.566 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:56.566 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:56.566 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:56.566 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.566 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:56.566 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.594011+0000", 2024-04-19T01:04:56.566 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.566 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.566 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.566 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:56.566 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.594011+0000", 2024-04-19T01:04:56.566 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.567 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.567 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.567 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:56.567 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.594012+0000", 2024-04-19T01:04:56.567 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8799999999999999e-07 2024-04-19T01:04:56.567 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.567 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.567 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:56.567 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.594592+0000", 2024-04-19T01:04:56.567 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000580048 2024-04-19T01:04:56.567 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.567 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.567 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:56.567 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.594603+0000", 2024-04-19T01:04:56.567 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1117999999999999e-05 2024-04-19T01:04:56.568 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.568 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.568 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:56.568 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.594609+0000", 2024-04-19T01:04:56.568 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3389999999999998e-06 2024-04-19T01:04:56.568 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.568 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.568 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:56.568 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.594702+0000", 2024-04-19T01:04:56.568 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.3096000000000004e-05 2024-04-19T01:04:56.568 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.568 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.568 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:56.569 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.594721+0000", 2024-04-19T01:04:56.569 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8734999999999999e-05 2024-04-19T01:04:56.569 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.569 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.569 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:56.569 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.602076+0000", 2024-04-19T01:04:56.569 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0073546879999999999 2024-04-19T01:04:56.569 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.569 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.569 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:56.569 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.602113+0000", 2024-04-19T01:04:56.569 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7400000000000001e-05 2024-04-19T01:04:56.569 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.569 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:56.569 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.570 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.570 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.570 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.2s3 79/31 ECSubWrite(tid=249, reqid=osd.11.0:249, at_version=79'21, trim_to=0'0, roll_forward_to=74'18))", 2024-04-19T01:04:56.570 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:49.638454+0000", 2024-04-19T01:04:56.570 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 6.8678815239999995, 2024-04-19T01:04:56.570 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011186546, 2024-04-19T01:04:56.570 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:56.570 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:56.570 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:56.570 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:56.570 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.570 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:56.570 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.638454+0000", 2024-04-19T01:04:56.570 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.571 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.571 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.571 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:56.571 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.638454+0000", 2024-04-19T01:04:56.571 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.571 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.571 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.571 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:56.571 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.638454+0000", 2024-04-19T01:04:56.571 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.58e-07 2024-04-19T01:04:56.571 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.571 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.571 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:56.571 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.638536+0000", 2024-04-19T01:04:56.571 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.2070999999999994e-05 2024-04-19T01:04:56.572 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.572 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.572 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:56.572 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.638561+0000", 2024-04-19T01:04:56.572 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5026999999999999e-05 2024-04-19T01:04:56.572 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.572 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.572 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:56.572 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.638585+0000", 2024-04-19T01:04:56.572 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4324999999999998e-05 2024-04-19T01:04:56.572 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.572 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.572 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:56.572 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.638684+0000", 2024-04-19T01:04:56.573 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.8196999999999995e-05 2024-04-19T01:04:56.573 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.573 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.573 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:56.573 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.638702+0000", 2024-04-19T01:04:56.573 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7989999999999999e-05 2024-04-19T01:04:56.573 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.573 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.573 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:56.573 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.649593+0000", 2024-04-19T01:04:56.573 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010891813 2024-04-19T01:04:56.573 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.573 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.573 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:56.573 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.649640+0000", 2024-04-19T01:04:56.574 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6665000000000001e-05 2024-04-19T01:04:56.574 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.574 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:56.574 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.574 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.574 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.574 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds1 79/31 ECSubWrite(tid=157, reqid=osd.12.0:157, at_version=79'66, trim_to=0'0, roll_forward_to=78'64))", 2024-04-19T01:04:56.574 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:49.638978+0000", 2024-04-19T01:04:56.574 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 6.8673574649999996, 2024-04-19T01:04:56.574 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0089370209999999999, 2024-04-19T01:04:56.574 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:56.574 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:56.574 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:56.574 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:56.574 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.575 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:56.575 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.638978+0000", 2024-04-19T01:04:56.575 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.575 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.575 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.575 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:56.575 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.638978+0000", 2024-04-19T01:04:56.575 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.575 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.575 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.575 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:56.575 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.638978+0000", 2024-04-19T01:04:56.575 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2399999999999998e-07 2024-04-19T01:04:56.575 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.575 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.576 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:56.576 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.639038+0000", 2024-04-19T01:04:56.576 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.9831999999999999e-05 2024-04-19T01:04:56.576 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.576 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.576 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:56.576 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.639070+0000", 2024-04-19T01:04:56.576 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2246000000000003e-05 2024-04-19T01:04:56.577 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.577 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.577 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:56.577 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.639094+0000", 2024-04-19T01:04:56.577 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4113000000000001e-05 2024-04-19T01:04:56.577 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.577 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.577 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:56.577 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.639344+0000", 2024-04-19T01:04:56.577 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00024928100000000001 2024-04-19T01:04:56.577 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.577 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.577 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:56.577 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.639358+0000", 2024-04-19T01:04:56.578 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4243000000000001e-05 2024-04-19T01:04:56.578 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.578 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.578 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:56.578 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.647858+0000", 2024-04-19T01:04:56.578 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0084999619999999998 2024-04-19T01:04:56.578 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.578 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.578 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:56.578 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.647915+0000", 2024-04-19T01:04:56.578 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6820000000000001e-05 2024-04-19T01:04:56.578 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.578 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:56.578 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.578 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.579 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.579 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.as1 79/31 ECSubWrite(tid=159, reqid=osd.12.0:159, at_version=79'66, trim_to=0'0, roll_forward_to=74'62))", 2024-04-19T01:04:56.579 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:49.640748+0000", 2024-04-19T01:04:56.579 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 6.865586725, 2024-04-19T01:04:56.579 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0089421429999999996, 2024-04-19T01:04:56.579 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:56.579 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:56.579 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:56.579 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:56.579 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.579 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:56.579 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.640748+0000", 2024-04-19T01:04:56.579 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.579 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.579 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.580 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:56.580 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.640748+0000", 2024-04-19T01:04:56.580 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.580 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.580 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.580 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:56.580 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.640750+0000", 2024-04-19T01:04:56.580 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1790000000000001e-06 2024-04-19T01:04:56.580 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.580 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.580 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:56.580 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.640782+0000", 2024-04-19T01:04:56.580 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2257e-05 2024-04-19T01:04:56.580 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.580 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.581 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:56.581 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.640801+0000", 2024-04-19T01:04:56.581 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9227000000000001e-05 2024-04-19T01:04:56.581 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.581 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.581 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:56.581 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.640814+0000", 2024-04-19T01:04:56.581 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3244999999999999e-05 2024-04-19T01:04:56.581 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.581 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.581 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:56.581 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.641654+0000", 2024-04-19T01:04:56.581 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00083967199999999997 2024-04-19T01:04:56.581 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.582 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.582 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:56.582 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.641670+0000", 2024-04-19T01:04:56.582 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6501e-05 2024-04-19T01:04:56.582 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.582 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.582 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:56.582 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.649646+0000", 2024-04-19T01:04:56.582 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0079758620000000002 2024-04-19T01:04:56.582 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.582 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.582 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:56.582 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.649690+0000", 2024-04-19T01:04:56.582 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4199999999999997e-05 2024-04-19T01:04:56.583 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.583 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:56.583 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.583 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.583 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.583 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.3s5 80/31 ECSubWrite(tid=63, reqid=client.4604.0:612, at_version=80'51, trim_to=0'0, roll_forward_to=57'49))", 2024-04-19T01:04:56.583 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:50.660352+0000", 2024-04-19T01:04:56.583 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.8459833540000004, 2024-04-19T01:04:56.583 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010734285999999999, 2024-04-19T01:04:56.583 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:56.583 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:56.583 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:56.583 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:56.583 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.583 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:56.584 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:50.660352+0000", 2024-04-19T01:04:56.584 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.584 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.584 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.584 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:56.584 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:50.660352+0000", 2024-04-19T01:04:56.584 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.584 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.584 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.584 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:56.584 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:50.660352+0000", 2024-04-19T01:04:56.584 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2899999999999999e-07 2024-04-19T01:04:56.584 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.584 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.585 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:56.585 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:50.660870+0000", 2024-04-19T01:04:56.585 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00051747399999999997 2024-04-19T01:04:56.585 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.585 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.585 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:56.585 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:50.660884+0000", 2024-04-19T01:04:56.585 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4608e-05 2024-04-19T01:04:56.585 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.585 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.585 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:56.585 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:50.660892+0000", 2024-04-19T01:04:56.585 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.9740000000000002e-06 2024-04-19T01:04:56.585 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.585 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.586 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:56.586 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:50.660991+0000", 2024-04-19T01:04:56.586 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.8862000000000003e-05 2024-04-19T01:04:56.586 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.586 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.586 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:56.586 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:50.661003+0000", 2024-04-19T01:04:56.586 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1629e-05 2024-04-19T01:04:56.586 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.586 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.586 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:56.586 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:50.671021+0000", 2024-04-19T01:04:56.586 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010018575 2024-04-19T01:04:56.586 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.587 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.587 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:56.587 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:50.671086+0000", 2024-04-19T01:04:56.587 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4734999999999996e-05 2024-04-19T01:04:56.587 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.587 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:56.587 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.587 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.587 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.587 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.3s5 80/31 ECSubWrite(tid=64, reqid=client.4604.0:613, at_version=80'52, trim_to=0'0, roll_forward_to=57'49))", 2024-04-19T01:04:56.587 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:50.675422+0000", 2024-04-19T01:04:56.587 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.8309128350000003, 2024-04-19T01:04:56.587 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0087839029999999992, 2024-04-19T01:04:56.587 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:56.587 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:56.588 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:56.588 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:56.588 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.588 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:56.588 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:50.675422+0000", 2024-04-19T01:04:56.588 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.588 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.588 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.588 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:56.588 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:50.675422+0000", 2024-04-19T01:04:56.588 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.588 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.588 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.588 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:56.588 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:50.675424+0000", 2024-04-19T01:04:56.589 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0880000000000002e-06 2024-04-19T01:04:56.589 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.589 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.589 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:56.589 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:50.675802+0000", 2024-04-19T01:04:56.589 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00037775500000000002 2024-04-19T01:04:56.589 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.589 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.589 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:56.589 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:50.675832+0000", 2024-04-19T01:04:56.589 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0280000000000001e-05 2024-04-19T01:04:56.589 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.589 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.590 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:56.590 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:50.675852+0000", 2024-04-19T01:04:56.590 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9885e-05 2024-04-19T01:04:56.590 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.590 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.590 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:56.590 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:50.675950+0000", 2024-04-19T01:04:56.590 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7322000000000001e-05 2024-04-19T01:04:56.590 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.590 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.590 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:56.590 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:50.675964+0000", 2024-04-19T01:04:56.590 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4652e-05 2024-04-19T01:04:56.590 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.590 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.591 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:56.591 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:50.684159+0000", 2024-04-19T01:04:56.591 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0081944519999999996 2024-04-19T01:04:56.591 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.591 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.591 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:56.591 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:50.684206+0000", 2024-04-19T01:04:56.591 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7469000000000002e-05 2024-04-19T01:04:56.591 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.591 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:56.591 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.591 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.591 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.591 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds1 84/31 ECSubWrite(tid=171, reqid=osd.12.0:168, at_version=0'0, trim_to=0'0, roll_forward_to=84'70))", 2024-04-19T01:04:56.592 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:54.560624+0000", 2024-04-19T01:04:56.592 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 1.9457107499999999, 2024-04-19T01:04:56.592 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0082320850000000001, 2024-04-19T01:04:56.592 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:56.592 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:56.592 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:56.592 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:56.592 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.592 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:56.592 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:54.560624+0000", 2024-04-19T01:04:56.592 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.592 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.592 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.592 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:56.593 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:54.560624+0000", 2024-04-19T01:04:56.593 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.593 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.593 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.593 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:56.593 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:54.560625+0000", 2024-04-19T01:04:56.593 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9100000000000002e-07 2024-04-19T01:04:56.593 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.593 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.593 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:56.593 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:54.560636+0000", 2024-04-19T01:04:56.593 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1157e-05 2024-04-19T01:04:56.593 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.593 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.593 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:56.594 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:54.560652+0000", 2024-04-19T01:04:56.594 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5582000000000001e-05 2024-04-19T01:04:56.594 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.594 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.594 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:56.594 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:54.560661+0000", 2024-04-19T01:04:56.594 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9530000000000005e-06 2024-04-19T01:04:56.594 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.594 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.594 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:56.594 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:54.560743+0000", 2024-04-19T01:04:56.594 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.2497000000000007e-05 2024-04-19T01:04:56.595 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.595 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.595 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:56.595 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:54.560757+0000", 2024-04-19T01:04:56.595 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4082e-05 2024-04-19T01:04:56.595 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.595 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.595 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:56.595 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:54.568614+0000", 2024-04-19T01:04:56.595 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0078566939999999991 2024-04-19T01:04:56.595 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.595 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.595 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:56.596 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:54.568856+0000", 2024-04-19T01:04:56.596 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000242229 2024-04-19T01:04:56.596 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.596 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:56.596 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.596 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.596 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.596 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.3s5 85/31 ECSubWrite(tid=74, reqid=osd.15.0:74, at_version=85'60, trim_to=0'0, roll_forward_to=84'58))", 2024-04-19T01:04:56.596 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:55.557963+0000", 2024-04-19T01:04:56.596 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.94837169399999999, 2024-04-19T01:04:56.596 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0093867440000000007, 2024-04-19T01:04:56.596 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:56.596 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:56.596 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:04:56.597 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:56.597 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.597 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:56.597 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:55.557963+0000", 2024-04-19T01:04:56.597 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.597 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.597 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.597 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:56.597 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:55.557963+0000", 2024-04-19T01:04:56.597 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:56.597 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.597 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.597 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:56.597 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:55.557964+0000", 2024-04-19T01:04:56.597 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.2399999999999997e-07 2024-04-19T01:04:56.598 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.598 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.598 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:56.598 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:55.558011+0000", 2024-04-19T01:04:56.598 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6854000000000002e-05 2024-04-19T01:04:56.598 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.598 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.598 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:56.598 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:55.558043+0000", 2024-04-19T01:04:56.599 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2382000000000003e-05 2024-04-19T01:04:56.599 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.599 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.599 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:56.599 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:55.558065+0000", 2024-04-19T01:04:56.599 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1216000000000002e-05 2024-04-19T01:04:56.599 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.599 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.599 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:56.599 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:55.558281+0000", 2024-04-19T01:04:56.599 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000216689 2024-04-19T01:04:56.599 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.600 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.600 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:56.600 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:55.558309+0000", 2024-04-19T01:04:56.600 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7276000000000001e-05 2024-04-19T01:04:56.600 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.600 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.600 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:56.600 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:55.567268+0000", 2024-04-19T01:04:56.600 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0089592189999999992 2024-04-19T01:04:56.600 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:56.600 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:56.600 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:56.600 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:55.567350+0000", 2024-04-19T01:04:56.601 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.2284e-05 2024-04-19T01:04:56.601 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.601 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:56.601 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.601 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:56.601 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:56.601 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:56.601 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2024-04-19T01:04:56.602 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-19T01:04:56.605 INFO:tasks.rados.rados.0.smithi155.stdout:269: done (1 left) 2024-04-19T01:04:56.605 INFO:tasks.rados.rados.0.smithi155.stdout:270: delete oid 15 current snap is 28 2024-04-19T01:04:56.605 INFO:tasks.rados.rados.0.smithi155.stdout:266: finishing rollback tid 0 to smithi15528762-5 2024-04-19T01:04:56.605 INFO:tasks.rados.rados.0.smithi155.stdout:266: finishing rollback tid 1 to smithi15528762-5 2024-04-19T01:04:56.605 INFO:tasks.rados.rados.0.smithi155.stdout:266: finishing rollback tid 2 to smithi15528762-5 2024-04-19T01:04:56.605 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 5 v 67 (ObjNum 72 snap 8 seq_num 72) dirty exists 2024-04-19T01:04:56.605 INFO:tasks.rados.rados.0.smithi155.stdout:266: done (1 left) 2024-04-19T01:04:56.605 INFO:tasks.rados.rados.0.smithi155.stdout:270: done (0 left) 2024-04-19T01:04:56.605 INFO:tasks.rados.rados.0.smithi155.stdout:271: copy_from oid 27 from oid 47 current snap is 28 2024-04-19T01:04:56.605 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 10 current snap is 28 2024-04-19T01:04:56.605 INFO:tasks.rados.rados.0.smithi155.stdout:272: seq_num 115 ranges {327680=770048,1097728=704512,1802240=458752} 2024-04-19T01:04:56.606 INFO:tasks.rados.rados.0.smithi155.stdout:272: writing smithi15528762-10 from 327680 to 1097728 tid 1 2024-04-19T01:04:56.606 INFO:tasks.rados.rados.0.smithi155.stdout:272: writing smithi15528762-10 from 1097728 to 1802240 tid 2 2024-04-19T01:04:56.606 INFO:tasks.rados.rados.0.smithi155.stdout:272: writing smithi15528762-10 from 1802240 to 2260992 tid 3 2024-04-19T01:04:56.606 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 6 current snap is 28 2024-04-19T01:04:56.606 INFO:tasks.rados.rados.0.smithi155.stdout:273: seq_num 116 ranges {2670592=786432,3457024=458752,3915776=442368,4358144=262144} 2024-04-19T01:04:56.606 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:04:56.603+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:04:56.630 INFO:tasks.rados.rados.0.smithi155.stdout:273: writing smithi15528762-6 from 2670592 to 3457024 tid 1 2024-04-19T01:04:56.632 INFO:tasks.rados.rados.0.smithi155.stdout:273: writing smithi15528762-6 from 3457024 to 3915776 tid 2 2024-04-19T01:04:56.636 INFO:tasks.rados.rados.0.smithi155.stdout:273: writing smithi15528762-6 from 3915776 to 4358144 tid 3 2024-04-19T01:04:56.638 INFO:tasks.rados.rados.0.smithi155.stdout:273: writing smithi15528762-6 from 4358144 to 4620288 tid 4 2024-04-19T01:04:56.638 INFO:tasks.rados.rados.0.smithi155.stdout:274: snap_remove snap 25 2024-04-19T01:04:56.706 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:04:56.706+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:04:56.773 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:56.773 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:04:56.773 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2024-04-19T01:04:56.773 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:56.787 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-19T01:04:56.807 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:04:56.807+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:04:56.898 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-19T01:04:56.912 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:56.915+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:56.960 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:56.960 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:04:56.960 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2024-04-19T01:04:56.960 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2024-04-19T01:04:56.960 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:56.976 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-19T01:04:57.022 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:04:57.022+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:04:57.050 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-04-19T01:04:57.051 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T01:04:57.072 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-04-19T01:04:57.072 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T01:04:57.096 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-04-19T01:04:57.096 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T01:04:57.103 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:04:57.103 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2024-04-19T01:04:57.103 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2024-04-19T01:04:57.103 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-04-19T01:04:57.103 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.103 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:423 3.5s0 3:a03c5c89:::smithi15528762-37:head [append 0~655360] snapc e=[d] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-19T01:04:57.104 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:29.537904+0000", 2024-04-19T01:04:57.104 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 27.559978214000001, 2024-04-19T01:04:57.104 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.049520382000000002, 2024-04-19T01:04:57.104 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:57.104 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:57.104 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:57.104 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:57.104 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:57.104 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:57.104 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 423 2024-04-19T01:04:57.104 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.104 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:57.105 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.105 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:57.105 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.537904+0000", 2024-04-19T01:04:57.105 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:57.105 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.105 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.105 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:57.105 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.537904+0000", 2024-04-19T01:04:57.105 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:57.105 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.105 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.106 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:57.106 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.537906+0000", 2024-04-19T01:04:57.106 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7859999999999999e-06 2024-04-19T01:04:57.106 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.106 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.106 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:57.106 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.538813+0000", 2024-04-19T01:04:57.106 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00090743099999999999 2024-04-19T01:04:57.107 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.107 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.107 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:57.107 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.538828+0000", 2024-04-19T01:04:57.108 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5206e-05 2024-04-19T01:04:57.108 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.108 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.108 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:57.108 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.538838+0000", 2024-04-19T01:04:57.108 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.3070000000000008e-06 2024-04-19T01:04:57.108 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.108 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.109 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:57.109 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.538910+0000", 2024-04-19T01:04:57.109 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2206000000000006e-05 2024-04-19T01:04:57.109 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.109 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.109 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:57.109 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.539016+0000", 2024-04-19T01:04:57.109 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010665199999999999 2024-04-19T01:04:57.109 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.109 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.109 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:57.109 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.540080+0000", 2024-04-19T01:04:57.110 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001063273 2024-04-19T01:04:57.110 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.110 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.110 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:57.110 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.545973+0000", 2024-04-19T01:04:57.110 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0058930570000000002 2024-04-19T01:04:57.110 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.110 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.110 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:57.110 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.587359+0000", 2024-04-19T01:04:57.110 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.041386128000000001 2024-04-19T01:04:57.110 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.111 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.111 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:57.111 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.587424+0000", 2024-04-19T01:04:57.111 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5335999999999999e-05 2024-04-19T01:04:57.111 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:57.111 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:57.111 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:57.111 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.111 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.111 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:424 3.5s0 3:a03c5c89:::smithi15528762-37:head [append 0~638976] snapc e=[d] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-19T01:04:57.111 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:29.545481+0000", 2024-04-19T01:04:57.111 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 27.552401226000001, 2024-04-19T01:04:57.112 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.042187482999999998, 2024-04-19T01:04:57.112 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:57.112 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:57.112 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:57.112 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:57.112 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:57.112 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:57.112 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 424 2024-04-19T01:04:57.112 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.112 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:57.112 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.112 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:57.112 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.545481+0000", 2024-04-19T01:04:57.113 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:57.113 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.113 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.113 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:57.113 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.545481+0000", 2024-04-19T01:04:57.113 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:57.113 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.113 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.113 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:57.113 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.545483+0000", 2024-04-19T01:04:57.114 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3879999999999998e-06 2024-04-19T01:04:57.114 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.114 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.114 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:57.114 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.546298+0000", 2024-04-19T01:04:57.114 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00081462099999999996 2024-04-19T01:04:57.114 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.114 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.114 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:57.114 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.546314+0000", 2024-04-19T01:04:57.114 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5722e-05 2024-04-19T01:04:57.115 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.115 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.115 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:57.115 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.546320+0000", 2024-04-19T01:04:57.115 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0939999999999996e-06 2024-04-19T01:04:57.115 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.115 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.115 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:57.115 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.546381+0000", 2024-04-19T01:04:57.115 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1696e-05 2024-04-19T01:04:57.115 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.116 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.116 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:57.116 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.546497+0000", 2024-04-19T01:04:57.116 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000115903 2024-04-19T01:04:57.116 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.116 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.116 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:57.116 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547522+0000", 2024-04-19T01:04:57.116 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010249289999999999 2024-04-19T01:04:57.116 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.116 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.117 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:57.117 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.556940+0000", 2024-04-19T01:04:57.118 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0094178899999999999 2024-04-19T01:04:57.118 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.118 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.118 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:57.119 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.587578+0000", 2024-04-19T01:04:57.119 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.030638021000000001 2024-04-19T01:04:57.119 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.119 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.119 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:57.119 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.587668+0000", 2024-04-19T01:04:57.120 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.0218999999999998e-05 2024-04-19T01:04:57.120 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:57.120 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:57.120 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:57.120 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.120 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.120 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:425 3.5s0 3:a03c5c89:::smithi15528762-37:head [append 0~131072] snapc e=[d] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-19T01:04:57.120 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:29.547018+0000", 2024-04-19T01:04:57.120 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 27.550863785000001, 2024-04-19T01:04:57.120 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.040774228000000003, 2024-04-19T01:04:57.121 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:57.121 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:57.121 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:57.121 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:57.121 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:57.121 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:57.121 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 425 2024-04-19T01:04:57.121 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.121 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:57.121 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.122 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:57.122 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547018+0000", 2024-04-19T01:04:57.122 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:57.122 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.122 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.122 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:57.122 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547018+0000", 2024-04-19T01:04:57.122 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:57.122 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.122 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.122 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:57.123 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547020+0000", 2024-04-19T01:04:57.123 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3710000000000001e-06 2024-04-19T01:04:57.123 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.123 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.123 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:57.123 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547148+0000", 2024-04-19T01:04:57.123 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012848699999999999 2024-04-19T01:04:57.123 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.123 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.124 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:57.124 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547162+0000", 2024-04-19T01:04:57.124 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4248e-05 2024-04-19T01:04:57.124 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.124 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.124 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:57.124 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547170+0000", 2024-04-19T01:04:57.124 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2069999999999998e-06 2024-04-19T01:04:57.125 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.125 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.125 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:57.125 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.554554+0000", 2024-04-19T01:04:57.125 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0073842159999999999 2024-04-19T01:04:57.125 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.125 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.125 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:57.125 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.554666+0000", 2024-04-19T01:04:57.125 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011222699999999999 2024-04-19T01:04:57.125 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.125 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.126 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:57.126 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.555080+0000", 2024-04-19T01:04:57.126 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00041430500000000001 2024-04-19T01:04:57.126 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.126 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.126 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:57.126 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.557075+0000", 2024-04-19T01:04:57.126 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001994198 2024-04-19T01:04:57.127 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.127 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.127 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:57.127 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.587746+0000", 2024-04-19T01:04:57.127 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.030671032000000001 2024-04-19T01:04:57.127 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.127 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.127 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:57.127 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.587793+0000", 2024-04-19T01:04:57.127 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6937000000000001e-05 2024-04-19T01:04:57.127 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:57.128 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:57.129 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:57.129 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.129 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.129 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:426 3.5s0 3:a03c5c89:::smithi15528762-37:head [setxattr _header (57) in=7b] snapc e=[d] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-19T01:04:57.129 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:29.547183+0000", 2024-04-19T01:04:57.129 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 27.550698950000001, 2024-04-19T01:04:57.129 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.040837638000000002, 2024-04-19T01:04:57.130 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:57.130 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:57.130 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:57.130 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:57.130 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:57.130 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:57.131 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 426 2024-04-19T01:04:57.131 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.131 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:57.131 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.131 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:57.131 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547183+0000", 2024-04-19T01:04:57.131 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:57.131 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.131 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.132 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:57.132 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547183+0000", 2024-04-19T01:04:57.132 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:57.132 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.132 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.132 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:57.132 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547184+0000", 2024-04-19T01:04:57.132 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.6899999999999997e-07 2024-04-19T01:04:57.132 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.133 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.133 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:57.133 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547188+0000", 2024-04-19T01:04:57.133 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4150000000000001e-06 2024-04-19T01:04:57.133 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.133 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.133 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:57.133 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547196+0000", 2024-04-19T01:04:57.133 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.8660000000000006e-06 2024-04-19T01:04:57.133 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.133 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.133 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:57.134 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547203+0000", 2024-04-19T01:04:57.134 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.0550000000000002e-06 2024-04-19T01:04:57.134 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.134 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.134 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:57.134 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.556074+0000", 2024-04-19T01:04:57.134 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0088710260000000006 2024-04-19T01:04:57.134 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.134 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.134 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:57.134 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.556179+0000", 2024-04-19T01:04:57.134 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000104947 2024-04-19T01:04:57.135 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.135 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.135 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:57.135 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.556694+0000", 2024-04-19T01:04:57.135 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000514563 2024-04-19T01:04:57.135 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.135 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.135 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:57.136 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.557301+0000", 2024-04-19T01:04:57.136 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00060763899999999999 2024-04-19T01:04:57.136 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.136 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.136 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:57.136 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.587923+0000", 2024-04-19T01:04:57.136 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.030621770999999999 2024-04-19T01:04:57.136 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.136 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.136 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:57.136 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.588021+0000", 2024-04-19T01:04:57.137 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7687000000000002e-05 2024-04-19T01:04:57.137 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:57.137 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:57.137 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:57.137 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.137 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.137 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:427 3.5s0 3:a03c5c89:::smithi15528762-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e59)", 2024-04-19T01:04:57.137 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:29.547214+0000", 2024-04-19T01:04:57.137 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 27.550668435999999, 2024-04-19T01:04:57.138 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.043029057000000002, 2024-04-19T01:04:57.138 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:57.138 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:57.138 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:57.140 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:57.140 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:57.140 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:57.140 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 427 2024-04-19T01:04:57.145 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.160 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:57.160 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.160 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:57.160 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547214+0000", 2024-04-19T01:04:57.160 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:57.160 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.160 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.160 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:57.160 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547214+0000", 2024-04-19T01:04:57.160 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:57.160 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.160 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.160 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:57.161 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547214+0000", 2024-04-19T01:04:57.161 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3799999999999998e-07 2024-04-19T01:04:57.161 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.161 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.161 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:57.161 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547216+0000", 2024-04-19T01:04:57.161 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.131e-06 2024-04-19T01:04:57.161 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.161 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.161 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:57.161 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547223+0000", 2024-04-19T01:04:57.161 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2790000000000004e-06 2024-04-19T01:04:57.161 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.161 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.162 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:57.162 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547226+0000", 2024-04-19T01:04:57.162 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4529999999999998e-06 2024-04-19T01:04:57.162 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.162 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.162 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:57.162 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.556968+0000", 2024-04-19T01:04:57.162 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0097420590000000008 2024-04-19T01:04:57.162 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.162 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.162 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:04:57.162 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.557060+0000", 2024-04-19T01:04:57.162 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.1807999999999999e-05 2024-04-19T01:04:57.162 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.163 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.163 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:57.163 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.588480+0000", 2024-04-19T01:04:57.163 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.031420333000000002 2024-04-19T01:04:57.163 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.163 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.163 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:57.163 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.588538+0000", 2024-04-19T01:04:57.163 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.7652000000000003e-05 2024-04-19T01:04:57.163 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.163 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.163 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:57.163 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.590243+0000", 2024-04-19T01:04:57.163 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0017049039999999999 2024-04-19T01:04:57.164 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:57.164 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:57.164 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:57.164 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.164 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.164 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:543 3.5s0 3:a8ddfdee:::smithi15528762-22:head [append 0~524288] snapc 17=[17,15,14,12] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-04-19T01:04:57.164 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:44.548241+0000", 2024-04-19T01:04:57.164 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 12.549641127999999, 2024-04-19T01:04:57.164 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.049103814000000003, 2024-04-19T01:04:57.164 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:57.164 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:57.164 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:57.164 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:57.165 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:57.165 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:57.165 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 543 2024-04-19T01:04:57.165 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.165 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:57.165 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.165 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:57.165 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.548241+0000", 2024-04-19T01:04:57.165 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:57.165 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.165 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.165 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:57.165 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.548241+0000", 2024-04-19T01:04:57.165 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:57.166 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.166 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.166 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:57.166 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.548244+0000", 2024-04-19T01:04:57.166 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.1609999999999999e-06 2024-04-19T01:04:57.166 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.166 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.166 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:57.166 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.548890+0000", 2024-04-19T01:04:57.166 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00064546500000000001 2024-04-19T01:04:57.166 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.166 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.166 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:57.167 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.548916+0000", 2024-04-19T01:04:57.167 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6563999999999998e-05 2024-04-19T01:04:57.167 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.167 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.167 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:57.167 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.548934+0000", 2024-04-19T01:04:57.167 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7351e-05 2024-04-19T01:04:57.167 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.167 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.167 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:57.167 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.549031+0000", 2024-04-19T01:04:57.167 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7528000000000005e-05 2024-04-19T01:04:57.167 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.167 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.167 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op must wait for map", 2024-04-19T01:04:57.168 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.549037+0000", 2024-04-19T01:04:57.168 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.7590000000000003e-06 2024-04-19T01:04:57.168 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.168 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.168 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:57.168 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.551497+0000", 2024-04-19T01:04:57.168 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0024606580000000001 2024-04-19T01:04:57.173 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.174 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.174 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:57.174 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.551597+0000", 2024-04-19T01:04:57.174 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.9525999999999995e-05 2024-04-19T01:04:57.174 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.174 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.174 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:57.174 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.552924+0000", 2024-04-19T01:04:57.174 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001327356 2024-04-19T01:04:57.174 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.174 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.174 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:57.175 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.559310+0000", 2024-04-19T01:04:57.175 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0063859900000000002 2024-04-19T01:04:57.175 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.175 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.175 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:57.175 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.597268+0000", 2024-04-19T01:04:57.175 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.037957492000000002 2024-04-19T01:04:57.175 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.175 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.175 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:57.175 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.597345+0000", 2024-04-19T01:04:57.175 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6964000000000004e-05 2024-04-19T01:04:57.175 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:57.175 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:57.176 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:57.176 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.176 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.176 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:591 3.5s0 3:a626fd0d:::smithi15528762-17:head [append 0~655360] snapc 1a=[19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-19T01:04:57.176 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:49.652801+0000", 2024-04-19T01:04:57.176 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 7.445081429, 2024-04-19T01:04:57.176 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.051043682999999999, 2024-04-19T01:04:57.176 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:57.176 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:57.176 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:57.176 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:57.176 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:57.176 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:57.177 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 591 2024-04-19T01:04:57.177 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.177 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:57.177 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.177 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:57.177 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.652801+0000", 2024-04-19T01:04:57.177 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:57.177 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.177 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.177 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:57.178 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.652801+0000", 2024-04-19T01:04:57.178 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:57.178 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.178 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.178 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:57.178 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.652826+0000", 2024-04-19T01:04:57.178 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5695e-05 2024-04-19T01:04:57.178 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.178 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.178 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:57.178 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.654116+0000", 2024-04-19T01:04:57.178 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001289807 2024-04-19T01:04:57.179 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.179 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.179 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:57.179 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.654135+0000", 2024-04-19T01:04:57.179 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8734999999999999e-05 2024-04-19T01:04:57.179 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.179 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.179 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:57.179 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.654147+0000", 2024-04-19T01:04:57.179 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2140000000000001e-05 2024-04-19T01:04:57.179 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.179 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.179 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:57.179 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.654229+0000", 2024-04-19T01:04:57.179 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1798000000000005e-05 2024-04-19T01:04:57.180 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.180 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.180 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:57.180 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.654371+0000", 2024-04-19T01:04:57.180 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000141817 2024-04-19T01:04:57.180 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.180 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.180 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:57.180 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.656095+0000", 2024-04-19T01:04:57.180 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00172386 2024-04-19T01:04:57.180 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.180 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.180 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:57.180 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.679643+0000", 2024-04-19T01:04:57.180 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.023548158999999999 2024-04-19T01:04:57.181 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.181 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.181 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:57.181 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.703771+0000", 2024-04-19T01:04:57.181 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.024128437999999999 2024-04-19T01:04:57.181 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.181 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.181 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:57.181 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.703844+0000", 2024-04-19T01:04:57.181 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.3233999999999997e-05 2024-04-19T01:04:57.181 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:57.181 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:57.181 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:57.181 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.182 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.182 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:592 3.5s0 3:a626fd0d:::smithi15528762-17:head [append 0~606208] snapc 1a=[19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-19T01:04:57.182 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:49.658117+0000", 2024-04-19T01:04:57.182 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 7.4397649619999999, 2024-04-19T01:04:57.182 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.047890332000000001, 2024-04-19T01:04:57.182 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:57.182 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:57.182 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:57.182 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:57.182 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:57.183 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:57.183 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 592 2024-04-19T01:04:57.183 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.183 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:57.183 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.183 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:57.183 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.658117+0000", 2024-04-19T01:04:57.183 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:57.183 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.183 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.183 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:57.183 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.658117+0000", 2024-04-19T01:04:57.183 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:57.183 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.183 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.184 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:57.184 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.658120+0000", 2024-04-19T01:04:57.184 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9129999999999999e-06 2024-04-19T01:04:57.184 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.184 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.184 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:57.184 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.658931+0000", 2024-04-19T01:04:57.184 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00081110999999999998 2024-04-19T01:04:57.184 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.184 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.184 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:57.185 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.658965+0000", 2024-04-19T01:04:57.185 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4001999999999997e-05 2024-04-19T01:04:57.185 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.185 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.185 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:57.185 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.658981+0000", 2024-04-19T01:04:57.185 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5443000000000001e-05 2024-04-19T01:04:57.185 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.185 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.185 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:57.185 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663331+0000", 2024-04-19T01:04:57.185 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0043501240000000004 2024-04-19T01:04:57.185 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.185 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.185 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:57.186 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663457+0000", 2024-04-19T01:04:57.186 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000125857 2024-04-19T01:04:57.186 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.186 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.186 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:57.186 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.664259+0000", 2024-04-19T01:04:57.186 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00080283700000000004 2024-04-19T01:04:57.186 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.186 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.186 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:57.186 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.679757+0000", 2024-04-19T01:04:57.186 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015497100999999999 2024-04-19T01:04:57.186 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.187 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.187 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:57.187 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.705944+0000", 2024-04-19T01:04:57.187 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.026187892000000001 2024-04-19T01:04:57.187 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.187 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.187 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:57.187 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.706007+0000", 2024-04-19T01:04:57.187 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3052999999999994e-05 2024-04-19T01:04:57.187 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:57.187 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:57.187 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:57.187 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.187 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.187 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:593 3.5s0 3:a626fd0d:::smithi15528762-17:head [append 0~475136] snapc 1a=[19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-19T01:04:57.188 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:49.662459+0000", 2024-04-19T01:04:57.188 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 7.4354229280000004, 2024-04-19T01:04:57.188 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.043664481999999998, 2024-04-19T01:04:57.188 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:57.188 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:57.188 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:57.188 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:57.188 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:57.188 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:57.188 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 593 2024-04-19T01:04:57.188 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.188 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:57.188 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.188 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:57.188 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.662459+0000", 2024-04-19T01:04:57.189 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:57.189 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.189 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.189 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:57.189 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.662459+0000", 2024-04-19T01:04:57.189 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:57.189 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.189 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.190 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:57.190 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.662465+0000", 2024-04-19T01:04:57.190 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2029999999999997e-06 2024-04-19T01:04:57.190 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.190 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.190 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:57.190 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663095+0000", 2024-04-19T01:04:57.190 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00062950400000000002 2024-04-19T01:04:57.190 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.190 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.190 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:57.190 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663191+0000", 2024-04-19T01:04:57.190 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.5866999999999995e-05 2024-04-19T01:04:57.191 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.191 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.191 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:57.191 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663213+0000", 2024-04-19T01:04:57.191 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1832e-05 2024-04-19T01:04:57.191 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.191 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.191 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:57.191 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.671032+0000", 2024-04-19T01:04:57.191 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0078190310000000006 2024-04-19T01:04:57.191 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.191 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.191 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:57.191 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.671178+0000", 2024-04-19T01:04:57.191 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014599400000000001 2024-04-19T01:04:57.192 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.192 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.192 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:57.192 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.672264+0000", 2024-04-19T01:04:57.192 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001086652 2024-04-19T01:04:57.192 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.192 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.192 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:57.192 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.679758+0000", 2024-04-19T01:04:57.192 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0074936059999999999 2024-04-19T01:04:57.192 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.192 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.192 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:57.192 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.706080+0000", 2024-04-19T01:04:57.193 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.026322534000000002 2024-04-19T01:04:57.193 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.193 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.193 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:57.193 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.706124+0000", 2024-04-19T01:04:57.193 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3259e-05 2024-04-19T01:04:57.193 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:57.193 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:57.193 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:57.193 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.193 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.193 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:594 3.5s0 3:a626fd0d:::smithi15528762-17:head [append 0~81920] snapc 1a=[19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-19T01:04:57.194 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:49.663273+0000", 2024-04-19T01:04:57.194 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 7.4346089129999999, 2024-04-19T01:04:57.194 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.043572570999999997, 2024-04-19T01:04:57.194 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:57.194 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:57.194 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:57.194 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:57.209 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:57.209 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:57.209 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 594 2024-04-19T01:04:57.209 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.210 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:57.210 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.210 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:57.210 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663273+0000", 2024-04-19T01:04:57.210 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:57.210 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.210 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.210 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:57.210 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663273+0000", 2024-04-19T01:04:57.210 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:57.210 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.210 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.210 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:57.211 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663275+0000", 2024-04-19T01:04:57.211 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.421e-06 2024-04-19T01:04:57.211 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.211 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.211 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:57.211 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663327+0000", 2024-04-19T01:04:57.211 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2669e-05 2024-04-19T01:04:57.211 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.211 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.211 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:57.211 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663342+0000", 2024-04-19T01:04:57.211 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5046e-05 2024-04-19T01:04:57.211 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.211 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.212 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:57.212 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663351+0000", 2024-04-19T01:04:57.212 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.0610000000000002e-06 2024-04-19T01:04:57.212 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.212 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.212 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:57.212 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.676918+0000", 2024-04-19T01:04:57.212 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013566147000000001 2024-04-19T01:04:57.212 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.212 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.212 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:57.212 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.677079+0000", 2024-04-19T01:04:57.212 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000161683 2024-04-19T01:04:57.212 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.219 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.220 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:57.220 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.678015+0000", 2024-04-19T01:04:57.220 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00093627399999999996 2024-04-19T01:04:57.220 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.221 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.221 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:57.221 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.681601+0000", 2024-04-19T01:04:57.221 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003585464 2024-04-19T01:04:57.222 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.222 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.222 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:57.222 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.706786+0000", 2024-04-19T01:04:57.222 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.025185553999999999 2024-04-19T01:04:57.222 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.222 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.222 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:57.222 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.706846+0000", 2024-04-19T01:04:57.222 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.9252000000000001e-05 2024-04-19T01:04:57.222 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:57.224 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:57.224 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:57.224 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.224 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.224 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:595 3.5s0 3:a626fd0d:::smithi15528762-17:head [setxattr _header (57) in=7b] snapc 1a=[19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-19T01:04:57.224 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:49.663370+0000", 2024-04-19T01:04:57.225 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 7.4345124550000001, 2024-04-19T01:04:57.225 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.044174650000000003, 2024-04-19T01:04:57.225 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:57.225 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:57.225 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:57.225 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:57.225 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:57.225 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:57.225 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 595 2024-04-19T01:04:57.225 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.225 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:57.225 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.225 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:57.225 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663370+0000", 2024-04-19T01:04:57.225 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:57.226 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.226 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.226 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:57.226 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663370+0000", 2024-04-19T01:04:57.226 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:57.226 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.226 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.226 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:57.226 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663371+0000", 2024-04-19T01:04:57.227 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.3699999999999999e-07 2024-04-19T01:04:57.227 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.227 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.227 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:57.227 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663376+0000", 2024-04-19T01:04:57.227 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6879999999999996e-06 2024-04-19T01:04:57.227 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.227 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.227 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:57.227 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663386+0000", 2024-04-19T01:04:57.227 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7829999999999993e-06 2024-04-19T01:04:57.227 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.227 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.227 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:57.227 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663393+0000", 2024-04-19T01:04:57.228 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2540000000000003e-06 2024-04-19T01:04:57.228 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.228 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.228 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:57.228 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.679885+0000", 2024-04-19T01:04:57.228 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.016492168000000001 2024-04-19T01:04:57.228 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.228 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.228 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:57.228 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.679999+0000", 2024-04-19T01:04:57.228 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000113638 2024-04-19T01:04:57.228 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.228 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.228 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:57.229 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.680695+0000", 2024-04-19T01:04:57.229 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00069550999999999999 2024-04-19T01:04:57.229 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.229 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.229 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:57.229 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.681605+0000", 2024-04-19T01:04:57.229 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00091078000000000003 2024-04-19T01:04:57.229 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.229 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.229 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:57.229 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.707485+0000", 2024-04-19T01:04:57.229 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.025879491000000001 2024-04-19T01:04:57.229 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.229 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.229 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:57.230 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.707544+0000", 2024-04-19T01:04:57.230 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.9401000000000003e-05 2024-04-19T01:04:57.230 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:57.230 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:57.230 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:57.230 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.230 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.230 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:596 3.5s0 3:a626fd0d:::smithi15528762-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e79)", 2024-04-19T01:04:57.230 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:49.663401+0000", 2024-04-19T01:04:57.231 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 7.434480744, 2024-04-19T01:04:57.231 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.045794460000000002, 2024-04-19T01:04:57.231 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:57.231 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:57.231 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:57.231 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:57.231 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:57.231 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:57.231 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 596 2024-04-19T01:04:57.231 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.231 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:57.231 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.231 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:57.231 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663401+0000", 2024-04-19T01:04:57.232 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:57.232 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.232 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.232 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:57.232 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663401+0000", 2024-04-19T01:04:57.232 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:57.232 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.232 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.232 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:57.232 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663402+0000", 2024-04-19T01:04:57.232 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2800000000000002e-07 2024-04-19T01:04:57.232 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.232 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.232 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:57.232 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663406+0000", 2024-04-19T01:04:57.233 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2830000000000003e-06 2024-04-19T01:04:57.233 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.233 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.233 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:57.233 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663412+0000", 2024-04-19T01:04:57.234 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.784e-06 2024-04-19T01:04:57.234 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.234 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.234 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:57.234 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663416+0000", 2024-04-19T01:04:57.234 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1339999999999997e-06 2024-04-19T01:04:57.234 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.234 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.235 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:57.235 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.681057+0000", 2024-04-19T01:04:57.235 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017640841000000001 2024-04-19T01:04:57.235 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.235 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.235 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:04:57.235 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.681119+0000", 2024-04-19T01:04:57.235 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2592e-05 2024-04-19T01:04:57.235 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.235 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.235 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:57.235 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.707942+0000", 2024-04-19T01:04:57.235 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.026822915999999999 2024-04-19T01:04:57.235 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.235 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.236 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:57.236 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.708008+0000", 2024-04-19T01:04:57.236 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.6088999999999997e-05 2024-04-19T01:04:57.236 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.236 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.236 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:57.236 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.709196+0000", 2024-04-19T01:04:57.236 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0011873929999999999 2024-04-19T01:04:57.236 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:57.236 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:57.236 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:57.236 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.236 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.236 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:626 3.5s0 3:ab059e23:::smithi15528762-38:head [append 0~737280] snapc 1c=[1c,1b,19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-19T01:04:57.236 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:51.587820+0000", 2024-04-19T01:04:57.237 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.5100625369999996, 2024-04-19T01:04:57.237 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.070148642999999997, 2024-04-19T01:04:57.237 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:57.237 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:57.237 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:57.237 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:57.237 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:57.237 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:57.237 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 626 2024-04-19T01:04:57.237 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.237 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:57.237 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.237 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:57.237 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.587820+0000", 2024-04-19T01:04:57.237 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:57.238 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.238 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.238 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:57.238 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.587820+0000", 2024-04-19T01:04:57.238 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:57.238 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.238 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.238 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:57.238 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.587822+0000", 2024-04-19T01:04:57.238 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5150000000000001e-06 2024-04-19T01:04:57.238 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.238 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.238 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:57.238 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.588850+0000", 2024-04-19T01:04:57.239 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010283029999999999 2024-04-19T01:04:57.239 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.239 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.239 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:57.239 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.588877+0000", 2024-04-19T01:04:57.239 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6593000000000001e-05 2024-04-19T01:04:57.239 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.239 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.239 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:57.239 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.588891+0000", 2024-04-19T01:04:57.239 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4245e-05 2024-04-19T01:04:57.239 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.239 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.239 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:57.240 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.588997+0000", 2024-04-19T01:04:57.240 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000106252 2024-04-19T01:04:57.240 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.240 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.240 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op must wait for map", 2024-04-19T01:04:57.240 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.589004+0000", 2024-04-19T01:04:57.240 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.8079999999999999e-06 2024-04-19T01:04:57.240 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.240 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.240 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:57.240 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.591485+0000", 2024-04-19T01:04:57.240 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0024811970000000001 2024-04-19T01:04:57.240 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.240 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.240 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:57.241 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.591587+0000", 2024-04-19T01:04:57.241 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000101387 2024-04-19T01:04:57.241 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.241 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.241 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:57.241 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.592588+0000", 2024-04-19T01:04:57.241 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010014850000000001 2024-04-19T01:04:57.241 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.241 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.241 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:57.241 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.602598+0000", 2024-04-19T01:04:57.241 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010010023999999999 2024-04-19T01:04:57.241 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.241 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.241 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:57.242 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.657845+0000", 2024-04-19T01:04:57.242 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.055246157999999997 2024-04-19T01:04:57.242 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.242 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.242 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:57.242 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.657968+0000", 2024-04-19T01:04:57.242 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000123676 2024-04-19T01:04:57.242 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:57.242 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:57.242 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:57.243 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.243 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.243 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:627 3.5s0 3:ab059e23:::smithi15528762-38:head [append 0~557056] snapc 1c=[1c,1b,19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-19T01:04:57.243 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:51.594492+0000", 2024-04-19T01:04:57.243 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.5033899250000005, 2024-04-19T01:04:57.243 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.06373906, 2024-04-19T01:04:57.243 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:57.243 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:57.243 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:57.243 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:57.243 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:57.243 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:57.243 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 627 2024-04-19T01:04:57.243 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.244 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:57.244 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.244 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:57.244 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.594492+0000", 2024-04-19T01:04:57.244 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:57.244 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.244 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.244 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:57.245 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.594492+0000", 2024-04-19T01:04:57.245 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:57.245 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.245 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.245 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:57.245 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.594494+0000", 2024-04-19T01:04:57.245 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8899999999999999e-06 2024-04-19T01:04:57.245 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.245 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.245 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:57.245 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.595085+0000", 2024-04-19T01:04:57.245 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00059048800000000002 2024-04-19T01:04:57.245 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.245 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.246 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:57.246 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.595102+0000", 2024-04-19T01:04:57.246 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7515000000000001e-05 2024-04-19T01:04:57.246 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.246 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.246 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:57.246 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.595114+0000", 2024-04-19T01:04:57.246 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1678000000000001e-05 2024-04-19T01:04:57.246 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.246 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.246 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:57.246 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.598103+0000", 2024-04-19T01:04:57.246 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0029891789999999998 2024-04-19T01:04:57.246 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.247 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.247 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:57.247 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.598183+0000", 2024-04-19T01:04:57.247 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.9659000000000004e-05 2024-04-19T01:04:57.247 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.247 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.247 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:57.247 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.598883+0000", 2024-04-19T01:04:57.247 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00069997600000000005 2024-04-19T01:04:57.247 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.247 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.247 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:57.247 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.608539+0000", 2024-04-19T01:04:57.247 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0096562739999999994 2024-04-19T01:04:57.247 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.248 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.248 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:57.248 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.658124+0000", 2024-04-19T01:04:57.248 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.049585021 2024-04-19T01:04:57.248 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.248 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.248 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:57.248 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.658231+0000", 2024-04-19T01:04:57.248 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010738000000000001 2024-04-19T01:04:57.248 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:57.248 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:57.248 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:57.248 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.248 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.248 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:626 3.5s0 3:ab059e23:::smithi15528762-38:head [append 0~737280] snapc 1c=[1c,1b,19,18,17,14,12] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e81)", 2024-04-19T01:04:57.249 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:51.601568+0000", 2024-04-19T01:04:57.249 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.4963140370000003, 2024-04-19T01:04:57.249 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.056289715999999997, 2024-04-19T01:04:57.249 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:57.249 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:57.249 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "waiting for ondisk", 2024-04-19T01:04:57.249 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:57.249 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:57.249 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:57.249 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 626 2024-04-19T01:04:57.249 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.249 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:57.249 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.249 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:57.250 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.601568+0000", 2024-04-19T01:04:57.250 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:57.250 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.250 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.250 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:57.250 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.601568+0000", 2024-04-19T01:04:57.250 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:57.250 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.250 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.250 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:57.250 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.601569+0000", 2024-04-19T01:04:57.250 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.9999999999999997e-07 2024-04-19T01:04:57.250 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.250 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.251 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:57.251 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.602838+0000", 2024-04-19T01:04:57.251 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0012697349999999999 2024-04-19T01:04:57.251 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.251 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.251 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:57.251 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.602848+0000", 2024-04-19T01:04:57.251 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.6970000000000005e-06 2024-04-19T01:04:57.251 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.251 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.251 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:57.251 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.602858+0000", 2024-04-19T01:04:57.251 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.482e-06 2024-04-19T01:04:57.251 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.251 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.252 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:57.252 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.602901+0000", 2024-04-19T01:04:57.252 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3711999999999998e-05 2024-04-19T01:04:57.252 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.252 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.252 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for ondisk", 2024-04-19T01:04:57.252 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.602952+0000", 2024-04-19T01:04:57.252 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0481e-05 2024-04-19T01:04:57.252 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.252 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.252 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:57.252 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.657858+0000", 2024-04-19T01:04:57.252 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.054905909000000003 2024-04-19T01:04:57.252 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:57.252 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:57.253 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:57.253 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.253 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.253 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:627 3.5s0 3:ab059e23:::smithi15528762-38:head [append 0~557056] snapc 1c=[1c,1b,19,18,17,14,12] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e81)", 2024-04-19T01:04:57.253 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:51.602874+0000", 2024-04-19T01:04:57.253 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.4950082140000003, 2024-04-19T01:04:57.253 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.055263801000000001, 2024-04-19T01:04:57.253 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:57.253 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:57.253 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "waiting for ondisk", 2024-04-19T01:04:57.253 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:57.253 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:57.253 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:57.253 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 627 2024-04-19T01:04:57.254 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.254 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:57.254 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.254 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:57.254 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.602874+0000", 2024-04-19T01:04:57.260 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:57.260 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.260 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.260 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:57.260 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.602874+0000", 2024-04-19T01:04:57.260 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:57.260 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.260 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.260 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:57.261 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.602875+0000", 2024-04-19T01:04:57.261 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.4300000000000002e-07 2024-04-19T01:04:57.261 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.261 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.261 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:57.261 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603321+0000", 2024-04-19T01:04:57.261 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000446468 2024-04-19T01:04:57.261 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.261 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.261 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:57.261 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603332+0000", 2024-04-19T01:04:57.261 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1079e-05 2024-04-19T01:04:57.261 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.261 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.261 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:57.262 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603337+0000", 2024-04-19T01:04:57.262 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0760000000000002e-06 2024-04-19T01:04:57.262 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.262 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.262 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:57.263 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603389+0000", 2024-04-19T01:04:57.263 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.1990000000000002e-05 2024-04-19T01:04:57.263 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.264 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.264 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for ondisk", 2024-04-19T01:04:57.264 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603501+0000", 2024-04-19T01:04:57.264 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000111539 2024-04-19T01:04:57.264 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.264 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.264 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:57.264 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.658138+0000", 2024-04-19T01:04:57.264 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.054636905999999999 2024-04-19T01:04:57.264 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:57.264 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:57.264 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:57.264 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.264 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.265 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:628 3.5s0 3:ab059e23:::smithi15528762-38:head [append 0~606208] snapc 1c=[1c,1b,19,18,17,14,12] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e81)", 2024-04-19T01:04:57.265 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:51.603350+0000", 2024-04-19T01:04:57.265 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.4945321219999999, 2024-04-19T01:04:57.265 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.055106055000000001, 2024-04-19T01:04:57.265 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:57.265 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:57.265 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:57.265 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:57.265 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:57.266 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:57.266 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 628 2024-04-19T01:04:57.266 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.266 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:57.267 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.267 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:57.267 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603350+0000", 2024-04-19T01:04:57.267 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:57.267 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.267 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.267 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:57.267 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603350+0000", 2024-04-19T01:04:57.267 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:57.267 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.267 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.267 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:57.267 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603351+0000", 2024-04-19T01:04:57.267 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3499999999999996e-07 2024-04-19T01:04:57.267 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.268 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.268 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:57.268 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603847+0000", 2024-04-19T01:04:57.268 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00049669000000000004 2024-04-19T01:04:57.268 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.268 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.268 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:57.268 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603858+0000", 2024-04-19T01:04:57.268 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0729e-05 2024-04-19T01:04:57.268 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.268 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.268 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:57.268 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603863+0000", 2024-04-19T01:04:57.268 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0869999999999997e-06 2024-04-19T01:04:57.268 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.269 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.269 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:57.269 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603919+0000", 2024-04-19T01:04:57.269 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.5618000000000002e-05 2024-04-19T01:04:57.269 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.269 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.269 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:57.269 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.604028+0000", 2024-04-19T01:04:57.269 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000109187 2024-04-19T01:04:57.269 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.269 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.269 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:57.269 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.604655+0000", 2024-04-19T01:04:57.269 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00062722400000000001 2024-04-19T01:04:57.269 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.270 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.270 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:57.270 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.609598+0000", 2024-04-19T01:04:57.270 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.004943037 2024-04-19T01:04:57.270 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.270 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.270 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:57.270 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.658378+0000", 2024-04-19T01:04:57.270 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.048779623000000001 2024-04-19T01:04:57.270 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.270 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.270 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:57.270 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.658456+0000", 2024-04-19T01:04:57.270 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.8225000000000004e-05 2024-04-19T01:04:57.271 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:57.271 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:57.271 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:57.271 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.271 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.271 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:629 3.5s0 3:ab059e23:::smithi15528762-38:head [setxattr _header (57) in=7b] snapc 1c=[1c,1b,19,18,17,14,12] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e81)", 2024-04-19T01:04:57.271 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:51.603874+0000", 2024-04-19T01:04:57.271 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.4940080079999998, 2024-04-19T01:04:57.271 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.054746045, 2024-04-19T01:04:57.271 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:57.271 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:57.271 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:57.271 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:57.271 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:57.272 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:57.272 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 629 2024-04-19T01:04:57.272 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.272 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:57.272 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.272 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:57.272 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603874+0000", 2024-04-19T01:04:57.272 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:57.272 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.273 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.273 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:57.273 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603874+0000", 2024-04-19T01:04:57.273 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:57.273 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.273 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.273 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:57.273 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603875+0000", 2024-04-19T01:04:57.273 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8800000000000002e-07 2024-04-19T01:04:57.273 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.273 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.274 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:57.274 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603878+0000", 2024-04-19T01:04:57.274 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.6399999999999999e-06 2024-04-19T01:04:57.274 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.274 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.274 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:57.274 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603885+0000", 2024-04-19T01:04:57.274 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.1640000000000004e-06 2024-04-19T01:04:57.274 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.274 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.274 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:57.274 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603892+0000", 2024-04-19T01:04:57.274 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1469999999999998e-06 2024-04-19T01:04:57.275 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.275 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.275 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:57.275 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.608675+0000", 2024-04-19T01:04:57.275 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0047830379999999999 2024-04-19T01:04:57.275 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.275 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.275 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:57.275 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.608741+0000", 2024-04-19T01:04:57.275 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.6212999999999998e-05 2024-04-19T01:04:57.275 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.275 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.275 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:57.275 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.609027+0000", 2024-04-19T01:04:57.276 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00028588600000000001 2024-04-19T01:04:57.276 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.276 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.276 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:57.276 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.609773+0000", 2024-04-19T01:04:57.276 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000746563 2024-04-19T01:04:57.276 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.277 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.277 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:57.277 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.658557+0000", 2024-04-19T01:04:57.277 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.048783343999999999 2024-04-19T01:04:57.277 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.277 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.277 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:57.277 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.658620+0000", 2024-04-19T01:04:57.277 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3461999999999996e-05 2024-04-19T01:04:57.277 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:57.277 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:57.277 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:57.277 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.277 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.277 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:630 3.5s0 3:ab059e23:::smithi15528762-38:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e81)", 2024-04-19T01:04:57.278 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:51.603912+0000", 2024-04-19T01:04:57.278 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.4939705600000002, 2024-04-19T01:04:57.278 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.056857458, 2024-04-19T01:04:57.278 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:57.278 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:57.278 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:04:57.278 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:57.278 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:57.278 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:57.278 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 630 2024-04-19T01:04:57.278 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.278 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:57.278 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.278 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:57.279 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603912+0000", 2024-04-19T01:04:57.279 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:57.279 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.279 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.279 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:57.279 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603912+0000", 2024-04-19T01:04:57.279 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:57.279 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.279 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.279 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:57.279 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603912+0000", 2024-04-19T01:04:57.279 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6899999999999997e-07 2024-04-19T01:04:57.279 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.279 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.279 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:57.280 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603915+0000", 2024-04-19T01:04:57.280 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9730000000000002e-06 2024-04-19T01:04:57.280 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.280 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.280 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:57.280 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603923+0000", 2024-04-19T01:04:57.280 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1100000000000003e-06 2024-04-19T01:04:57.280 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.280 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.280 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:57.280 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603929+0000", 2024-04-19T01:04:57.281 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0170000000000002e-06 2024-04-19T01:04:57.281 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.281 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.281 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:57.281 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.609249+0000", 2024-04-19T01:04:57.281 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0053197380000000001 2024-04-19T01:04:57.281 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.281 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.281 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:04:57.281 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.609358+0000", 2024-04-19T01:04:57.281 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010952599999999999 2024-04-19T01:04:57.281 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.281 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.281 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:57.282 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.658676+0000", 2024-04-19T01:04:57.282 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.049317986000000001 2024-04-19T01:04:57.282 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.282 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.282 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:57.282 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.658789+0000", 2024-04-19T01:04:57.282 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011216400000000001 2024-04-19T01:04:57.282 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.282 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.282 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:57.282 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.660769+0000", 2024-04-19T01:04:57.282 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0019803749999999999 2024-04-19T01:04:57.282 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:57.282 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:57.282 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:57.283 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.283 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.283 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:631 3.5s0 3:abe8a6c4:::smithi15528762-18:head [copy-from ver 8] snapc 1c=[1c,1b,19,18,17,14,12] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e81)", 2024-04-19T01:04:57.283 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:51.603938+0000", 2024-04-19T01:04:57.283 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.4939440859999999, 2024-04-19T01:04:57.283 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.055423066, 2024-04-19T01:04:57.283 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:04:57.283 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:04:57.283 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:04:57.283 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:04:57.283 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:04:57.283 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:04:57.283 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 631 2024-04-19T01:04:57.283 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.284 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:04:57.284 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.284 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:04:57.284 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603938+0000", 2024-04-19T01:04:57.284 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:57.284 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.284 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.284 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:04:57.284 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603938+0000", 2024-04-19T01:04:57.284 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:04:57.284 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.284 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.284 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:04:57.284 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603938+0000", 2024-04-19T01:04:57.284 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7199999999999999e-07 2024-04-19T01:04:57.285 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.285 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.285 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:04:57.285 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603942+0000", 2024-04-19T01:04:57.285 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0809999999999998e-06 2024-04-19T01:04:57.285 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.285 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.285 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:04:57.285 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603953+0000", 2024-04-19T01:04:57.285 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1477e-05 2024-04-19T01:04:57.285 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.285 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.285 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:04:57.285 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603962+0000", 2024-04-19T01:04:57.285 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.5010000000000004e-06 2024-04-19T01:04:57.286 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.286 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.286 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:04:57.286 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.609402+0000", 2024-04-19T01:04:57.286 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0054401069999999996 2024-04-19T01:04:57.286 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.286 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.286 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:04:57.287 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.609457+0000", 2024-04-19T01:04:57.287 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.5016000000000003e-05 2024-04-19T01:04:57.287 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.287 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.287 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:04:57.287 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.617371+0000", 2024-04-19T01:04:57.287 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0079141149999999993 2024-04-19T01:04:57.287 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.287 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.287 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:04:57.287 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.630663+0000", 2024-04-19T01:04:57.287 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013291956000000001 2024-04-19T01:04:57.287 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.288 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.288 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:04:57.288 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.659260+0000", 2024-04-19T01:04:57.288 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.028597651000000002 2024-04-19T01:04:57.288 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:04:57.288 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:04:57.288 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:04:57.288 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.659361+0000", 2024-04-19T01:04:57.288 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010069 2024-04-19T01:04:57.288 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:57.288 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:57.289 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:57.289 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:04:57.289 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:04:57.289 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:04:57.290 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-04-19T01:04:57.291 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-04-19T01:04:57.291 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T01:04:57.291 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-04-19T01:04:57.292 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T01:04:57.292 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-04-19T01:04:57.292 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T01:04:57.292 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-04-19T01:04:57.293 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T01:04:57.293 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-04-19T01:04:57.293 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T01:04:57.293 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2024-04-19T01:04:57.294 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T01:04:57.294 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2024-04-19T01:04:57.294 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T01:04:57.295 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2024-04-19T01:04:57.295 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T01:04:57.295 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:04:57.288+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:04:57.305 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2024-04-19T01:04:57.306 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T01:04:57.328 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2024-04-19T01:04:57.328 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-19T01:04:57.353 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2024-04-19T01:04:57.353 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-19T01:04:57.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:04:57.354+0000 7f41ceffd640 0 -- 172.21.15.33:0/660822282 >> [v2:172.21.15.118:6800/506416463,v1:172.21.15.118:6801/506416463] conn(0x7f41d0221d90 msgr2=0x7f41d0224180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-19T01:04:57.391 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:04:57.391+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:04:57.446 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:57.446 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:57.461 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-04-19T01:04:57.493 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:04:57.493+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:04:57.560 INFO:tasks.rados.rados.0.smithi155.stdout:274: done (3 left) 2024-04-19T01:04:57.560 INFO:tasks.rados.rados.0.smithi155.stdout:275: read oid 34 snap -1 2024-04-19T01:04:57.560 INFO:tasks.rados.rados.0.smithi155.stdout:275: expect (ObjNum 66 snap 6 seq_num 66) 2024-04-19T01:04:57.560 INFO:tasks.rados.rados.0.smithi155.stdout:276: snap_remove snap 27 2024-04-19T01:04:57.593 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2024-04-19T01:04:57.593 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-19T01:04:57.594 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:04:57.594+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:04:57.626 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2024-04-19T01:04:57.626 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-19T01:04:57.648 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:57.648 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:57.662 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-04-19T01:04:57.694 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:04:57.692+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:04:57.795 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:04:57.795+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:04:57.832 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:57.832 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:57.845 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-19T01:04:57.896 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:04:57.895+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:04:57.979 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:57.979 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:57.993 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-19T01:04:58.001 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:04:57.998+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:04:58.101 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:04:58.097+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:04:58.117 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:58.117 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:58.132 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-19T01:04:58.202 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:04:58.202+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:04:58.302 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:58.302 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:58.303 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:04:58.302+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:04:58.317 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-19T01:04:58.404 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:58.401+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:58.440 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:58.440 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:58.454 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-19T01:04:58.454 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-04-19T01:04:58.454 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-19T01:04:58.455 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2024-04-19T01:04:58.455 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T01:04:58.543 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:04:58.541+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:04:58.567 INFO:tasks.rados.rados.0.smithi155.stdout:276: done (4 left) 2024-04-19T01:04:58.568 INFO:tasks.rados.rados.0.smithi155.stdout:277: delete oid 33 current snap is 28 2024-04-19T01:04:58.568 INFO:tasks.rados.rados.0.smithi155.stdout:271: finishing copy_from racing read to smithi15528762-27 2024-04-19T01:04:58.568 INFO:tasks.rados.rados.0.smithi155.stdout:271: finishing copy_from to smithi15528762-27 2024-04-19T01:04:58.568 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 27 v 38 (ObjNum 76 snap 8 seq_num 76) dirty exists 2024-04-19T01:04:58.568 INFO:tasks.rados.rados.0.smithi155.stdout:272: finishing write tid 1 to smithi15528762-10 2024-04-19T01:04:58.568 INFO:tasks.rados.rados.0.smithi155.stdout:272: finishing write tid 2 to smithi15528762-10 2024-04-19T01:04:58.568 INFO:tasks.rados.rados.0.smithi155.stdout:272: finishing write tid 3 to smithi15528762-10 2024-04-19T01:04:58.568 INFO:tasks.rados.rados.0.smithi155.stdout:272: finishing write tid 4 to smithi15528762-10 2024-04-19T01:04:58.568 INFO:tasks.rados.rados.0.smithi155.stdout:272: finishing write tid 5 to smithi15528762-10 2024-04-19T01:04:58.568 INFO:tasks.rados.rados.0.smithi155.stdout:272: oid 10 updating version 0 to 35 2024-04-19T01:04:58.568 INFO:tasks.rados.rados.0.smithi155.stdout:272: oid 10 updating version 35 to 37 2024-04-19T01:04:58.568 INFO:tasks.rados.rados.0.smithi155.stdout:272: oid 10 updating version 37 to 38 2024-04-19T01:04:58.568 INFO:tasks.rados.rados.0.smithi155.stdout:272: oid 10 version 38 is already newer than 36 2024-04-19T01:04:58.569 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 10 v 38 (ObjNum 115 snap 28 seq_num 115) dirty exists 2024-04-19T01:04:58.569 INFO:tasks.rados.rados.0.smithi155.stdout:272: left oid 10 (ObjNum 115 snap 28 seq_num 115) 2024-04-19T01:04:58.569 INFO:tasks.rados.rados.0.smithi155.stdout:273: finishing write tid 1 to smithi15528762-6 2024-04-19T01:04:58.569 INFO:tasks.rados.rados.0.smithi155.stdout:273: finishing write tid 2 to smithi15528762-6 2024-04-19T01:04:58.569 INFO:tasks.rados.rados.0.smithi155.stdout:273: finishing write tid 3 to smithi15528762-6 2024-04-19T01:04:58.569 INFO:tasks.rados.rados.0.smithi155.stdout:273: finishing write tid 4 to smithi15528762-6 2024-04-19T01:04:58.569 INFO:tasks.rados.rados.0.smithi155.stdout:273: finishing write tid 5 to smithi15528762-6 2024-04-19T01:04:58.569 INFO:tasks.rados.rados.0.smithi155.stdout:273: finishing write tid 6 to smithi15528762-6 2024-04-19T01:04:58.569 INFO:tasks.rados.rados.0.smithi155.stdout:273: oid 6 updating version 0 to 57 2024-04-19T01:04:58.569 INFO:tasks.rados.rados.0.smithi155.stdout:273: oid 6 updating version 57 to 58 2024-04-19T01:04:58.569 INFO:tasks.rados.rados.0.smithi155.stdout:273: oid 6 updating version 58 to 61 2024-04-19T01:04:58.569 INFO:tasks.rados.rados.0.smithi155.stdout:273: oid 6 version 61 is already newer than 60 2024-04-19T01:04:58.569 INFO:tasks.rados.rados.0.smithi155.stdout:273: oid 6 version 61 is already newer than 59 2024-04-19T01:04:58.569 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 6 v 61 (ObjNum 116 snap 28 seq_num 116) dirty exists 2024-04-19T01:04:58.570 INFO:tasks.rados.rados.0.smithi155.stdout:273: left oid 6 (ObjNum 116 snap 28 seq_num 116) 2024-04-19T01:04:58.570 INFO:tasks.rados.rados.0.smithi155.stdout:275: expect (ObjNum 66 snap 6 seq_num 66) 2024-04-19T01:04:58.604 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-19T01:04:58.606 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:04:58.604+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:04:58.628 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:58.628 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:58.642 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-19T01:04:58.713 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:04:58.708+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:04:58.779 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:04:58.780 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-04-19T01:04:58.789 INFO:tasks.rados.rados.0.smithi155.stdout:271: done (4 left) 2024-04-19T01:04:58.789 INFO:tasks.rados.rados.0.smithi155.stdout:272: done (3 left) 2024-04-19T01:04:58.789 INFO:tasks.rados.rados.0.smithi155.stdout:273: done (2 left) 2024-04-19T01:04:58.789 INFO:tasks.rados.rados.0.smithi155.stdout:275: done (1 left) 2024-04-19T01:04:58.789 INFO:tasks.rados.rados.0.smithi155.stdout:277: done (0 left) 2024-04-19T01:04:58.789 INFO:tasks.rados.rados.0.smithi155.stdout:278: setattr oid 33 current snap is 28 2024-04-19T01:04:58.789 INFO:tasks.rados.rados.0.smithi155.stdout:279: snap_remove snap 18 2024-04-19T01:04:58.798 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":108,"stamp":"2024-04-19T01:04:58.564488+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":75973152,"num_objects":78,"num_object_clones":29,"num_object_copies":536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":7,"num_read":453,"num_read_kb":73681,"num_write":793,"num_write_kb":96484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1065,"ondisk_log_size":1065,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":621768,"kb_used_data":143644,"kb_used_omap":24,"kb_used_meta":477863,"kb_avail":1509327672,"statfs":{"total":1546188226560,"available":1545551536128,"internally_reserved":0,"allocated":147091456,"data_stored":170843320,"data_compressed":1954379,"data_compressed_allocated":4063232,"data_compressed_original":9019392,"omap_allocated":25520,"internal_metadata":489331792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":43,"apply_latency_ms":43,"commit_latency_ns":43000000,"apply_latency_ns":43000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4685824,"num_objects":3,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":532,"num_write":20,"num_write_kb":8448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.033486"},"pg_stats":[{"pgid":"3.f","version":"86'55","reported_seq":331,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-19T01:04:56.539643+0000","last_change":"2024-04-19T01:04:56.540550+0000","last_active":"2024-04-19T01:04:56.539643+0000","last_peered":"2024-04-19T01:04:56.539643+0000","last_clean":"2024-04-19T01:04:56.539643+0000","last_became_active":"2024-04-19T01:04:03.238651+0000","last_became_peered":"2024-04-19T01:04:03.238651+0000","last_unstale":"2024-04-19T01:04:56.539643+0000","last_undegraded":"2024-04-19T01:04:56.539643+0000","last_fullsized":"2024-04-19T01:04:56.539643+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:31.299733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016710958000000001,"stat_sum":{"num_bytes":4472832,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":5288,"num_write":41,"num_write_kb":4368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,5,11,7,1,2],"acting":[0,13,5,11,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"4"}]},{"pgid":"3.e","version":"86'39","reported_seq":320,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-19T01:04:56.523634+0000","last_change":"2024-04-19T01:04:56.523634+0000","last_active":"2024-04-19T01:04:56.523634+0000","last_peered":"2024-04-19T01:04:56.523634+0000","last_clean":"2024-04-19T01:04:56.523634+0000","last_became_active":"2024-04-19T01:04:03.238386+0000","last_became_peered":"2024-04-19T01:04:03.238386+0000","last_unstale":"2024-04-19T01:04:56.523634+0000","last_undegraded":"2024-04-19T01:04:56.523634+0000","last_fullsized":"2024-04-19T01:04:56.523634+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:09.448497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025222180000000001,"stat_sum":{"num_bytes":2736128,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":8695,"num_write":15,"num_write_kb":4304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11,5,4,2,9,15],"acting":[13,11,5,4,2,9,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"12","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"4"}]},{"pgid":"3.d","version":"84'70","reported_seq":325,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-19T01:04:54.560331+0000","last_change":"2024-04-19T01:04:54.561945+0000","last_active":"2024-04-19T01:04:54.560331+0000","last_peered":"2024-04-19T01:04:54.560331+0000","last_clean":"2024-04-19T01:04:54.560331+0000","last_became_active":"2024-04-19T01:04:03.025363+0000","last_became_peered":"2024-04-19T01:04:03.025363+0000","last_unstale":"2024-04-19T01:04:54.560331+0000","last_undegraded":"2024-04-19T01:04:54.560331+0000","last_fullsized":"2024-04-19T01:04:54.560331+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:30.129678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016839711,"stat_sum":{"num_bytes":4751360,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":4327,"num_write":25,"num_write_kb":5728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,9,0,8,2,6],"acting":[12,4,9,0,8,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"12","length":"2"},{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"3.c","version":"86'57","reported_seq":325,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-19T01:04:56.554147+0000","last_change":"2024-04-19T01:04:56.554701+0000","last_active":"2024-04-19T01:04:56.554147+0000","last_peered":"2024-04-19T01:04:56.554147+0000","last_clean":"2024-04-19T01:04:56.554147+0000","last_became_active":"2024-04-19T01:04:03.025808+0000","last_became_peered":"2024-04-19T01:04:03.025808+0000","last_unstale":"2024-04-19T01:04:56.554147+0000","last_undegraded":"2024-04-19T01:04:56.554147+0000","last_fullsized":"2024-04-19T01:04:56.554147+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:28.094433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.031237742999999998,"stat_sum":{"num_bytes":1425408,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":14,"num_read_kb":2453,"num_write":36,"num_write_kb":1904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,12,1,9,2,6],"acting":[0,10,12,1,9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"4"}]},{"pgid":"3.b","version":"84'44","reported_seq":333,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-19T01:04:57.562785+0000","last_change":"2024-04-19T01:04:57.563672+0000","last_active":"2024-04-19T01:04:57.562785+0000","last_peered":"2024-04-19T01:04:57.562785+0000","last_clean":"2024-04-19T01:04:57.562785+0000","last_became_active":"2024-04-19T01:04:02.781569+0000","last_became_peered":"2024-04-19T01:04:02.781569+0000","last_unstale":"2024-04-19T01:04:57.562785+0000","last_undegraded":"2024-04-19T01:04:57.562785+0000","last_fullsized":"2024-04-19T01:04:57.562785+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:29.373040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001423964,"stat_sum":{"num_bytes":5095424,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":6897,"num_write":37,"num_write_kb":4976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11,15,1,4,14],"acting":[10,6,11,15,1,4,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"12","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"5"}]},{"pgid":"3.a","version":"84'74","reported_seq":340,"reported_epoch":84,"state":"active+clean+snaptrim","last_fresh":"2024-04-19T01:04:54.562364+0000","last_change":"2024-04-19T01:04:54.546686+0000","last_active":"2024-04-19T01:04:54.562364+0000","last_peered":"2024-04-19T01:04:54.562364+0000","last_clean":"2024-04-19T01:04:54.562364+0000","last_became_active":"2024-04-19T01:04:03.323177+0000","last_became_peered":"2024-04-19T01:04:03.323177+0000","last_unstale":"2024-04-19T01:04:54.562364+0000","last_undegraded":"2024-04-19T01:04:54.562364+0000","last_fullsized":"2024-04-19T01:04:54.562364+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:10.951748+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.018168846999999998,"stat_sum":{"num_bytes":8847360,"num_objects":8,"num_object_clones":4,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":2,"num_read":30,"num_read_kb":5819,"num_write":27,"num_write_kb":11104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,7,3,13,1,8],"acting":[12,4,7,3,13,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"3.9","version":"85'120","reported_seq":396,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-19T01:04:56.522512+0000","last_change":"2024-04-19T01:04:56.523292+0000","last_active":"2024-04-19T01:04:56.522512+0000","last_peered":"2024-04-19T01:04:56.522512+0000","last_clean":"2024-04-19T01:04:56.522512+0000","last_became_active":"2024-04-19T01:04:03.027372+0000","last_became_peered":"2024-04-19T01:04:03.027372+0000","last_unstale":"2024-04-19T01:04:56.522512+0000","last_undegraded":"2024-04-19T01:04:56.522512+0000","last_fullsized":"2024-04-19T01:04:56.522512+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:27.408813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002412394,"stat_sum":{"num_bytes":7667712,"num_objects":7,"num_object_clones":1,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":32,"num_read_kb":1071,"num_write":83,"num_write_kb":11712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11,14,9,4,10,6],"acting":[13,11,14,9,4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"12","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"4"}]},{"pgid":"3.8","version":"84'59","reported_seq":318,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-19T01:04:54.562458+0000","last_change":"2024-04-19T01:04:54.564029+0000","last_active":"2024-04-19T01:04:54.562458+0000","last_peered":"2024-04-19T01:04:54.562458+0000","last_clean":"2024-04-19T01:04:54.562458+0000","last_became_active":"2024-04-19T01:04:03.237653+0000","last_became_peered":"2024-04-19T01:04:03.237653+0000","last_unstale":"2024-04-19T01:04:54.562458+0000","last_undegraded":"2024-04-19T01:04:54.562458+0000","last_fullsized":"2024-04-19T01:04:54.562458+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:10.330265+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.023614693999999999,"stat_sum":{"num_bytes":3112960,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":1647,"num_write":41,"num_write_kb":5664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,1,15,14,0,5],"acting":[11,10,1,15,14,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"12","length":"2"},{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"1.0","version":"25'157","reported_seq":326,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-19T01:04:54.540897+0000","last_change":"2024-04-19T01:03:50.776344+0000","last_active":"2024-04-19T01:04:54.540897+0000","last_peered":"2024-04-19T01:04:54.540897+0000","last_clean":"2024-04-19T01:04:54.540897+0000","last_became_active":"2024-04-19T01:03:50.774078+0000","last_became_peered":"2024-04-19T01:03:50.774078+0000","last_unstale":"2024-04-19T01:04:54.540897+0000","last_undegraded":"2024-04-19T01:04:54.540897+0000","last_fullsized":"2024-04-19T01:04:54.540897+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:49.580851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:49.580851+0000","last_clean_scrub_stamp":"2024-04-19T01:03:49.580851+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:07.495976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":124,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-19T01:04:54.540840+0000","last_change":"2024-04-19T01:03:53.766587+0000","last_active":"2024-04-19T01:04:54.540840+0000","last_peered":"2024-04-19T01:04:54.540840+0000","last_clean":"2024-04-19T01:04:54.540840+0000","last_became_active":"2024-04-19T01:03:53.765967+0000","last_became_peered":"2024-04-19T01:03:53.765967+0000","last_unstale":"2024-04-19T01:04:54.540840+0000","last_undegraded":"2024-04-19T01:04:54.540840+0000","last_fullsized":"2024-04-19T01:04:54.540840+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:18.833542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"84'29","reported_seq":289,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-19T01:04:54.552261+0000","last_change":"2024-04-19T01:04:54.556685+0000","last_active":"2024-04-19T01:04:54.552261+0000","last_peered":"2024-04-19T01:04:54.552261+0000","last_clean":"2024-04-19T01:04:54.552261+0000","last_became_active":"2024-04-19T01:04:03.322437+0000","last_became_peered":"2024-04-19T01:04:03.322437+0000","last_unstale":"2024-04-19T01:04:54.552261+0000","last_undegraded":"2024-04-19T01:04:54.552261+0000","last_fullsized":"2024-04-19T01:04:54.552261+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:04.682761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016499702000000002,"stat_sum":{"num_bytes":671744,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":10,"num_read_kb":531,"num_write":8,"num_write_kb":656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0,4,8,3,7],"acting":[11,5,0,4,8,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"12","length":"2"},{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":124,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-19T01:04:54.543640+0000","last_change":"2024-04-19T01:03:53.760075+0000","last_active":"2024-04-19T01:04:54.543640+0000","last_peered":"2024-04-19T01:04:54.543640+0000","last_clean":"2024-04-19T01:04:54.543640+0000","last_became_active":"2024-04-19T01:03:53.759326+0000","last_became_peered":"2024-04-19T01:03:53.759326+0000","last_unstale":"2024-04-19T01:04:54.543640+0000","last_undegraded":"2024-04-19T01:04:54.543640+0000","last_fullsized":"2024-04-19T01:04:54.543640+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:02.718012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"82'28","reported_seq":282,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-19T01:04:53.540458+0000","last_change":"2024-04-19T01:04:52.546302+0000","last_active":"2024-04-19T01:04:53.540458+0000","last_peered":"2024-04-19T01:04:53.540458+0000","last_clean":"2024-04-19T01:04:53.540458+0000","last_became_active":"2024-04-19T01:04:03.323727+0000","last_became_peered":"2024-04-19T01:04:03.323727+0000","last_unstale":"2024-04-19T01:04:53.540458+0000","last_undegraded":"2024-04-19T01:04:53.540458+0000","last_fullsized":"2024-04-19T01:04:53.540458+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:30.563086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015941371999999999,"stat_sum":{"num_bytes":2195456,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":12,"num_read_kb":2020,"num_write":50,"num_write_kb":2272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,11,9,1,7],"acting":[6,4,3,11,9,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":123,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-19T01:04:54.539531+0000","last_change":"2024-04-19T01:03:53.650434+0000","last_active":"2024-04-19T01:04:54.539531+0000","last_peered":"2024-04-19T01:04:54.539531+0000","last_clean":"2024-04-19T01:04:54.539531+0000","last_became_active":"2024-04-19T01:03:53.649606+0000","last_became_peered":"2024-04-19T01:03:53.649606+0000","last_unstale":"2024-04-19T01:04:54.539531+0000","last_undegraded":"2024-04-19T01:04:54.539531+0000","last_fullsized":"2024-04-19T01:04:54.539531+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:17.424729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"84'71","reported_seq":329,"reported_epoch":84,"state":"active+clean+snaptrim","last_fresh":"2024-04-19T01:04:54.574003+0000","last_change":"2024-04-19T01:04:54.544525+0000","last_active":"2024-04-19T01:04:54.574003+0000","last_peered":"2024-04-19T01:04:54.574003+0000","last_clean":"2024-04-19T01:04:54.574003+0000","last_became_active":"2024-04-19T01:04:03.322503+0000","last_became_peered":"2024-04-19T01:04:03.322503+0000","last_unstale":"2024-04-19T01:04:54.574003+0000","last_undegraded":"2024-04-19T01:04:54.574003+0000","last_fullsized":"2024-04-19T01:04:54.574003+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:20.353967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.019852951000000001,"stat_sum":{"num_bytes":5029888,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":10472,"num_write":26,"num_write_kb":6896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4,3,10,11,6],"acting":[1,9,4,3,10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":124,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-19T01:04:54.545156+0000","last_change":"2024-04-19T01:03:54.012138+0000","last_active":"2024-04-19T01:04:54.545156+0000","last_peered":"2024-04-19T01:04:54.545156+0000","last_clean":"2024-04-19T01:04:54.545156+0000","last_became_active":"2024-04-19T01:03:54.011460+0000","last_became_peered":"2024-04-19T01:03:54.011460+0000","last_unstale":"2024-04-19T01:04:54.545156+0000","last_undegraded":"2024-04-19T01:04:54.545156+0000","last_fullsized":"2024-04-19T01:04:54.545156+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:04:58.336612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"84'58","reported_seq":319,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-19T01:04:54.558337+0000","last_change":"2024-04-19T01:04:54.558823+0000","last_active":"2024-04-19T01:04:54.558337+0000","last_peered":"2024-04-19T01:04:54.558337+0000","last_clean":"2024-04-19T01:04:54.558337+0000","last_became_active":"2024-04-19T01:04:02.835716+0000","last_became_peered":"2024-04-19T01:04:02.835716+0000","last_unstale":"2024-04-19T01:04:54.558337+0000","last_undegraded":"2024-04-19T01:04:54.558337+0000","last_fullsized":"2024-04-19T01:04:54.558337+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:09.736755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018519570999999999,"stat_sum":{"num_bytes":2932736,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":1880,"num_write":79,"num_write_kb":5568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2,0,1,4,12],"acting":[15,8,2,0,1,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"12","length":"2"},{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"3.5","version":"84'113","reported_seq":391,"reported_epoch":84,"state":"active+clean+snaptrim","last_fresh":"2024-04-19T01:04:54.562982+0000","last_change":"2024-04-19T01:04:54.544312+0000","last_active":"2024-04-19T01:04:54.562982+0000","last_peered":"2024-04-19T01:04:54.562982+0000","last_clean":"2024-04-19T01:04:54.562982+0000","last_became_active":"2024-04-19T01:04:03.237886+0000","last_became_peered":"2024-04-19T01:04:03.237886+0000","last_unstale":"2024-04-19T01:04:54.562982+0000","last_undegraded":"2024-04-19T01:04:54.562982+0000","last_fullsized":"2024-04-19T01:04:54.562982+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:07.015795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.025599787999999998,"stat_sum":{"num_bytes":15958016,"num_objects":15,"num_object_clones":8,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":1,"num_read":58,"num_read_kb":11840,"num_write":90,"num_write_kb":17296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,14,11,10,2,5],"acting":[8,6,14,11,10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":122,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-19T01:04:53.540844+0000","last_change":"2024-04-19T01:03:53.716606+0000","last_active":"2024-04-19T01:04:53.540844+0000","last_peered":"2024-04-19T01:04:53.540844+0000","last_clean":"2024-04-19T01:04:53.540844+0000","last_became_active":"2024-04-19T01:03:53.716040+0000","last_became_peered":"2024-04-19T01:03:53.716040+0000","last_unstale":"2024-04-19T01:04:53.540844+0000","last_undegraded":"2024-04-19T01:04:53.540844+0000","last_fullsized":"2024-04-19T01:04:53.540844+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:01.975515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"87'70","reported_seq":355,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-19T01:04:57.581568+0000","last_change":"2024-04-19T01:04:57.583215+0000","last_active":"2024-04-19T01:04:57.581568+0000","last_peered":"2024-04-19T01:04:57.581568+0000","last_clean":"2024-04-19T01:04:57.581568+0000","last_became_active":"2024-04-19T01:04:03.322815+0000","last_became_peered":"2024-04-19T01:04:03.322815+0000","last_unstale":"2024-04-19T01:04:57.581568+0000","last_undegraded":"2024-04-19T01:04:57.581568+0000","last_fullsized":"2024-04-19T01:04:57.581568+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:05.197731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.021002982999999999,"stat_sum":{"num_bytes":7405568,"num_objects":9,"num_object_clones":5,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":2,"num_read":30,"num_read_kb":1595,"num_write":23,"num_write_kb":7568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,13,14,3,15,1,5],"acting":[10,13,14,3,15,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"12","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"5"}]},{"pgid":"2.5","version":"0'0","reported_seq":127,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-19T01:04:56.525635+0000","last_change":"2024-04-19T01:03:53.772709+0000","last_active":"2024-04-19T01:04:56.525635+0000","last_peered":"2024-04-19T01:04:56.525635+0000","last_clean":"2024-04-19T01:04:56.525635+0000","last_became_active":"2024-04-19T01:03:53.771887+0000","last_became_peered":"2024-04-19T01:03:53.771887+0000","last_unstale":"2024-04-19T01:04:56.525635+0000","last_undegraded":"2024-04-19T01:04:56.525635+0000","last_fullsized":"2024-04-19T01:04:56.525635+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:17.479766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"67'21","reported_seq":297,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-19T01:04:56.539677+0000","last_change":"2024-04-19T01:04:56.543369+0000","last_active":"2024-04-19T01:04:56.539677+0000","last_peered":"2024-04-19T01:04:56.539677+0000","last_clean":"2024-04-19T01:04:56.539677+0000","last_became_active":"2024-04-19T01:04:03.238517+0000","last_became_peered":"2024-04-19T01:04:03.238517+0000","last_unstale":"2024-04-19T01:04:56.539677+0000","last_undegraded":"2024-04-19T01:04:56.539677+0000","last_fullsized":"2024-04-19T01:04:56.539677+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:23.032818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046291630000000004,"stat_sum":{"num_bytes":3080192,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":9028,"num_write":9,"num_write_kb":3008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,7,5,11,6,0],"acting":[14,13,7,5,11,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"12","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"4"}]},{"pgid":"2.6","version":"0'0","reported_seq":128,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-19T01:04:56.537684+0000","last_change":"2024-04-19T01:03:53.736290+0000","last_active":"2024-04-19T01:04:56.537684+0000","last_peered":"2024-04-19T01:04:56.537684+0000","last_clean":"2024-04-19T01:04:56.537684+0000","last_became_active":"2024-04-19T01:03:53.735549+0000","last_became_peered":"2024-04-19T01:03:53.735549+0000","last_unstale":"2024-04-19T01:04:56.537684+0000","last_undegraded":"2024-04-19T01:04:56.537684+0000","last_fullsized":"2024-04-19T01:04:56.537684+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:07.640918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13],"acting":[14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":272,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-19T01:04:56.537437+0000","last_change":"2024-04-19T01:04:56.539591+0000","last_active":"2024-04-19T01:04:56.537437+0000","last_peered":"2024-04-19T01:04:56.537437+0000","last_clean":"2024-04-19T01:04:56.537437+0000","last_became_active":"2024-04-19T01:04:02.889974+0000","last_became_peered":"2024-04-19T01:04:02.889974+0000","last_unstale":"2024-04-19T01:04:56.537437+0000","last_undegraded":"2024-04-19T01:04:56.537437+0000","last_fullsized":"2024-04-19T01:04:56.537437+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:11.776533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030190099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7,15,8,11,0,1],"acting":[14,7,15,8,11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"12","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"4"}]},{"pgid":"2.7","version":"0'0","reported_seq":124,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-19T01:04:54.544279+0000","last_change":"2024-04-19T01:03:53.939441+0000","last_active":"2024-04-19T01:04:54.544279+0000","last_peered":"2024-04-19T01:04:54.544279+0000","last_clean":"2024-04-19T01:04:54.544279+0000","last_became_active":"2024-04-19T01:03:53.938774+0000","last_became_peered":"2024-04-19T01:03:53.938774+0000","last_unstale":"2024-04-19T01:04:54.544279+0000","last_undegraded":"2024-04-19T01:04:54.544279+0000","last_fullsized":"2024-04-19T01:04:54.544279+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:14.081709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":75382784,"num_objects":76,"num_object_clones":29,"num_object_copies":532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":7,"num_read":317,"num_read_kb":73563,"num_write":590,"num_write_kb":93024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":129900544,"data_stored":155193344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":908,"ondisk_log_size":908,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":13},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":15546,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":14,"up_from":24,"seq":103079215121,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46632,"kb_used_data":11288,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94325208,"statfs":{"total":96636764160,"available":96589012992,"internally_reserved":0,"allocated":11558912,"data_stored":12935802,"data_compressed":125208,"data_compressed_allocated":258048,"data_compressed_original":516096,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,5,6,7,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":13,"up_from":24,"seq":103079215122,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44276,"kb_used_data":8936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94327564,"statfs":{"total":96636764160,"available":96591425536,"internally_reserved":0,"allocated":9150464,"data_stored":9816553,"data_compressed":123194,"data_compressed_allocated":253952,"data_compressed_original":507904,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[2,4,5,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":24,"seq":103079215123,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41736,"kb_used_data":6396,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94330104,"statfs":{"total":96636764160,"available":96594026496,"internally_reserved":0,"allocated":6549504,"data_stored":7293417,"data_compressed":123194,"data_compressed_allocated":253952,"data_compressed_original":507904,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[1,2,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313234,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39152,"kb_used_data":12124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94332688,"statfs":{"total":96636764160,"available":96596672512,"internally_reserved":0,"allocated":12414976,"data_stored":14605011,"data_compressed":127226,"data_compressed_allocated":262144,"data_compressed_original":524288,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,3,4,5,6,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280527,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38980,"kb_used_data":11956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94332860,"statfs":{"total":96636764160,"available":96596848640,"internally_reserved":0,"allocated":12242944,"data_stored":15059223,"data_compressed":119160,"data_compressed_allocated":245760,"data_compressed_original":491520,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280529,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38308,"kb_used_data":11284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94333532,"statfs":{"total":96636764160,"available":96597536768,"internally_reserved":0,"allocated":11554816,"data_stored":13535511,"data_compressed":119160,"data_compressed_allocated":245760,"data_compressed_original":491520,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280528,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33840,"kb_used_data":6816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338000,"statfs":{"total":96636764160,"available":96602112000,"internally_reserved":0,"allocated":6979584,"data_stored":7858455,"data_compressed":119160,"data_compressed_allocated":245760,"data_compressed_original":491520,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[2,4,5,6,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280528,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37440,"kb_used_data":10416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334400,"statfs":{"total":96636764160,"available":96598425600,"internally_reserved":0,"allocated":10665984,"data_stored":12093719,"data_compressed":119160,"data_compressed_allocated":245760,"data_compressed_original":491520,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280528,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37356,"kb_used_data":10332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334484,"statfs":{"total":96636764160,"available":96598511616,"internally_reserved":0,"allocated":10579968,"data_stored":13241143,"data_compressed":126933,"data_compressed_allocated":278528,"data_compressed_original":1003520,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,6,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280529,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36040,"kb_used_data":9016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335800,"statfs":{"total":96636764160,"available":96599859200,"internally_reserved":0,"allocated":9232384,"data_stored":10991895,"data_compressed":119160,"data_compressed_allocated":245760,"data_compressed_original":491520,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[2,5,6,7,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280527,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44588,"kb_used_data":13404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327252,"statfs":{"total":96636764160,"available":96591106048,"internally_reserved":0,"allocated":13725696,"data_stored":17677111,"data_compressed":126933,"data_compressed_allocated":278528,"data_compressed_original":1003520,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,3,4,5,6,7,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280527,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32312,"kb_used_data":5292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339528,"statfs":{"total":96636764160,"available":96603676672,"internally_reserved":0,"allocated":5419008,"data_stored":5480678,"data_compressed":117149,"data_compressed_allocated":241664,"data_compressed_original":483328,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280528,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33360,"kb_used_data":6336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338480,"statfs":{"total":96636764160,"available":96602603520,"internally_reserved":0,"allocated":6488064,"data_stored":6740247,"data_compressed":119160,"data_compressed_allocated":245760,"data_compressed_original":491520,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,4,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":24,"seq":103079215121,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43804,"kb_used_data":8464,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94328036,"statfs":{"total":96636764160,"available":96591908864,"internally_reserved":0,"allocated":8667136,"data_stored":10770921,"data_compressed":123194,"data_compressed_allocated":253952,"data_compressed_original":507904,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215122,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37116,"kb_used_data":5936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334724,"statfs":{"total":96636764160,"available":96598757376,"internally_reserved":0,"allocated":6078464,"data_stored":6556137,"data_compressed":123194,"data_compressed_allocated":253952,"data_compressed_original":507904,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":24,"seq":103079215121,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36828,"kb_used_data":5648,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335012,"statfs":{"total":96636764160,"available":96599052288,"internally_reserved":0,"allocated":5783552,"data_stored":6187497,"data_compressed":123194,"data_compressed_allocated":253952,"data_compressed_original":507904,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5111808,"data_stored":5853184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10129408,"data_stored":12177408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7700480,"data_stored":9453568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":6037504,"data_stored":6983680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":9613312,"data_stored":11055104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8990720,"data_stored":11362304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":11300864,"data_stored":14376960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":5009408,"data_stored":5758976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8290304,"data_stored":10117120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":5582848,"data_stored":6590464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":11423744,"data_stored":13881344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12136448,"data_stored":15798272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4489216,"data_stored":4616192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":8552448,"data_stored":9482240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":10579968,"data_stored":12222464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":4952064,"data_stored":5464064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T01:04:58.800 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pgp_num 16 2024-04-19T01:04:58.809 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:04:58.807+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:04:58.810 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:58.810 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:58.824 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-19T01:04:58.910 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:04:58.908+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:04:58.947 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:58.947 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:58.961 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-19T01:04:59.012 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:04:59.010+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:04:59.082 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:59.082 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:59.096 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-19T01:04:59.116 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:04:59.115+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:04:59.213 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:59.213 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:59.216 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:04:59.213+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:04:59.227 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-04-19T01:04:59.317 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:04:59.313+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:04:59.389 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:59.389 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:59.404 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-04-19T01:04:59.418 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:04:59.416+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:04:59.518 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:04:59.520+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:04:59.569 INFO:teuthology.orchestra.run.smithi033.stderr:set pool 3 pgp_num to 16 2024-04-19T01:04:59.572 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:59.572 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:59.576 INFO:tasks.rados.rados.0.smithi155.stdout:279: done (1 left) 2024-04-19T01:04:59.577 INFO:tasks.rados.rados.0.smithi155.stdout:280: read oid 17 snap -1 2024-04-19T01:04:59.577 INFO:tasks.rados.rados.0.smithi155.stdout:280: expect (ObjNum 108 snap 25 seq_num 108) 2024-04-19T01:04:59.577 INFO:tasks.rados.rados.0.smithi155.stdout:281: delete oid 15 current snap is 28 2024-04-19T01:04:59.577 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 33 v 61 (ObjNum 129 snap 0 seq_num 1053499032) dirty exists 2024-04-19T01:04:59.580 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2024-04-19T01:04:59.596 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-04-19T01:04:59.607 INFO:tasks.rados.rados.0.smithi155.stdout:278: done (2 left) 2024-04-19T01:04:59.607 INFO:tasks.rados.rados.0.smithi155.stdout:281: done (1 left) 2024-04-19T01:04:59.607 INFO:tasks.rados.rados.0.smithi155.stdout:282: snap_create 2024-04-19T01:04:59.617 INFO:tasks.rados.rados.0.smithi155.stdout:280: expect (ObjNum 108 snap 25 seq_num 108) 2024-04-19T01:04:59.624 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:04:59.618+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:04:59.642 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-19T01:04:59.721 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:04:59.723+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:04:59.723 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:04:59.723 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:04:59.737 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-19T01:04:59.810 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-04-19T01:04:59.811 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T01:04:59.825 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:04:59.822+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:04:59.830 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-04-19T01:04:59.831 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T01:04:59.850 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-04-19T01:04:59.851 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T01:04:59.870 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:04:59.870 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-04-19T01:04:59.870 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-04-19T01:04:59.870 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:04:59.874 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-04-19T01:04:59.875 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T01:04:59.886 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-19T01:04:59.890 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-04-19T01:04:59.891 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T01:04:59.913 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-04-19T01:04:59.914 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T01:04:59.927 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:04:59.928+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:04:59.930 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-04-19T01:04:59.931 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T01:04:59.952 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-04-19T01:04:59.952 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T01:04:59.974 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2024-04-19T01:04:59.974 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T01:04:59.997 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2024-04-19T01:04:59.998 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T01:05:00.020 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2024-04-19T01:05:00.021 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T01:05:00.021 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:05:00.021 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-04-19T01:05:00.021 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-04-19T01:05:00.021 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-04-19T01:05:00.021 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:05:00.028 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:05:00.028+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:05:00.037 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-19T01:05:00.044 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2024-04-19T01:05:00.044 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T01:05:00.068 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2024-04-19T01:05:00.068 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-19T01:05:00.092 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2024-04-19T01:05:00.092 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-19T01:05:00.117 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2024-04-19T01:05:00.118 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-19T01:05:00.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:05:00.114+0000 7f6aadb42640 0 -- 172.21.15.33:0/2715463746 >> [v2:172.21.15.155:6807/741132602,v1:172.21.15.155:6810/741132602] conn(0x7f6aa8224ad0 msgr2=0x7f6aa8226ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-19T01:05:00.128 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:05:00.128+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:05:00.169 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:05:00.169 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-04-19T01:05:00.169 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-04-19T01:05:00.169 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-04-19T01:05:00.169 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.169 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 43/31 ECSubWrite(tid=43, reqid=client.4604.0:251, at_version=43'29, trim_to=0'0, roll_forward_to=43'27))", 2024-04-19T01:05:00.169 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.986035+0000", 2024-04-19T01:05:00.169 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 46.182673792999999, 2024-04-19T01:05:00.170 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024180535999999999, 2024-04-19T01:05:00.170 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:00.170 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:00.170 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:00.170 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:00.170 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.170 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:00.170 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986035+0000", 2024-04-19T01:05:00.170 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.170 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.170 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.170 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:00.171 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986035+0000", 2024-04-19T01:05:00.171 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.171 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.171 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.171 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:00.171 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986058+0000", 2024-04-19T01:05:00.171 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2903000000000001e-05 2024-04-19T01:05:00.171 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.171 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.171 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:00.171 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986841+0000", 2024-04-19T01:05:00.172 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00078285599999999996 2024-04-19T01:05:00.172 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.172 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.172 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:00.172 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986887+0000", 2024-04-19T01:05:00.172 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6202000000000003e-05 2024-04-19T01:05:00.172 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.172 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.172 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:00.172 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986913+0000", 2024-04-19T01:05:00.172 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6162000000000001e-05 2024-04-19T01:05:00.172 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.172 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.172 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:00.173 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987069+0000", 2024-04-19T01:05:00.173 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015625099999999999 2024-04-19T01:05:00.173 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.173 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.173 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:00.173 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987099+0000", 2024-04-19T01:05:00.173 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9578999999999999e-05 2024-04-19T01:05:00.173 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.173 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.173 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:00.173 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.010110+0000", 2024-04-19T01:05:00.173 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023011357999999999 2024-04-19T01:05:00.173 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.173 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.174 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:00.174 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.010215+0000", 2024-04-19T01:05:00.174 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000105225 2024-04-19T01:05:00.174 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.174 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.174 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.174 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.174 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.174 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds5 43/31 ECSubWrite(tid=38, reqid=client.4604.0:253, at_version=43'14, trim_to=0'0, roll_forward_to=43'12))", 2024-04-19T01:05:00.174 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.987245+0000", 2024-04-19T01:05:00.174 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 46.181463770999997, 2024-04-19T01:05:00.174 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022838027, 2024-04-19T01:05:00.175 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:00.175 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:00.175 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:00.175 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:00.175 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.175 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:00.175 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987245+0000", 2024-04-19T01:05:00.175 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.175 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.175 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.175 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:00.175 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987245+0000", 2024-04-19T01:05:00.175 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.175 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.176 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.176 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:00.176 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987247+0000", 2024-04-19T01:05:00.176 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2800000000000002e-06 2024-04-19T01:05:00.176 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.176 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.176 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:00.176 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988142+0000", 2024-04-19T01:05:00.176 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00089449199999999999 2024-04-19T01:05:00.176 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.176 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.176 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:00.176 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988180+0000", 2024-04-19T01:05:00.176 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8615000000000002e-05 2024-04-19T01:05:00.176 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.177 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.177 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:00.177 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988206+0000", 2024-04-19T01:05:00.177 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6023000000000001e-05 2024-04-19T01:05:00.177 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.177 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.177 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:00.177 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988349+0000", 2024-04-19T01:05:00.177 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014252199999999999 2024-04-19T01:05:00.177 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.177 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.177 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:00.177 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988365+0000", 2024-04-19T01:05:00.178 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5855e-05 2024-04-19T01:05:00.178 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.178 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.178 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:00.178 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.010029+0000", 2024-04-19T01:05:00.178 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.021664454999999999 2024-04-19T01:05:00.178 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.179 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.179 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:00.179 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.010083+0000", 2024-04-19T01:05:00.179 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3785000000000001e-05 2024-04-19T01:05:00.179 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.179 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.179 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.179 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.179 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.180 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.fs6 43/31 ECSubWrite(tid=47, reqid=client.4604.0:263, at_version=43'8, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:05:00.180 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.994092+0000", 2024-04-19T01:05:00.180 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 46.174616786000001, 2024-04-19T01:05:00.180 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.027264745999999999, 2024-04-19T01:05:00.180 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:00.180 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:00.180 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:00.180 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:00.180 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.180 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:00.180 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.994092+0000", 2024-04-19T01:05:00.180 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.180 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.180 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.181 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:00.181 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.994092+0000", 2024-04-19T01:05:00.181 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.181 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.181 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.181 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:00.181 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.994094+0000", 2024-04-19T01:05:00.181 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1399999999999998e-06 2024-04-19T01:05:00.181 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.181 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.181 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:00.181 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.994786+0000", 2024-04-19T01:05:00.181 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00069247999999999998 2024-04-19T01:05:00.181 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.181 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.182 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:00.182 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.994827+0000", 2024-04-19T01:05:00.182 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0105000000000003e-05 2024-04-19T01:05:00.182 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.182 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.182 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:00.182 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.994853+0000", 2024-04-19T01:05:00.182 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6259e-05 2024-04-19T01:05:00.182 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.182 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.182 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:00.182 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.995043+0000", 2024-04-19T01:05:00.182 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0001906 2024-04-19T01:05:00.182 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.182 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.183 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:00.183 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.995069+0000", 2024-04-19T01:05:00.183 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5633e-05 2024-04-19T01:05:00.183 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.183 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.183 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:00.183 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.021288+0000", 2024-04-19T01:05:00.183 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.026218865000000001 2024-04-19T01:05:00.183 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.183 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.183 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:00.183 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.021357+0000", 2024-04-19T01:05:00.183 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8664000000000005e-05 2024-04-19T01:05:00.183 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.183 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.184 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.184 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.184 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.184 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.es4 47/31 ECSubWrite(tid=104, reqid=client.4604.0:312, at_version=47'18, trim_to=0'0, roll_forward_to=45'16))", 2024-04-19T01:05:00.184 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.582390+0000", 2024-04-19T01:05:00.184 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 42.586319005, 2024-04-19T01:05:00.184 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022795626999999999, 2024-04-19T01:05:00.186 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:00.186 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:00.186 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:00.186 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:00.186 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.186 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:00.186 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.582390+0000", 2024-04-19T01:05:00.186 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.186 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.186 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.186 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:00.186 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.582390+0000", 2024-04-19T01:05:00.187 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.187 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.187 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.187 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:00.187 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.582390+0000", 2024-04-19T01:05:00.187 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8299999999999996e-07 2024-04-19T01:05:00.187 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.187 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.187 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:00.187 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.582798+0000", 2024-04-19T01:05:00.187 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00040746499999999998 2024-04-19T01:05:00.187 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.187 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.187 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:00.188 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.582829+0000", 2024-04-19T01:05:00.188 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0988999999999998e-05 2024-04-19T01:05:00.188 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.188 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.188 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:00.188 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.582846+0000", 2024-04-19T01:05:00.188 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7252e-05 2024-04-19T01:05:00.188 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.188 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.188 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:00.188 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.582989+0000", 2024-04-19T01:05:00.188 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014342000000000001 2024-04-19T01:05:00.188 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.188 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.189 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:00.189 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.583006+0000", 2024-04-19T01:05:00.189 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6671999999999999e-05 2024-04-19T01:05:00.189 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.189 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.189 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:00.189 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.605104+0000", 2024-04-19T01:05:00.189 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022098222000000001 2024-04-19T01:05:00.189 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.189 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.189 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:00.189 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.605185+0000", 2024-04-19T01:05:00.189 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0823999999999997e-05 2024-04-19T01:05:00.189 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.189 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.190 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.190 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.190 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.190 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.fs6 47/31 ECSubWrite(tid=61, reqid=client.4604.0:317, at_version=47'16, trim_to=0'0, roll_forward_to=45'14))", 2024-04-19T01:05:00.190 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.605368+0000", 2024-04-19T01:05:00.190 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 42.5633409, 2024-04-19T01:05:00.190 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018611628000000002, 2024-04-19T01:05:00.191 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:00.191 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:00.191 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:00.191 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:00.191 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.191 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:00.191 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.605368+0000", 2024-04-19T01:05:00.192 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.192 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.192 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.192 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:00.192 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.605368+0000", 2024-04-19T01:05:00.192 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.192 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.192 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.192 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:00.192 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.605368+0000", 2024-04-19T01:05:00.192 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3700000000000001e-07 2024-04-19T01:05:00.192 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.192 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.192 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:00.193 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.605723+0000", 2024-04-19T01:05:00.193 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035445499999999999 2024-04-19T01:05:00.193 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.193 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.193 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:00.193 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.605744+0000", 2024-04-19T01:05:00.193 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1229e-05 2024-04-19T01:05:00.193 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.193 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.193 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:00.193 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.605759+0000", 2024-04-19T01:05:00.193 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5e-05 2024-04-19T01:05:00.193 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.193 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.193 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:00.194 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.605888+0000", 2024-04-19T01:05:00.194 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012950200000000001 2024-04-19T01:05:00.194 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.194 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.194 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:00.194 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.605906+0000", 2024-04-19T01:05:00.194 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7462e-05 2024-04-19T01:05:00.194 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.194 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.194 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:00.194 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.623858+0000", 2024-04-19T01:05:00.194 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017952630000000001 2024-04-19T01:05:00.194 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.194 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.195 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:00.195 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.623979+0000", 2024-04-19T01:05:00.195 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000120913 2024-04-19T01:05:00.195 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.195 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.195 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.195 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.195 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.195 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds5 47/31 ECSubWrite(tid=61, reqid=client.4604.0:330, at_version=47'32, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:05:00.195 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.677579+0000", 2024-04-19T01:05:00.195 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 42.491129946000001, 2024-04-19T01:05:00.195 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.033702862, 2024-04-19T01:05:00.195 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:00.195 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:00.196 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:00.196 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:00.196 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.196 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:00.196 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.677579+0000", 2024-04-19T01:05:00.196 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.196 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.196 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.196 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:00.196 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.677579+0000", 2024-04-19T01:05:00.196 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.196 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.196 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.196 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:00.196 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.677581+0000", 2024-04-19T01:05:00.197 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2800000000000002e-06 2024-04-19T01:05:00.197 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.197 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.197 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:00.197 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678062+0000", 2024-04-19T01:05:00.197 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00048067900000000002 2024-04-19T01:05:00.197 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.197 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.197 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:00.197 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678096+0000", 2024-04-19T01:05:00.197 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4381999999999997e-05 2024-04-19T01:05:00.197 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.197 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.197 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:00.197 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678117+0000", 2024-04-19T01:05:00.198 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0679999999999999e-05 2024-04-19T01:05:00.198 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.198 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.198 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:00.198 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678249+0000", 2024-04-19T01:05:00.198 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013211899999999999 2024-04-19T01:05:00.198 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.198 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.198 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:00.198 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678270+0000", 2024-04-19T01:05:00.198 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1409000000000001e-05 2024-04-19T01:05:00.198 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.198 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.198 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:00.199 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711228+0000", 2024-04-19T01:05:00.199 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.032957575000000003 2024-04-19T01:05:00.199 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.199 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.199 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:00.199 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711281+0000", 2024-04-19T01:05:00.199 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3737999999999999e-05 2024-04-19T01:05:00.199 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.199 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.199 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.199 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.199 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.199 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds5 47/31 ECSubWrite(tid=62, reqid=client.4604.0:331, at_version=47'33, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:05:00.199 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.682545+0000", 2024-04-19T01:05:00.199 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 42.486163904999998, 2024-04-19T01:05:00.200 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.028771769999999999, 2024-04-19T01:05:00.200 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:00.200 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:00.200 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:00.200 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:00.200 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.200 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:00.200 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682545+0000", 2024-04-19T01:05:00.200 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.200 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.200 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.200 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:00.202 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682545+0000", 2024-04-19T01:05:00.202 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.202 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.202 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.202 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:00.202 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682546+0000", 2024-04-19T01:05:00.202 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.567e-06 2024-04-19T01:05:00.202 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.202 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.203 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:00.203 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682965+0000", 2024-04-19T01:05:00.203 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00041881399999999999 2024-04-19T01:05:00.203 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.203 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.203 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:00.203 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682999+0000", 2024-04-19T01:05:00.203 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3738999999999997e-05 2024-04-19T01:05:00.203 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.203 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.203 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:00.203 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.683021+0000", 2024-04-19T01:05:00.203 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2592e-05 2024-04-19T01:05:00.204 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.204 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.204 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:00.204 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.691707+0000", 2024-04-19T01:05:00.204 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0086853629999999998 2024-04-19T01:05:00.204 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.204 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.204 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:00.204 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.691732+0000", 2024-04-19T01:05:00.204 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5366000000000001e-05 2024-04-19T01:05:00.204 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.204 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.204 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:00.205 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711288+0000", 2024-04-19T01:05:00.205 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019555922999999999 2024-04-19T01:05:00.205 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.205 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.205 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:00.205 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711316+0000", 2024-04-19T01:05:00.205 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8405999999999999e-05 2024-04-19T01:05:00.205 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.205 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.205 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.205 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.205 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.205 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds5 47/31 ECSubWrite(tid=63, reqid=client.4604.0:332, at_version=47'34, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:05:00.205 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.688708+0000", 2024-04-19T01:05:00.206 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 42.480001008000002, 2024-04-19T01:05:00.206 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.02327926, 2024-04-19T01:05:00.206 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:00.206 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:00.206 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:00.206 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:00.206 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.206 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:00.206 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.688708+0000", 2024-04-19T01:05:00.206 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.206 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.206 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.206 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:00.207 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.688708+0000", 2024-04-19T01:05:00.207 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.207 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.207 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.207 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:00.207 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.688709+0000", 2024-04-19T01:05:00.207 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5e-06 2024-04-19T01:05:00.207 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.207 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.207 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:00.207 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.688989+0000", 2024-04-19T01:05:00.207 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00028006800000000002 2024-04-19T01:05:00.207 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.208 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.208 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:00.208 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.689021+0000", 2024-04-19T01:05:00.208 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2026000000000001e-05 2024-04-19T01:05:00.208 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.208 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.208 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:00.208 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.689047+0000", 2024-04-19T01:05:00.208 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5809999999999999e-05 2024-04-19T01:05:00.208 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.208 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.208 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:00.208 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.704765+0000", 2024-04-19T01:05:00.208 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015717913 2024-04-19T01:05:00.209 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.209 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.209 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:00.209 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.704793+0000", 2024-04-19T01:05:00.209 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8143000000000001e-05 2024-04-19T01:05:00.209 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.209 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.209 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:00.209 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711907+0000", 2024-04-19T01:05:00.209 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071141879999999996 2024-04-19T01:05:00.209 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.209 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.209 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:00.209 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711987+0000", 2024-04-19T01:05:00.209 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9611999999999996e-05 2024-04-19T01:05:00.210 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.210 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.210 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.210 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.210 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.210 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds5 47/31 ECSubWrite(tid=64, reqid=client.4604.0:333, at_version=47'35, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:05:00.210 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.692104+0000", 2024-04-19T01:05:00.210 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 42.476604748, 2024-04-19T01:05:00.210 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020504654000000001, 2024-04-19T01:05:00.210 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:00.210 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:00.210 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:00.210 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:00.211 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.211 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:00.211 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692104+0000", 2024-04-19T01:05:00.211 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.211 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.211 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.211 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:00.211 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692104+0000", 2024-04-19T01:05:00.211 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.211 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.211 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.211 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:00.211 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692105+0000", 2024-04-19T01:05:00.211 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.64e-06 2024-04-19T01:05:00.212 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.212 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.212 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:00.212 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692137+0000", 2024-04-19T01:05:00.213 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1631999999999998e-05 2024-04-19T01:05:00.213 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.213 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.213 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:00.213 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692165+0000", 2024-04-19T01:05:00.213 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7526e-05 2024-04-19T01:05:00.213 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.213 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.213 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:00.213 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692180+0000", 2024-04-19T01:05:00.213 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5146e-05 2024-04-19T01:05:00.213 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.213 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.214 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:00.214 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711392+0000", 2024-04-19T01:05:00.214 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019211967 2024-04-19T01:05:00.214 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.214 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.214 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:00.214 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711407+0000", 2024-04-19T01:05:00.214 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4866e-05 2024-04-19T01:05:00.214 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.214 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.214 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:00.214 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.712524+0000", 2024-04-19T01:05:00.214 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011173159999999999 2024-04-19T01:05:00.214 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.215 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.215 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:00.215 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.712608+0000", 2024-04-19T01:05:00.215 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4561000000000006e-05 2024-04-19T01:05:00.215 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.215 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.215 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.215 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.215 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.215 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.fs6 48/31 ECSubWrite(tid=76, reqid=client.4604.0:345, at_version=48'22, trim_to=0'0, roll_forward_to=47'19))", 2024-04-19T01:05:00.215 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:18.538520+0000", 2024-04-19T01:05:00.215 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 41.630188294, 2024-04-19T01:05:00.215 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019548551000000001, 2024-04-19T01:05:00.215 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:00.216 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:00.216 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:00.216 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:00.216 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.216 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:00.216 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.538520+0000", 2024-04-19T01:05:00.216 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.216 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.216 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.216 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:00.216 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.538520+0000", 2024-04-19T01:05:00.216 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.216 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.216 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.216 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:00.217 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.538521+0000", 2024-04-19T01:05:00.217 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.001e-06 2024-04-19T01:05:00.217 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.217 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.217 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:00.217 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.538786+0000", 2024-04-19T01:05:00.217 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026452700000000002 2024-04-19T01:05:00.217 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.217 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.217 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:00.217 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.538803+0000", 2024-04-19T01:05:00.217 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7428999999999999e-05 2024-04-19T01:05:00.217 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.217 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.218 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:00.218 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.538813+0000", 2024-04-19T01:05:00.218 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7040000000000006e-06 2024-04-19T01:05:00.218 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.218 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.218 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:00.218 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.538943+0000", 2024-04-19T01:05:00.218 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013024400000000001 2024-04-19T01:05:00.218 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.218 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.218 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:00.218 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.538970+0000", 2024-04-19T01:05:00.218 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7279000000000001e-05 2024-04-19T01:05:00.219 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.219 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.219 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:00.219 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.557980+0000", 2024-04-19T01:05:00.219 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019009169999999999 2024-04-19T01:05:00.219 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.219 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.219 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:00.219 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:18.558069+0000", 2024-04-19T01:05:00.219 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9197000000000007e-05 2024-04-19T01:05:00.219 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.219 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.219 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.219 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.219 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.220 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 59/31 ECSubWrite(tid=66, reqid=client.4604.0:423, at_version=59'51, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:05:00.220 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.541317+0000", 2024-04-19T01:05:00.220 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 30.627391402000001, 2024-04-19T01:05:00.220 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.047133948000000002, 2024-04-19T01:05:00.220 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:00.220 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:00.220 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:00.220 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:00.220 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.220 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:00.220 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.541317+0000", 2024-04-19T01:05:00.220 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.220 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.220 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.221 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:00.221 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.541317+0000", 2024-04-19T01:05:00.221 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.221 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.221 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.221 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:00.221 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.541318+0000", 2024-04-19T01:05:00.221 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3399999999999999e-07 2024-04-19T01:05:00.221 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.221 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.221 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:00.221 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.543831+0000", 2024-04-19T01:05:00.221 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025130999999999999 2024-04-19T01:05:00.221 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.222 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.222 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:00.222 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.543879+0000", 2024-04-19T01:05:00.222 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8208999999999999e-05 2024-04-19T01:05:00.222 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.222 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.222 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:00.222 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.543903+0000", 2024-04-19T01:05:00.223 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.404e-05 2024-04-19T01:05:00.223 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.223 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.223 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:00.223 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.544070+0000", 2024-04-19T01:05:00.223 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000166372 2024-04-19T01:05:00.223 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.223 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.223 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:00.223 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.544097+0000", 2024-04-19T01:05:00.223 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7256000000000001e-05 2024-04-19T01:05:00.223 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.223 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.223 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:00.224 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.588382+0000", 2024-04-19T01:05:00.224 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.044284780000000003 2024-04-19T01:05:00.224 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.224 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.224 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:00.224 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.588451+0000", 2024-04-19T01:05:00.224 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9556999999999998e-05 2024-04-19T01:05:00.224 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.224 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.224 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.224 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.224 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.224 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 59/31 ECSubWrite(tid=67, reqid=client.4604.0:424, at_version=59'52, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:05:00.224 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.548963+0000", 2024-04-19T01:05:00.225 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 30.619745636000001, 2024-04-19T01:05:00.225 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.039534443000000002, 2024-04-19T01:05:00.225 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:00.225 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:00.225 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:00.225 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:00.225 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.225 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:00.225 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.548963+0000", 2024-04-19T01:05:00.225 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.225 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.225 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.225 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:00.225 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.548963+0000", 2024-04-19T01:05:00.225 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.226 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.226 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.226 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:00.226 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.548965+0000", 2024-04-19T01:05:00.226 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1500000000000002e-06 2024-04-19T01:05:00.226 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.226 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.226 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:00.226 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549551+0000", 2024-04-19T01:05:00.226 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00058640299999999999 2024-04-19T01:05:00.226 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.226 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.226 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:00.226 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549596+0000", 2024-04-19T01:05:00.227 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4604999999999997e-05 2024-04-19T01:05:00.227 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.227 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.227 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:00.227 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549631+0000", 2024-04-19T01:05:00.227 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4725999999999999e-05 2024-04-19T01:05:00.227 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.227 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.227 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:00.227 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.567909+0000", 2024-04-19T01:05:00.227 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018277755999999999 2024-04-19T01:05:00.227 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.227 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.227 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:00.228 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.567944+0000", 2024-04-19T01:05:00.228 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5862999999999999e-05 2024-04-19T01:05:00.228 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.228 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.228 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:00.228 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.588471+0000", 2024-04-19T01:05:00.228 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020526902999999999 2024-04-19T01:05:00.228 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.228 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.228 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:00.228 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.588497+0000", 2024-04-19T01:05:00.228 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6037000000000002e-05 2024-04-19T01:05:00.228 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.228 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.228 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.229 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.229 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.229 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 59/31 ECSubWrite(tid=68, reqid=client.4604.0:425, at_version=59'53, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:05:00.229 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.556523+0000", 2024-04-19T01:05:00.229 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 30.612185225000001, 2024-04-19T01:05:00.229 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.032109275999999999, 2024-04-19T01:05:00.229 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:00.229 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:00.229 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:00.229 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:00.229 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.229 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:00.229 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556523+0000", 2024-04-19T01:05:00.229 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.229 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.230 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.230 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:00.230 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556523+0000", 2024-04-19T01:05:00.230 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.230 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.230 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.230 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:00.230 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556526+0000", 2024-04-19T01:05:00.230 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4930000000000001e-06 2024-04-19T01:05:00.230 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.230 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.230 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:00.230 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556650+0000", 2024-04-19T01:05:00.230 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000124487 2024-04-19T01:05:00.231 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.231 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.231 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:00.231 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556701+0000", 2024-04-19T01:05:00.231 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1158e-05 2024-04-19T01:05:00.231 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.231 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.231 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:00.231 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556727+0000", 2024-04-19T01:05:00.231 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5843999999999999e-05 2024-04-19T01:05:00.231 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.231 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.231 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:00.231 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.585945+0000", 2024-04-19T01:05:00.232 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0292181 2024-04-19T01:05:00.232 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.232 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.232 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:00.232 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.585972+0000", 2024-04-19T01:05:00.232 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6356e-05 2024-04-19T01:05:00.232 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.232 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.232 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:00.232 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.588601+0000", 2024-04-19T01:05:00.232 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0026295839999999999 2024-04-19T01:05:00.232 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.232 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.233 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:00.233 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.588633+0000", 2024-04-19T01:05:00.233 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1254000000000003e-05 2024-04-19T01:05:00.233 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.233 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.233 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.233 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.233 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.234 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 59/31 ECSubWrite(tid=69, reqid=client.4604.0:426, at_version=59'54, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:05:00.234 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.558033+0000", 2024-04-19T01:05:00.234 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 30.610675274999998, 2024-04-19T01:05:00.234 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.030934886000000002, 2024-04-19T01:05:00.234 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:00.234 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:00.234 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:00.234 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:00.234 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.234 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:00.234 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558033+0000", 2024-04-19T01:05:00.234 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.234 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.236 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.236 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:00.236 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558033+0000", 2024-04-19T01:05:00.236 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.236 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.236 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.236 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:00.236 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558035+0000", 2024-04-19T01:05:00.236 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7370000000000001e-06 2024-04-19T01:05:00.236 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.236 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.236 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:00.236 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558080+0000", 2024-04-19T01:05:00.237 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5089000000000001e-05 2024-04-19T01:05:00.237 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.237 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.237 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:00.237 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558126+0000", 2024-04-19T01:05:00.237 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5902000000000002e-05 2024-04-19T01:05:00.237 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.237 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.237 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:00.237 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558149+0000", 2024-04-19T01:05:00.237 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3136e-05 2024-04-19T01:05:00.237 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.237 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.237 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:00.238 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.588071+0000", 2024-04-19T01:05:00.238 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.029921515999999999 2024-04-19T01:05:00.238 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.238 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.238 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:00.238 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.588089+0000", 2024-04-19T01:05:00.238 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8788e-05 2024-04-19T01:05:00.238 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.238 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.239 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:00.239 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.588944+0000", 2024-04-19T01:05:00.239 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00085441300000000005 2024-04-19T01:05:00.239 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.239 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.239 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:00.239 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.588968+0000", 2024-04-19T01:05:00.239 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4304999999999999e-05 2024-04-19T01:05:00.239 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.239 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.239 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.239 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.239 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.239 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 79/31 ECSubWrite(tid=115, reqid=client.4604.0:591, at_version=79'90, trim_to=0'0, roll_forward_to=79'86))", 2024-04-19T01:05:00.240 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:49.657667+0000", 2024-04-19T01:05:00.240 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.511041564999999, 2024-04-19T01:05:00.240 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.042570781000000002, 2024-04-19T01:05:00.240 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:00.240 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:00.240 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:00.240 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:00.240 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.240 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:00.240 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.657667+0000", 2024-04-19T01:05:00.240 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.240 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.240 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.240 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:00.240 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.657667+0000", 2024-04-19T01:05:00.241 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.241 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.241 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.241 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:00.241 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.657669+0000", 2024-04-19T01:05:00.241 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.564e-06 2024-04-19T01:05:00.241 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.241 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.241 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:00.241 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658422+0000", 2024-04-19T01:05:00.241 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00075376200000000005 2024-04-19T01:05:00.241 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.242 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.242 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:00.242 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658456+0000", 2024-04-19T01:05:00.242 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4059e-05 2024-04-19T01:05:00.242 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.242 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.242 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:00.242 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658479+0000", 2024-04-19T01:05:00.242 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2235e-05 2024-04-19T01:05:00.242 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.242 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.242 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:00.242 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658610+0000", 2024-04-19T01:05:00.242 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013109600000000001 2024-04-19T01:05:00.243 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.243 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.243 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:00.243 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658624+0000", 2024-04-19T01:05:00.243 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4756000000000001e-05 2024-04-19T01:05:00.243 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.243 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.243 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:00.243 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.699665+0000", 2024-04-19T01:05:00.243 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.041040764 2024-04-19T01:05:00.243 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.243 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.243 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:00.243 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.700238+0000", 2024-04-19T01:05:00.243 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00057254500000000004 2024-04-19T01:05:00.244 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.244 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.244 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.244 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.244 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.244 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 79/31 ECSubWrite(tid=116, reqid=client.4604.0:592, at_version=79'91, trim_to=0'0, roll_forward_to=79'88))", 2024-04-19T01:05:00.244 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:49.665950+0000", 2024-04-19T01:05:00.244 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.502758979999999, 2024-04-19T01:05:00.244 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.037031065000000002, 2024-04-19T01:05:00.244 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:00.244 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:00.244 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:00.245 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:00.245 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.245 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:00.245 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.665950+0000", 2024-04-19T01:05:00.245 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.245 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.245 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.245 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:00.245 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.665950+0000", 2024-04-19T01:05:00.245 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.246 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.246 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.246 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:00.246 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.665952+0000", 2024-04-19T01:05:00.246 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0360000000000001e-06 2024-04-19T01:05:00.246 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.246 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.246 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:00.246 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.666593+0000", 2024-04-19T01:05:00.246 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00064100300000000002 2024-04-19T01:05:00.246 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.246 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.246 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:00.247 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.666640+0000", 2024-04-19T01:05:00.247 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7817e-05 2024-04-19T01:05:00.247 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.247 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.247 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:00.247 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.666676+0000", 2024-04-19T01:05:00.247 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5222000000000003e-05 2024-04-19T01:05:00.247 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.248 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.248 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:00.248 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.676934+0000", 2024-04-19T01:05:00.248 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010258801999999999 2024-04-19T01:05:00.249 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.249 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.249 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:00.250 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.676964+0000", 2024-04-19T01:05:00.250 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9108999999999999e-05 2024-04-19T01:05:00.250 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.250 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.250 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:00.250 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.702830+0000", 2024-04-19T01:05:00.250 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025866865999999999 2024-04-19T01:05:00.250 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.250 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.251 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:00.251 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.702981+0000", 2024-04-19T01:05:00.251 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015020999999999999 2024-04-19T01:05:00.251 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.251 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.251 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.251 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.251 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.251 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 79/31 ECSubWrite(tid=117, reqid=client.4604.0:593, at_version=79'92, trim_to=0'0, roll_forward_to=79'88))", 2024-04-19T01:05:00.251 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:49.674833+0000", 2024-04-19T01:05:00.251 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.493875461, 2024-04-19T01:05:00.251 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.028247395000000002, 2024-04-19T01:05:00.251 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:00.252 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:00.252 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:00.252 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:00.252 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.252 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:00.252 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.674833+0000", 2024-04-19T01:05:00.252 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.252 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.252 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.253 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:00.253 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.674833+0000", 2024-04-19T01:05:00.253 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.253 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.253 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.253 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:00.253 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.674834+0000", 2024-04-19T01:05:00.253 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3680000000000001e-06 2024-04-19T01:05:00.253 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.253 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.253 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:00.254 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.675074+0000", 2024-04-19T01:05:00.254 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00023945699999999999 2024-04-19T01:05:00.254 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.254 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.254 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:00.254 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.675108+0000", 2024-04-19T01:05:00.254 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4424000000000001e-05 2024-04-19T01:05:00.254 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.254 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.254 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:00.254 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.675127+0000", 2024-04-19T01:05:00.254 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9038999999999998e-05 2024-04-19T01:05:00.254 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.255 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.255 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:00.255 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.690509+0000", 2024-04-19T01:05:00.255 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015381234000000001 2024-04-19T01:05:00.255 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.255 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.255 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:00.255 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.690527+0000", 2024-04-19T01:05:00.255 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8155000000000002e-05 2024-04-19T01:05:00.255 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.255 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.255 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:00.255 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.702999+0000", 2024-04-19T01:05:00.255 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.01247173 2024-04-19T01:05:00.255 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.256 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.256 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:00.256 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.703081+0000", 2024-04-19T01:05:00.256 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1988000000000002e-05 2024-04-19T01:05:00.256 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.256 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.256 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.256 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.256 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.256 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 79/31 ECSubWrite(tid=118, reqid=client.4604.0:594, at_version=79'93, trim_to=0'0, roll_forward_to=79'88))", 2024-04-19T01:05:00.256 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:49.679572+0000", 2024-04-19T01:05:00.256 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.489136422, 2024-04-19T01:05:00.256 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024026347999999999, 2024-04-19T01:05:00.257 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:00.257 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:00.257 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:00.257 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:00.257 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.257 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:00.257 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.679572+0000", 2024-04-19T01:05:00.257 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.257 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.257 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.257 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:00.257 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.679572+0000", 2024-04-19T01:05:00.257 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.257 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.258 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.258 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:00.258 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.679574+0000", 2024-04-19T01:05:00.258 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9700000000000002e-06 2024-04-19T01:05:00.258 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.258 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.258 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:00.258 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.679637+0000", 2024-04-19T01:05:00.258 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2646999999999998e-05 2024-04-19T01:05:00.258 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.258 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.258 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:00.258 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.679677+0000", 2024-04-19T01:05:00.259 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0540000000000001e-05 2024-04-19T01:05:00.259 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.259 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.259 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:00.259 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.679697+0000", 2024-04-19T01:05:00.259 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0108000000000001e-05 2024-04-19T01:05:00.259 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.259 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.259 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:00.259 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.700314+0000", 2024-04-19T01:05:00.259 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020616520999999999 2024-04-19T01:05:00.259 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.260 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.260 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:00.260 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.700332+0000", 2024-04-19T01:05:00.260 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7665000000000001e-05 2024-04-19T01:05:00.260 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.260 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.261 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:00.261 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.703547+0000", 2024-04-19T01:05:00.261 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0032152729999999998 2024-04-19T01:05:00.261 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.261 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.261 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:00.261 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.703598+0000", 2024-04-19T01:05:00.261 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1623999999999999e-05 2024-04-19T01:05:00.261 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.261 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.261 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.261 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.262 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.262 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 79/31 ECSubWrite(tid=119, reqid=client.4604.0:595, at_version=79'94, trim_to=0'0, roll_forward_to=79'88))", 2024-04-19T01:05:00.262 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:49.682205+0000", 2024-04-19T01:05:00.262 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.486503992999999, 2024-04-19T01:05:00.262 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022707974999999998, 2024-04-19T01:05:00.262 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:00.262 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:00.262 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:00.263 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:00.263 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.263 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:00.263 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.682205+0000", 2024-04-19T01:05:00.263 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.263 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.263 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.263 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:00.263 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.682205+0000", 2024-04-19T01:05:00.263 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.263 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.264 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.264 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:00.264 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.682206+0000", 2024-04-19T01:05:00.264 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5149999999999999e-06 2024-04-19T01:05:00.264 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.264 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.264 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:00.264 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.682263+0000", 2024-04-19T01:05:00.264 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6456000000000003e-05 2024-04-19T01:05:00.264 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.265 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.265 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:00.265 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.682303+0000", 2024-04-19T01:05:00.265 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0101000000000001e-05 2024-04-19T01:05:00.265 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.265 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.265 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:00.265 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.682322+0000", 2024-04-19T01:05:00.265 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9266000000000002e-05 2024-04-19T01:05:00.265 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.265 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.265 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:00.266 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.703175+0000", 2024-04-19T01:05:00.266 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020853097000000001 2024-04-19T01:05:00.266 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.266 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.266 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:00.266 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.703191+0000", 2024-04-19T01:05:00.266 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6234000000000001e-05 2024-04-19T01:05:00.266 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.266 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.266 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:00.266 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.704842+0000", 2024-04-19T01:05:00.267 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0016502699999999999 2024-04-19T01:05:00.267 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.267 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.267 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:00.267 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.704913+0000", 2024-04-19T01:05:00.268 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1036000000000002e-05 2024-04-19T01:05:00.268 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.268 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.268 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.268 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.268 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.268 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.es4 86/31 ECSubWrite(tid=195, reqid=client.4604.0:653, at_version=86'39, trim_to=0'0, roll_forward_to=84'38))", 2024-04-19T01:05:00.268 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:56.572490+0000", 2024-04-19T01:05:00.268 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 3.596218296, 2024-04-19T01:05:00.269 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022945307000000002, 2024-04-19T01:05:00.269 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:00.269 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:00.269 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:00.269 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:00.269 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.269 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:00.269 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.572490+0000", 2024-04-19T01:05:00.269 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.269 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.269 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.269 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:00.269 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.572490+0000", 2024-04-19T01:05:00.269 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.269 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.270 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.270 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:00.270 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.572491+0000", 2024-04-19T01:05:00.270 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1800000000000005e-07 2024-04-19T01:05:00.270 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.270 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.270 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:00.270 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.574244+0000", 2024-04-19T01:05:00.270 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017527179999999999 2024-04-19T01:05:00.270 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.270 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.270 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:00.270 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.574268+0000", 2024-04-19T01:05:00.271 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3689999999999998e-05 2024-04-19T01:05:00.271 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.271 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.272 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:00.272 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.574282+0000", 2024-04-19T01:05:00.272 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4011e-05 2024-04-19T01:05:00.272 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.272 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.272 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:00.272 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.574403+0000", 2024-04-19T01:05:00.272 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012181 2024-04-19T01:05:00.272 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.273 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.273 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:00.273 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.574420+0000", 2024-04-19T01:05:00.273 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6390000000000001e-05 2024-04-19T01:05:00.273 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.273 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.273 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:00.273 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.595364+0000", 2024-04-19T01:05:00.274 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020944312999999999 2024-04-19T01:05:00.274 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.274 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.274 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:00.274 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.595436+0000", 2024-04-19T01:05:00.274 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1557000000000006e-05 2024-04-19T01:05:00.274 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.274 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.274 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.274 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.274 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.274 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:05:00.276 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-19T01:05:00.276 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:05:00.275+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:05:00.349 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2024-04-19T01:05:00.349 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-19T01:05:00.377 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:05:00.376+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:05:00.447 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:05:00.447 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-04-19T01:05:00.447 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-04-19T01:05:00.447 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:05:00.461 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-19T01:05:00.507 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:05:00.507+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:05:00.578 INFO:tasks.rados.rados.0.smithi155.stdout:280: done (1 left) 2024-04-19T01:05:00.578 INFO:tasks.rados.rados.0.smithi155.stdout:282: done (0 left) 2024-04-19T01:05:00.578 INFO:tasks.rados.rados.0.smithi155.stdout:283: snap_create 2024-04-19T01:05:00.578 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:05:00.573+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:05:00.582 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:05:00.582 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-04-19T01:05:00.582 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-04-19T01:05:00.582 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-04-19T01:05:00.582 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:05:00.594 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-19T01:05:00.679 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:05:00.680+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:05:00.766 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:05:00.766 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-04-19T01:05:00.767 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-04-19T01:05:00.767 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-04-19T01:05:00.767 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.767 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:240 3.1s0 3:82ff4f48:::smithi15528762-41:head [read 0~688128,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:05:00.767 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.861180+0000", 2024-04-19T01:05:00.767 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 46.905202596999999, 2024-04-19T01:05:00.767 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.026860730999999999, 2024-04-19T01:05:00.767 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:00.767 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:00.767 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-04-19T01:05:00.767 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:05:00.767 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:05:00.767 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:00.768 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 240 2024-04-19T01:05:00.768 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.768 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:00.768 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.768 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:00.768 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861180+0000", 2024-04-19T01:05:00.768 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.768 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.768 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.768 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:00.768 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861180+0000", 2024-04-19T01:05:00.768 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.768 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.769 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.769 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:00.769 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861182+0000", 2024-04-19T01:05:00.769 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.654e-06 2024-04-19T01:05:00.769 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.769 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.769 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:00.769 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861212+0000", 2024-04-19T01:05:00.769 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0511999999999999e-05 2024-04-19T01:05:00.769 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.769 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.769 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:00.769 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861234+0000", 2024-04-19T01:05:00.769 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2135999999999999e-05 2024-04-19T01:05:00.770 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.770 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.770 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:00.770 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861250+0000", 2024-04-19T01:05:00.770 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6141999999999999e-05 2024-04-19T01:05:00.770 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.770 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.770 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:00.770 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.865146+0000", 2024-04-19T01:05:00.770 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0038953350000000002 2024-04-19T01:05:00.770 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.770 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.770 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:05:00.770 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.865375+0000", 2024-04-19T01:05:00.770 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00022880900000000001 2024-04-19T01:05:00.771 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.771 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.771 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:00.771 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.888041+0000", 2024-04-19T01:05:00.771 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022666143 2024-04-19T01:05:00.771 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.771 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.771 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.771 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.771 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.771 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:241 3.1s0 3:82ff4f48:::smithi15528762-41:head [read 0~688128,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:05:00.772 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.861274+0000", 2024-04-19T01:05:00.772 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 46.905108749999997, 2024-04-19T01:05:00.772 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.027990521000000001, 2024-04-19T01:05:00.772 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:00.772 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:00.772 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-04-19T01:05:00.772 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:05:00.772 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:05:00.772 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:00.772 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 241 2024-04-19T01:05:00.773 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.773 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:00.773 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.773 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:00.773 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861274+0000", 2024-04-19T01:05:00.773 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.773 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.773 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.773 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:00.775 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861274+0000", 2024-04-19T01:05:00.775 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.775 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.775 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.775 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:00.775 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861275+0000", 2024-04-19T01:05:00.775 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3030000000000001e-06 2024-04-19T01:05:00.776 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.776 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.776 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:00.776 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861282+0000", 2024-04-19T01:05:00.776 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8569999999999997e-06 2024-04-19T01:05:00.776 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.776 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.776 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:00.776 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861300+0000", 2024-04-19T01:05:00.776 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7898999999999999e-05 2024-04-19T01:05:00.776 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.776 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.776 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:00.776 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.861314+0000", 2024-04-19T01:05:00.776 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.364e-05 2024-04-19T01:05:00.777 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.777 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.777 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:00.777 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.868359+0000", 2024-04-19T01:05:00.777 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0070459019999999997 2024-04-19T01:05:00.777 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.777 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.777 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:05:00.777 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.868594+0000", 2024-04-19T01:05:00.777 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000234942 2024-04-19T01:05:00.777 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.777 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.777 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:00.777 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.889264+0000", 2024-04-19T01:05:00.778 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020669977999999999 2024-04-19T01:05:00.778 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.778 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.778 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.778 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.778 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.778 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s1 43/31 ECSubWrite(tid=43, reqid=client.4604.0:251, at_version=43'29, trim_to=0'0, roll_forward_to=43'27))", 2024-04-19T01:05:00.778 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.985924+0000", 2024-04-19T01:05:00.778 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 46.780458297000003, 2024-04-19T01:05:00.778 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.029591623000000001, 2024-04-19T01:05:00.778 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:00.778 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:00.778 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:00.778 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:00.778 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.779 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:00.779 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.985924+0000", 2024-04-19T01:05:00.779 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.779 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.783 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.783 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:00.783 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.985924+0000", 2024-04-19T01:05:00.783 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.783 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.783 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.783 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:00.783 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.985927+0000", 2024-04-19T01:05:00.783 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3460000000000001e-06 2024-04-19T01:05:00.783 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.784 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.784 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:00.784 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986452+0000", 2024-04-19T01:05:00.784 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00052555199999999999 2024-04-19T01:05:00.784 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.784 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.784 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:00.784 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986541+0000", 2024-04-19T01:05:00.784 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8352999999999999e-05 2024-04-19T01:05:00.785 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.785 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.785 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:00.785 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986597+0000", 2024-04-19T01:05:00.785 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6144999999999999e-05 2024-04-19T01:05:00.785 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.785 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.785 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:00.785 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986776+0000", 2024-04-19T01:05:00.786 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017914999999999999 2024-04-19T01:05:00.786 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.786 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.786 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:00.786 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986811+0000", 2024-04-19T01:05:00.786 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5233e-05 2024-04-19T01:05:00.786 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.786 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.786 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:00.786 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.015444+0000", 2024-04-19T01:05:00.786 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.028632676999999999 2024-04-19T01:05:00.787 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.787 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.787 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:00.787 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.015516+0000", 2024-04-19T01:05:00.787 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2167000000000002e-05 2024-04-19T01:05:00.802 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.802 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.802 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.802 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.802 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.802 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds6 43/31 ECSubWrite(tid=38, reqid=client.4604.0:253, at_version=43'14, trim_to=0'0, roll_forward_to=43'12))", 2024-04-19T01:05:00.802 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.987449+0000", 2024-04-19T01:05:00.802 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 46.778933416999998, 2024-04-19T01:05:00.802 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.027803888999999998, 2024-04-19T01:05:00.802 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:00.803 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:00.803 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:00.803 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:00.803 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.803 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:00.803 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987449+0000", 2024-04-19T01:05:00.803 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.803 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.803 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:00.803 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987449+0000", 2024-04-19T01:05:00.803 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.803 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:00.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987451+0000", 2024-04-19T01:05:00.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2929999999999999e-06 2024-04-19T01:05:00.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:00.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988097+0000", 2024-04-19T01:05:00.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00064595099999999997 2024-04-19T01:05:00.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:00.809 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988140+0000", 2024-04-19T01:05:00.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2809000000000003e-05 2024-04-19T01:05:00.809 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.809 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.809 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:00.809 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988168+0000", 2024-04-19T01:05:00.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7498999999999999e-05 2024-04-19T01:05:00.809 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.809 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.809 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:00.809 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988319+0000", 2024-04-19T01:05:00.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000151621 2024-04-19T01:05:00.810 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.810 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.810 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:00.810 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988345+0000", 2024-04-19T01:05:00.810 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5846e-05 2024-04-19T01:05:00.810 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.810 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.810 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:00.810 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.015183+0000", 2024-04-19T01:05:00.810 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.026837928 2024-04-19T01:05:00.810 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.810 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.810 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:00.810 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.015253+0000", 2024-04-19T01:05:00.811 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9942000000000002e-05 2024-04-19T01:05:00.811 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.811 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.811 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.811 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.811 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.811 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:289 3.1s0 3:8ea800b0:::smithi15528762-8:head [stat,delete] snapc 7=[7,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-19T01:05:00.811 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.872650+0000", 2024-04-19T01:05:00.811 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 45.893732528999998, 2024-04-19T01:05:00.811 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.043399042999999998, 2024-04-19T01:05:00.811 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:00.811 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:00.811 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:00.811 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:05:00.811 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:05:00.812 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:00.812 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 289 2024-04-19T01:05:00.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.812 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:00.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.812 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:00.812 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.872650+0000", 2024-04-19T01:05:00.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.812 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:00.812 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.872650+0000", 2024-04-19T01:05:00.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:00.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.872653+0000", 2024-04-19T01:05:00.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.23e-06 2024-04-19T01:05:00.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:00.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.872669+0000", 2024-04-19T01:05:00.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5293000000000001e-05 2024-04-19T01:05:00.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:00.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.872701+0000", 2024-04-19T01:05:00.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2391999999999998e-05 2024-04-19T01:05:00.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:00.814 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.872721+0000", 2024-04-19T01:05:00.814 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0400000000000001e-05 2024-04-19T01:05:00.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:00.814 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.872889+0000", 2024-04-19T01:05:00.814 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016723000000000001 2024-04-19T01:05:00.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:05:00.814 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.873172+0000", 2024-04-19T01:05:00.814 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000283338 2024-04-19T01:05:00.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.815 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.815 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:00.815 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.875406+0000", 2024-04-19T01:05:00.815 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022335839999999998 2024-04-19T01:05:00.815 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.815 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.815 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:00.815 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.914580+0000", 2024-04-19T01:05:00.816 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.039174253999999999 2024-04-19T01:05:00.816 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.816 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.816 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-04-19T01:05:00.816 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.914900+0000", 2024-04-19T01:05:00.816 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000320663 2024-04-19T01:05:00.816 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.816 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.816 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:00.816 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.916049+0000", 2024-04-19T01:05:00.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001148659 2024-04-19T01:05:00.817 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.817 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.817 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.817 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s6 44/31 ECSubWrite(tid=82, reqid=client.4604.0:280, at_version=44'50, trim_to=0'0, roll_forward_to=43'48))", 2024-04-19T01:05:00.817 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.882093+0000", 2024-04-19T01:05:00.817 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 45.884289234999997, 2024-04-19T01:05:00.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.042938993000000002, 2024-04-19T01:05:00.817 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:00.817 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:00.817 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:00.817 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:00.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:00.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882093+0000", 2024-04-19T01:05:00.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:00.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882093+0000", 2024-04-19T01:05:00.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:00.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882094+0000", 2024-04-19T01:05:00.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0130000000000001e-06 2024-04-19T01:05:00.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.819 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:00.819 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882590+0000", 2024-04-19T01:05:00.819 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00049558999999999996 2024-04-19T01:05:00.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.819 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:00.819 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882631+0000", 2024-04-19T01:05:00.819 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1399000000000001e-05 2024-04-19T01:05:00.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.819 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:00.819 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882654+0000", 2024-04-19T01:05:00.820 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2535999999999999e-05 2024-04-19T01:05:00.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.820 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:00.820 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882871+0000", 2024-04-19T01:05:00.820 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021730999999999999 2024-04-19T01:05:00.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.820 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:00.820 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882896+0000", 2024-04-19T01:05:00.820 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5111999999999999e-05 2024-04-19T01:05:00.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.821 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:00.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.924961+0000", 2024-04-19T01:05:00.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.042065022000000001 2024-04-19T01:05:00.821 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.821 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:00.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.925032+0000", 2024-04-19T01:05:00.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1011000000000002e-05 2024-04-19T01:05:00.821 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.821 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.821 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.821 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.821 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.0s6 44/31 ECSubWrite(tid=25, reqid=client.4604.0:284, at_version=44'22, trim_to=0'0, roll_forward_to=44'20))", 2024-04-19T01:05:00.822 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.921980+0000", 2024-04-19T01:05:00.822 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 45.844402961999997, 2024-04-19T01:05:00.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.036689795999999997, 2024-04-19T01:05:00.822 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:00.822 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:00.822 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:00.822 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:00.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:00.822 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.921980+0000", 2024-04-19T01:05:00.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:00.822 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.921980+0000", 2024-04-19T01:05:00.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:00.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.921981+0000", 2024-04-19T01:05:00.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.378e-06 2024-04-19T01:05:00.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:00.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.923355+0000", 2024-04-19T01:05:00.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0013742820000000001 2024-04-19T01:05:00.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:00.824 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.923387+0000", 2024-04-19T01:05:00.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1702999999999998e-05 2024-04-19T01:05:00.824 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.824 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.824 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:00.824 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.923405+0000", 2024-04-19T01:05:00.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8071999999999999e-05 2024-04-19T01:05:00.824 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.824 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.824 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:00.824 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.923699+0000", 2024-04-19T01:05:00.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029384700000000002 2024-04-19T01:05:00.824 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.824 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.824 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:00.825 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.923727+0000", 2024-04-19T01:05:00.825 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8085999999999998e-05 2024-04-19T01:05:00.825 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.825 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.825 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:00.825 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.958618+0000", 2024-04-19T01:05:00.825 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.034891349000000002 2024-04-19T01:05:00.825 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.825 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.826 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:00.826 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.958669+0000", 2024-04-19T01:05:00.826 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1078999999999997e-05 2024-04-19T01:05:00.826 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.826 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.826 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.826 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.826 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.826 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds6 47/31 ECSubWrite(tid=61, reqid=client.4604.0:330, at_version=47'32, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:05:00.826 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.677697+0000", 2024-04-19T01:05:00.827 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 43.088686017999997, 2024-04-19T01:05:00.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.034267463999999997, 2024-04-19T01:05:00.827 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:00.827 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:00.827 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:00.827 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:00.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.827 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:00.827 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.677697+0000", 2024-04-19T01:05:00.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.828 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:00.828 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.677697+0000", 2024-04-19T01:05:00.828 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.828 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:00.828 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.677699+0000", 2024-04-19T01:05:00.828 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3929999999999998e-06 2024-04-19T01:05:00.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.828 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:00.828 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678182+0000", 2024-04-19T01:05:00.828 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00048326199999999998 2024-04-19T01:05:00.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.829 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:00.829 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678227+0000", 2024-04-19T01:05:00.829 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4301999999999997e-05 2024-04-19T01:05:00.829 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.829 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.829 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:00.829 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678254+0000", 2024-04-19T01:05:00.829 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7396e-05 2024-04-19T01:05:00.829 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.829 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.829 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:00.829 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678409+0000", 2024-04-19T01:05:00.829 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000154759 2024-04-19T01:05:00.829 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.829 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.830 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:00.830 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678438+0000", 2024-04-19T01:05:00.830 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8926000000000001e-05 2024-04-19T01:05:00.830 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.830 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.830 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:00.830 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711880+0000", 2024-04-19T01:05:00.830 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.033442209000000001 2024-04-19T01:05:00.830 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.830 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.830 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:00.830 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711964+0000", 2024-04-19T01:05:00.830 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4216999999999997e-05 2024-04-19T01:05:00.830 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.830 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.831 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.831 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.831 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.831 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds6 47/31 ECSubWrite(tid=62, reqid=client.4604.0:331, at_version=47'33, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:05:00.831 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.682679+0000", 2024-04-19T01:05:00.831 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 43.083703628000002, 2024-04-19T01:05:00.831 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.029354201999999999, 2024-04-19T01:05:00.831 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:00.831 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:00.831 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:00.831 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:00.831 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.831 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:00.831 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682679+0000", 2024-04-19T01:05:00.831 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.832 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.832 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.832 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:00.832 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682679+0000", 2024-04-19T01:05:00.832 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.832 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.832 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.832 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:00.832 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682681+0000", 2024-04-19T01:05:00.832 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7799999999999999e-06 2024-04-19T01:05:00.832 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.832 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.832 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:00.832 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682945+0000", 2024-04-19T01:05:00.832 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000263875 2024-04-19T01:05:00.833 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.833 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.833 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:00.833 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682975+0000", 2024-04-19T01:05:00.833 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0216e-05 2024-04-19T01:05:00.833 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.833 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.833 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:00.833 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682990+0000", 2024-04-19T01:05:00.833 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5333e-05 2024-04-19T01:05:00.833 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.833 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.833 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:00.833 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692463+0000", 2024-04-19T01:05:00.833 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0094724660000000006 2024-04-19T01:05:00.834 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.834 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:00.834 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692479+0000", 2024-04-19T01:05:00.834 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5892e-05 2024-04-19T01:05:00.834 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.834 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:00.834 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711976+0000", 2024-04-19T01:05:00.834 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019497012000000001 2024-04-19T01:05:00.834 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.834 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:00.834 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.712033+0000", 2024-04-19T01:05:00.834 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7627999999999998e-05 2024-04-19T01:05:00.835 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.835 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.835 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.835 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.835 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.835 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s1 59/31 ECSubWrite(tid=66, reqid=client.4604.0:423, at_version=59'51, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:05:00.835 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.541655+0000", 2024-04-19T01:05:00.835 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 31.224727582, 2024-04-19T01:05:00.835 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.038825746000000001, 2024-04-19T01:05:00.835 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:00.835 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:00.835 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:00.835 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:00.835 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.836 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:00.836 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.541655+0000", 2024-04-19T01:05:00.836 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.836 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:00.836 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.541655+0000", 2024-04-19T01:05:00.836 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.837 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.837 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.837 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:00.837 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.541656+0000", 2024-04-19T01:05:00.837 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1699999999999998e-07 2024-04-19T01:05:00.837 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.837 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.837 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:00.837 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.544278+0000", 2024-04-19T01:05:00.838 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002622728 2024-04-19T01:05:00.838 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.838 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.838 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:00.838 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.544314+0000", 2024-04-19T01:05:00.840 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5751999999999999e-05 2024-04-19T01:05:00.840 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.840 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.840 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:00.840 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.544338+0000", 2024-04-19T01:05:00.841 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4003e-05 2024-04-19T01:05:00.841 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.841 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.841 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:00.841 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.544577+0000", 2024-04-19T01:05:00.841 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00023931500000000001 2024-04-19T01:05:00.841 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.841 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.841 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:00.841 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.544603+0000", 2024-04-19T01:05:00.841 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5854999999999999e-05 2024-04-19T01:05:00.841 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.841 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.841 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:00.841 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.580392+0000", 2024-04-19T01:05:00.842 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.035788477999999999 2024-04-19T01:05:00.842 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.842 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.842 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:00.842 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.580481+0000", 2024-04-19T01:05:00.842 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8998000000000004e-05 2024-04-19T01:05:00.842 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.842 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.842 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.842 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.842 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.842 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s1 59/31 ECSubWrite(tid=67, reqid=client.4604.0:424, at_version=59'52, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:05:00.842 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.549050+0000", 2024-04-19T01:05:00.842 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 31.217332209999999, 2024-04-19T01:05:00.843 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.031505582999999997, 2024-04-19T01:05:00.843 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:00.843 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:00.843 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:00.843 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:00.843 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.843 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:00.843 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549050+0000", 2024-04-19T01:05:00.843 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.843 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.843 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.843 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:00.843 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549050+0000", 2024-04-19T01:05:00.843 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.843 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.844 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.844 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:00.844 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549052+0000", 2024-04-19T01:05:00.844 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0669999999999999e-06 2024-04-19T01:05:00.844 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.844 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.844 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:00.844 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.550013+0000", 2024-04-19T01:05:00.844 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00096065699999999996 2024-04-19T01:05:00.844 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.844 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.844 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:00.844 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.550048+0000", 2024-04-19T01:05:00.844 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4535000000000001e-05 2024-04-19T01:05:00.844 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.845 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.845 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:00.845 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.550071+0000", 2024-04-19T01:05:00.845 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3057e-05 2024-04-19T01:05:00.845 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.845 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.845 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:00.845 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.561847+0000", 2024-04-19T01:05:00.845 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.01177603 2024-04-19T01:05:00.845 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.845 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.845 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:00.845 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.561873+0000", 2024-04-19T01:05:00.845 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6679999999999999e-05 2024-04-19T01:05:00.845 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.846 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.846 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:00.846 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.580493+0000", 2024-04-19T01:05:00.846 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018619125 2024-04-19T01:05:00.846 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.846 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.846 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:00.846 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.580556+0000", 2024-04-19T01:05:00.846 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3431999999999998e-05 2024-04-19T01:05:00.847 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.847 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.847 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.847 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.847 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.847 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s1 79/31 ECSubWrite(tid=115, reqid=client.4604.0:591, at_version=79'90, trim_to=0'0, roll_forward_to=79'86))", 2024-04-19T01:05:00.847 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:49.658363+0000", 2024-04-19T01:05:00.847 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.108019777000001, 2024-04-19T01:05:00.848 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.046590765999999999, 2024-04-19T01:05:00.848 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:00.848 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:00.848 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:00.848 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:00.848 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.848 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:00.848 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658363+0000", 2024-04-19T01:05:00.848 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.848 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.848 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.848 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:00.848 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658363+0000", 2024-04-19T01:05:00.848 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.848 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.849 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.849 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:00.849 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658364+0000", 2024-04-19T01:05:00.849 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6759999999999999e-06 2024-04-19T01:05:00.849 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.849 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.849 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:00.849 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658923+0000", 2024-04-19T01:05:00.849 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00055818399999999998 2024-04-19T01:05:00.849 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.849 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.849 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:00.849 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658954+0000", 2024-04-19T01:05:00.849 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1554999999999999e-05 2024-04-19T01:05:00.849 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.850 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.850 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:00.850 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658977+0000", 2024-04-19T01:05:00.850 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3005000000000001e-05 2024-04-19T01:05:00.850 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.850 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.850 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:00.850 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.659163+0000", 2024-04-19T01:05:00.850 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018526400000000001 2024-04-19T01:05:00.850 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.850 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.850 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:00.850 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.659193+0000", 2024-04-19T01:05:00.850 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0348000000000001e-05 2024-04-19T01:05:00.851 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.851 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.851 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:00.851 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.704869+0000", 2024-04-19T01:05:00.851 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.045676481999999997 2024-04-19T01:05:00.851 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.851 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.851 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:00.851 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.704954+0000", 2024-04-19T01:05:00.851 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4252000000000006e-05 2024-04-19T01:05:00.851 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.851 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.851 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.851 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.851 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.852 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s1 79/31 ECSubWrite(tid=116, reqid=client.4604.0:592, at_version=79'91, trim_to=0'0, roll_forward_to=79'88))", 2024-04-19T01:05:00.852 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:49.666488+0000", 2024-04-19T01:05:00.852 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.099894461, 2024-04-19T01:05:00.852 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.040748924999999998, 2024-04-19T01:05:00.852 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:00.852 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:00.852 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:00.852 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:00.852 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.852 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:00.852 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.666488+0000", 2024-04-19T01:05:00.852 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.852 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.852 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.852 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:00.852 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.666488+0000", 2024-04-19T01:05:00.853 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.853 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.853 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.853 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:00.853 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.666490+0000", 2024-04-19T01:05:00.853 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0030000000000001e-06 2024-04-19T01:05:00.853 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.853 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.853 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:00.853 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.666740+0000", 2024-04-19T01:05:00.853 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00024945000000000002 2024-04-19T01:05:00.853 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.853 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.853 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:00.854 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.666785+0000", 2024-04-19T01:05:00.854 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5145000000000002e-05 2024-04-19T01:05:00.854 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.854 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.854 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:00.854 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.666810+0000", 2024-04-19T01:05:00.854 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4915000000000001e-05 2024-04-19T01:05:00.854 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.854 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.854 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:00.854 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.677635+0000", 2024-04-19T01:05:00.854 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010825293999999999 2024-04-19T01:05:00.854 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.854 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.855 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:00.855 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.677659+0000", 2024-04-19T01:05:00.855 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3896e-05 2024-04-19T01:05:00.855 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.855 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.855 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:00.855 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.707167+0000", 2024-04-19T01:05:00.855 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.029507681000000001 2024-04-19T01:05:00.855 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.855 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.855 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:00.855 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.707237+0000", 2024-04-19T01:05:00.855 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0541000000000001e-05 2024-04-19T01:05:00.855 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.855 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.856 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.856 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.856 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.856 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s1 79/31 ECSubWrite(tid=117, reqid=client.4604.0:593, at_version=79'92, trim_to=0'0, roll_forward_to=79'88))", 2024-04-19T01:05:00.856 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:49.674022+0000", 2024-04-19T01:05:00.856 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.092360296000001, 2024-04-19T01:05:00.856 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.033276779999999999, 2024-04-19T01:05:00.856 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:00.856 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:00.856 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:00.856 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:00.856 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.857 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:00.857 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.674022+0000", 2024-04-19T01:05:00.857 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.857 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.857 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.857 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:00.857 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.674022+0000", 2024-04-19T01:05:00.857 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.857 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.857 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.857 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:00.858 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.674025+0000", 2024-04-19T01:05:00.858 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.384e-06 2024-04-19T01:05:00.858 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.858 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.858 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:00.858 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.674363+0000", 2024-04-19T01:05:00.858 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00033799500000000001 2024-04-19T01:05:00.858 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.858 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.858 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:00.858 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.674412+0000", 2024-04-19T01:05:00.858 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8844000000000002e-05 2024-04-19T01:05:00.858 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.858 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.859 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:00.859 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.674436+0000", 2024-04-19T01:05:00.859 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4409e-05 2024-04-19T01:05:00.859 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.859 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.859 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:00.859 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.693200+0000", 2024-04-19T01:05:00.859 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018764268000000001 2024-04-19T01:05:00.859 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.859 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.859 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:00.859 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.693224+0000", 2024-04-19T01:05:00.859 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3646999999999999e-05 2024-04-19T01:05:00.859 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.859 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.859 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:00.860 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.707249+0000", 2024-04-19T01:05:00.860 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014025055 2024-04-19T01:05:00.860 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.860 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.860 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:00.860 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.707299+0000", 2024-04-19T01:05:00.860 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0178e-05 2024-04-19T01:05:00.860 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.860 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.860 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.860 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.860 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.860 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s1 79/31 ECSubWrite(tid=118, reqid=client.4604.0:594, at_version=79'93, trim_to=0'0, roll_forward_to=79'88))", 2024-04-19T01:05:00.860 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:49.679987+0000", 2024-04-19T01:05:00.860 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.086395123000001, 2024-04-19T01:05:00.861 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.02808188, 2024-04-19T01:05:00.861 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:00.861 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:00.861 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:00.861 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:00.861 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.861 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:00.861 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.679987+0000", 2024-04-19T01:05:00.861 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.861 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.861 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.861 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:00.861 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.679987+0000", 2024-04-19T01:05:00.861 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.862 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.862 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.862 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:00.862 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.679990+0000", 2024-04-19T01:05:00.862 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1600000000000001e-06 2024-04-19T01:05:00.862 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.862 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.862 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:00.862 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.680086+0000", 2024-04-19T01:05:00.862 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6485999999999997e-05 2024-04-19T01:05:00.862 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.862 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.862 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:00.862 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.680125+0000", 2024-04-19T01:05:00.863 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9212000000000003e-05 2024-04-19T01:05:00.863 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.863 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.863 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:00.863 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.680147+0000", 2024-04-19T01:05:00.863 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1951e-05 2024-04-19T01:05:00.863 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.863 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.863 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:00.863 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.705049+0000", 2024-04-19T01:05:00.863 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024901878999999998 2024-04-19T01:05:00.863 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.863 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.863 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:00.863 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.705070+0000", 2024-04-19T01:05:00.864 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0577999999999998e-05 2024-04-19T01:05:00.864 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.864 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.864 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:00.864 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.708014+0000", 2024-04-19T01:05:00.864 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0029441519999999998 2024-04-19T01:05:00.864 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.864 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.864 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:00.864 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.708069+0000", 2024-04-19T01:05:00.864 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5461999999999999e-05 2024-04-19T01:05:00.864 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.864 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.864 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.864 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.864 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.865 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s1 81/31 ECSubWrite(tid=122, reqid=client.4604.0:626, at_version=81'96, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:05:00.865 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:51.594420+0000", 2024-04-19T01:05:00.865 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 9.1719623460000008, 2024-04-19T01:05:00.865 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.041892138000000002, 2024-04-19T01:05:00.865 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:00.865 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:00.865 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:00.865 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:00.865 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.865 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:00.865 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.594420+0000", 2024-04-19T01:05:00.865 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.865 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.865 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.865 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:00.866 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.594420+0000", 2024-04-19T01:05:00.866 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.866 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.866 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.866 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:00.866 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.594421+0000", 2024-04-19T01:05:00.866 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9299999999999997e-07 2024-04-19T01:05:00.866 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.866 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.866 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:00.866 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.595515+0000", 2024-04-19T01:05:00.866 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0010942829999999999 2024-04-19T01:05:00.866 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.866 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.867 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:00.867 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.595545+0000", 2024-04-19T01:05:00.867 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.957e-05 2024-04-19T01:05:00.867 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.867 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.867 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:00.867 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.595570+0000", 2024-04-19T01:05:00.867 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5486999999999998e-05 2024-04-19T01:05:00.868 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.868 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.868 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:00.868 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.595717+0000", 2024-04-19T01:05:00.868 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000146487 2024-04-19T01:05:00.868 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.868 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.868 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:00.868 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.595739+0000", 2024-04-19T01:05:00.868 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2279000000000001e-05 2024-04-19T01:05:00.868 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.868 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.869 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:00.869 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.636238+0000", 2024-04-19T01:05:00.869 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.040499304999999999 2024-04-19T01:05:00.869 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.869 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.869 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:00.869 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.636312+0000", 2024-04-19T01:05:00.869 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3934000000000001e-05 2024-04-19T01:05:00.869 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.869 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.869 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.869 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.869 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.869 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s1 81/31 ECSubWrite(tid=123, reqid=client.4604.0:627, at_version=81'97, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:05:00.869 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:51.600837+0000", 2024-04-19T01:05:00.870 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 9.1655458749999994, 2024-04-19T01:05:00.870 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.035530869, 2024-04-19T01:05:00.870 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:00.870 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:00.870 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:00.870 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:00.870 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.870 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:00.870 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.600837+0000", 2024-04-19T01:05:00.870 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.870 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.870 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.870 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:00.870 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.600837+0000", 2024-04-19T01:05:00.871 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.871 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.871 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.871 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:00.871 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.600839+0000", 2024-04-19T01:05:00.871 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9130000000000001e-06 2024-04-19T01:05:00.871 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.871 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.871 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:00.871 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.601189+0000", 2024-04-19T01:05:00.871 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035037499999999999 2024-04-19T01:05:00.871 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.871 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.871 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:00.871 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.601223+0000", 2024-04-19T01:05:00.872 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4215999999999999e-05 2024-04-19T01:05:00.872 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.872 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.872 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:00.872 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.601246+0000", 2024-04-19T01:05:00.872 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2775000000000001e-05 2024-04-19T01:05:00.872 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.872 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.872 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:00.872 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.613649+0000", 2024-04-19T01:05:00.872 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012402826 2024-04-19T01:05:00.872 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.872 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.872 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:00.872 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.613672+0000", 2024-04-19T01:05:00.873 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2719000000000001e-05 2024-04-19T01:05:00.873 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.873 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.873 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:00.873 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.636328+0000", 2024-04-19T01:05:00.873 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022656859000000001 2024-04-19T01:05:00.873 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.873 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.873 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:00.873 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.636368+0000", 2024-04-19T01:05:00.873 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9186000000000001e-05 2024-04-19T01:05:00.873 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.873 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.873 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.873 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.873 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.874 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s1 81/31 ECSubWrite(tid=124, reqid=client.4604.0:628, at_version=81'98, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:05:00.874 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:51.606688+0000", 2024-04-19T01:05:00.874 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 9.1596943849999999, 2024-04-19T01:05:00.874 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.045868434999999999, 2024-04-19T01:05:00.874 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:00.874 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:00.874 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:00.874 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:00.874 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.874 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:00.874 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.606688+0000", 2024-04-19T01:05:00.874 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.874 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.874 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.875 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:00.875 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.606688+0000", 2024-04-19T01:05:00.875 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.875 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.875 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.875 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:00.875 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.606690+0000", 2024-04-19T01:05:00.875 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9130000000000001e-06 2024-04-19T01:05:00.875 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.875 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.875 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:00.875 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.607071+0000", 2024-04-19T01:05:00.875 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00038100699999999999 2024-04-19T01:05:00.875 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.875 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.876 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:00.876 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.607111+0000", 2024-04-19T01:05:00.876 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9915e-05 2024-04-19T01:05:00.876 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.876 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.876 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:00.876 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.607133+0000", 2024-04-19T01:05:00.876 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2252999999999999e-05 2024-04-19T01:05:00.876 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.876 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.876 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:00.876 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.626657+0000", 2024-04-19T01:05:00.876 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019523934999999999 2024-04-19T01:05:00.876 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.876 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.877 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:00.877 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.626714+0000", 2024-04-19T01:05:00.877 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6804e-05 2024-04-19T01:05:00.877 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.877 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.877 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:00.877 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.652491+0000", 2024-04-19T01:05:00.877 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025776838999999999 2024-04-19T01:05:00.877 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.878 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.878 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:00.878 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.652557+0000", 2024-04-19T01:05:00.878 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5769e-05 2024-04-19T01:05:00.878 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.878 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.878 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.878 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.878 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.878 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s1 81/31 ECSubWrite(tid=125, reqid=client.4604.0:629, at_version=81'99, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:05:00.878 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:51.610687+0000", 2024-04-19T01:05:00.879 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 9.1556960909999994, 2024-04-19T01:05:00.879 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.041959031000000001, 2024-04-19T01:05:00.879 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:00.879 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:00.879 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:00.879 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:00.879 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.879 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:00.879 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.610687+0000", 2024-04-19T01:05:00.879 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.879 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.879 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.879 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:00.879 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.610687+0000", 2024-04-19T01:05:00.879 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.880 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.880 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.880 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:00.880 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.610688+0000", 2024-04-19T01:05:00.880 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.99e-06 2024-04-19T01:05:00.880 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.880 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.880 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:00.880 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.610734+0000", 2024-04-19T01:05:00.880 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5674999999999999e-05 2024-04-19T01:05:00.880 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.880 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.880 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:00.880 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.610779+0000", 2024-04-19T01:05:00.880 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5277e-05 2024-04-19T01:05:00.881 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.881 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.881 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:00.881 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.610803+0000", 2024-04-19T01:05:00.881 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3742000000000001e-05 2024-04-19T01:05:00.881 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.881 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.881 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:00.881 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.635814+0000", 2024-04-19T01:05:00.881 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025010674 2024-04-19T01:05:00.881 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.881 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.881 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:00.881 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.635834+0000", 2024-04-19T01:05:00.882 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9755999999999999e-05 2024-04-19T01:05:00.882 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.882 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.882 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:00.882 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.652600+0000", 2024-04-19T01:05:00.882 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016766476999999998 2024-04-19T01:05:00.882 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.882 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.882 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:00.882 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.652646+0000", 2024-04-19T01:05:00.882 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5439999999999999e-05 2024-04-19T01:05:00.882 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.882 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.882 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.882 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.883 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.883 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s1 81/31 ECSubWrite(tid=126, reqid=client.4604.0:631, at_version=81'101, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:05:00.883 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:51.619941+0000", 2024-04-19T01:05:00.883 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 9.1464414479999991, 2024-04-19T01:05:00.883 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.03518276, 2024-04-19T01:05:00.883 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:00.883 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:00.883 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:00.883 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:00.883 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.883 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:00.883 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.619941+0000", 2024-04-19T01:05:00.883 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.883 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.883 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.884 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:00.884 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.619941+0000", 2024-04-19T01:05:00.884 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:00.884 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.884 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.884 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:00.884 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.619942+0000", 2024-04-19T01:05:00.884 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.31e-06 2024-04-19T01:05:00.884 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.884 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.884 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:00.884 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.620655+0000", 2024-04-19T01:05:00.884 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00071261299999999996 2024-04-19T01:05:00.884 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.885 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.885 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:00.885 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.620689+0000", 2024-04-19T01:05:00.885 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3809000000000001e-05 2024-04-19T01:05:00.885 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.885 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.885 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:00.885 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.620708+0000", 2024-04-19T01:05:00.885 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8929000000000001e-05 2024-04-19T01:05:00.885 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.885 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.885 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:00.885 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.636467+0000", 2024-04-19T01:05:00.885 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015758892 2024-04-19T01:05:00.885 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.886 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.886 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:00.886 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.636484+0000", 2024-04-19T01:05:00.886 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6892999999999999e-05 2024-04-19T01:05:00.886 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.886 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.886 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:00.886 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.655090+0000", 2024-04-19T01:05:00.886 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018606304000000001 2024-04-19T01:05:00.886 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:00.886 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:00.886 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:00.886 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.655124+0000", 2024-04-19T01:05:00.886 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4010000000000001e-05 2024-04-19T01:05:00.886 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.887 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.887 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.887 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:00.887 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:00.887 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:05:00.889 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-19T01:05:00.889 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:05:00.891+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:05:00.989 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:05:00.985+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:05:01.060 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:05:01.060 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-04-19T01:05:01.060 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-04-19T01:05:01.060 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:05:01.074 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-19T01:05:01.090 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:05:01.089+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:05:01.191 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:05:01.189+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:05:01.193 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:05:01.193 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-04-19T01:05:01.193 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-04-19T01:05:01.193 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-04-19T01:05:01.193 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:05:01.209 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-19T01:05:01.292 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:05:01.292+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:05:01.338 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:05:01.339 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-04-19T01:05:01.339 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-04-19T01:05:01.339 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-04-19T01:05:01.339 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.339 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s5 44/31 ECSubWrite(tid=82, reqid=client.4604.0:280, at_version=44'50, trim_to=0'0, roll_forward_to=43'48))", 2024-04-19T01:05:01.339 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.886693+0000", 2024-04-19T01:05:01.339 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 46.451058525999997, 2024-04-19T01:05:01.339 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.044983205999999998, 2024-04-19T01:05:01.339 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:01.339 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:01.341 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:01.341 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:01.341 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.341 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:01.341 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.886693+0000", 2024-04-19T01:05:01.341 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.341 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.341 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.342 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:01.342 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.886693+0000", 2024-04-19T01:05:01.342 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.342 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.342 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.342 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:01.342 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.886694+0000", 2024-04-19T01:05:01.342 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2699999999999998e-07 2024-04-19T01:05:01.342 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.342 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.342 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:01.342 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.887101+0000", 2024-04-19T01:05:01.342 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00040740299999999998 2024-04-19T01:05:01.342 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.343 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.343 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:01.343 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.887130+0000", 2024-04-19T01:05:01.343 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9479e-05 2024-04-19T01:05:01.343 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.343 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.343 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:01.343 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.887152+0000", 2024-04-19T01:05:01.343 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1708999999999998e-05 2024-04-19T01:05:01.343 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.343 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.343 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:01.343 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.887281+0000", 2024-04-19T01:05:01.344 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012845599999999999 2024-04-19T01:05:01.344 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.344 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.344 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:01.344 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.887296+0000", 2024-04-19T01:05:01.344 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5642999999999999e-05 2024-04-19T01:05:01.344 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.344 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.344 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:01.344 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.931601+0000", 2024-04-19T01:05:01.344 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.044304705 2024-04-19T01:05:01.344 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.344 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.344 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:01.345 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.931676+0000", 2024-04-19T01:05:01.345 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4983999999999999e-05 2024-04-19T01:05:01.345 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.345 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:01.345 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.345 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.345 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.345 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 59/31 ECSubWrite(tid=66, reqid=client.4604.0:423, at_version=59'51, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:05:01.345 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.544994+0000", 2024-04-19T01:05:01.345 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 31.792757588000001, 2024-04-19T01:05:01.345 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.042329451999999997, 2024-04-19T01:05:01.345 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:01.346 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:01.346 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:01.346 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:01.346 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.346 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:01.348 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.544994+0000", 2024-04-19T01:05:01.348 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.348 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.348 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.348 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:01.348 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.544994+0000", 2024-04-19T01:05:01.349 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.349 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.349 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.349 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:01.349 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.544994+0000", 2024-04-19T01:05:01.349 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2399999999999997e-07 2024-04-19T01:05:01.349 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.349 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.349 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:01.349 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.545710+0000", 2024-04-19T01:05:01.349 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00071517300000000005 2024-04-19T01:05:01.349 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.349 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.349 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:01.349 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.545755+0000", 2024-04-19T01:05:01.350 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5105000000000003e-05 2024-04-19T01:05:01.350 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.350 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.350 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:01.350 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.545780+0000", 2024-04-19T01:05:01.350 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5052999999999999e-05 2024-04-19T01:05:01.350 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.350 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.350 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:01.350 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.546377+0000", 2024-04-19T01:05:01.350 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000597779 2024-04-19T01:05:01.350 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.350 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.351 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:01.351 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.546405+0000", 2024-04-19T01:05:01.351 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7809000000000001e-05 2024-04-19T01:05:01.351 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.351 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.351 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:01.351 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.587263+0000", 2024-04-19T01:05:01.351 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.040857841999999998 2024-04-19T01:05:01.351 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.351 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.351 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:01.352 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.587323+0000", 2024-04-19T01:05:01.352 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9966999999999997e-05 2024-04-19T01:05:01.352 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.352 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:01.352 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.352 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.352 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.352 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 59/31 ECSubWrite(tid=67, reqid=client.4604.0:424, at_version=59'52, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:05:01.352 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.549026+0000", 2024-04-19T01:05:01.352 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 31.788724906999999, 2024-04-19T01:05:01.352 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.038380015000000003, 2024-04-19T01:05:01.352 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:01.352 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:01.352 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:01.352 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:01.353 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.353 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:01.353 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549026+0000", 2024-04-19T01:05:01.353 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.353 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.353 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.353 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:01.353 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549026+0000", 2024-04-19T01:05:01.353 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.354 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.354 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.354 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:01.354 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549029+0000", 2024-04-19T01:05:01.354 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7269999999999999e-06 2024-04-19T01:05:01.354 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.354 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.355 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:01.355 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549687+0000", 2024-04-19T01:05:01.355 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00065801800000000001 2024-04-19T01:05:01.355 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.355 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.355 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:01.355 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549722+0000", 2024-04-19T01:05:01.355 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4882000000000002e-05 2024-04-19T01:05:01.355 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.355 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.355 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:01.356 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549745+0000", 2024-04-19T01:05:01.356 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2895999999999999e-05 2024-04-19T01:05:01.356 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.356 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.356 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:01.356 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.568051+0000", 2024-04-19T01:05:01.356 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018306368999999999 2024-04-19T01:05:01.356 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.356 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.356 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:01.356 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.568076+0000", 2024-04-19T01:05:01.357 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.493e-05 2024-04-19T01:05:01.357 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.357 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.357 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:01.357 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.587351+0000", 2024-04-19T01:05:01.357 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019274593999999999 2024-04-19T01:05:01.357 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.357 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.357 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:01.357 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.587406+0000", 2024-04-19T01:05:01.357 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5599000000000001e-05 2024-04-19T01:05:01.357 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.357 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:01.357 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.358 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.358 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.358 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:518 3.4s0 3:232ae717:::smithi15528762-15:head [copy-from ver 52] snapc 16=[16,15,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-04-19T01:05:01.358 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:41.513636+0000", 2024-04-19T01:05:01.358 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 19.824115044999999, 2024-04-19T01:05:01.358 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.099118577999999999, 2024-04-19T01:05:01.358 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:01.358 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:01.358 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:01.358 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:05:01.358 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:05:01.358 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:01.359 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 518 2024-04-19T01:05:01.359 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.359 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:01.359 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.359 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:01.359 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513636+0000", 2024-04-19T01:05:01.359 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.359 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.359 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.359 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:01.359 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513636+0000", 2024-04-19T01:05:01.359 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.359 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.359 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.360 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:01.360 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513639+0000", 2024-04-19T01:05:01.360 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1240000000000001e-06 2024-04-19T01:05:01.360 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.360 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.360 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:01.360 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513655+0000", 2024-04-19T01:05:01.360 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6143000000000001e-05 2024-04-19T01:05:01.360 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.360 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.360 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:01.360 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513688+0000", 2024-04-19T01:05:01.360 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2396e-05 2024-04-19T01:05:01.360 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.361 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.361 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:01.361 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513705+0000", 2024-04-19T01:05:01.361 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7295999999999998e-05 2024-04-19T01:05:01.361 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.361 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.361 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:01.361 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513854+0000", 2024-04-19T01:05:01.361 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014896700000000001 2024-04-19T01:05:01.361 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.361 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.361 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op must wait for map", 2024-04-19T01:05:01.361 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513868+0000", 2024-04-19T01:05:01.361 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3604000000000001e-05 2024-04-19T01:05:01.361 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.361 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.362 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:01.362 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.519112+0000", 2024-04-19T01:05:01.362 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005243808 2024-04-19T01:05:01.362 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.362 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.362 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:05:01.362 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.519314+0000", 2024-04-19T01:05:01.362 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020256300000000001 2024-04-19T01:05:01.362 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.362 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.362 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:01.362 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.543242+0000", 2024-04-19T01:05:01.362 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023928362000000002 2024-04-19T01:05:01.362 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.363 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.363 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:01.363 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.591560+0000", 2024-04-19T01:05:01.363 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.048317225999999998 2024-04-19T01:05:01.363 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.363 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.363 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-04-19T01:05:01.363 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.612629+0000", 2024-04-19T01:05:01.363 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.021069623999999999 2024-04-19T01:05:01.363 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.363 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.363 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:01.363 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.612755+0000", 2024-04-19T01:05:01.363 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012546500000000001 2024-04-19T01:05:01.364 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.364 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:01.364 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.364 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.364 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.364 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:519 3.4s0 3:232ae717:::smithi15528762-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e71)", 2024-04-19T01:05:01.364 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:41.513738+0000", 2024-04-19T01:05:01.364 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 19.824012916000001, 2024-04-19T01:05:01.364 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.077425087000000004, 2024-04-19T01:05:01.364 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:01.364 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:01.364 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-04-19T01:05:01.364 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:05:01.364 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:05:01.364 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:01.365 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 519 2024-04-19T01:05:01.365 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.365 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:01.365 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.365 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:01.365 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513738+0000", 2024-04-19T01:05:01.365 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.365 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.365 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.365 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:01.365 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513738+0000", 2024-04-19T01:05:01.365 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.365 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.366 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.366 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:01.366 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513740+0000", 2024-04-19T01:05:01.366 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2300000000000001e-06 2024-04-19T01:05:01.366 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.366 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.366 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:01.366 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513746+0000", 2024-04-19T01:05:01.366 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5660000000000003e-06 2024-04-19T01:05:01.366 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.366 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.366 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:01.366 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513767+0000", 2024-04-19T01:05:01.367 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0990000000000001e-05 2024-04-19T01:05:01.367 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.367 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.367 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:01.367 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513779+0000", 2024-04-19T01:05:01.367 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2033e-05 2024-04-19T01:05:01.367 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.367 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.367 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:01.367 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513972+0000", 2024-04-19T01:05:01.367 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019274599999999999 2024-04-19T01:05:01.367 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.367 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.367 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:05:01.367 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513984+0000", 2024-04-19T01:05:01.368 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.165e-05 2024-04-19T01:05:01.368 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.368 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.368 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:01.368 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.519641+0000", 2024-04-19T01:05:01.368 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056575640000000003 2024-04-19T01:05:01.368 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.368 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.368 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for blocked object", 2024-04-19T01:05:01.368 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.519731+0000", 2024-04-19T01:05:01.368 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9462000000000005e-05 2024-04-19T01:05:01.368 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.368 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.368 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:01.368 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.590802+0000", 2024-04-19T01:05:01.369 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.071071239999999994 2024-04-19T01:05:01.369 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.369 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.369 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:05:01.369 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.590982+0000", 2024-04-19T01:05:01.369 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017997 2024-04-19T01:05:01.369 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.369 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.369 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:01.369 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.591163+0000", 2024-04-19T01:05:01.369 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018163599999999999 2024-04-19T01:05:01.369 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.369 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:01.369 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.370 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.370 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.370 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 79/31 ECSubWrite(tid=115, reqid=client.4604.0:591, at_version=79'90, trim_to=0'0, roll_forward_to=79'86))", 2024-04-19T01:05:01.370 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:49.657669+0000", 2024-04-19T01:05:01.370 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.680082334, 2024-04-19T01:05:01.370 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.041645041000000001, 2024-04-19T01:05:01.370 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:01.370 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:01.370 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:01.371 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:01.371 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.371 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:01.371 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.657669+0000", 2024-04-19T01:05:01.371 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.371 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.371 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.371 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:01.371 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.657669+0000", 2024-04-19T01:05:01.371 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.371 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.371 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.371 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:01.371 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.657670+0000", 2024-04-19T01:05:01.371 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.424e-06 2024-04-19T01:05:01.372 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.372 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.372 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:01.372 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658272+0000", 2024-04-19T01:05:01.372 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00060176100000000003 2024-04-19T01:05:01.372 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.382 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.382 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:01.382 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658307+0000", 2024-04-19T01:05:01.382 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4820999999999997e-05 2024-04-19T01:05:01.382 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.382 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.382 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:01.382 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658331+0000", 2024-04-19T01:05:01.382 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3898999999999999e-05 2024-04-19T01:05:01.382 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.382 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.383 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:01.383 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658490+0000", 2024-04-19T01:05:01.383 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015878499999999999 2024-04-19T01:05:01.383 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.383 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.383 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:01.383 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658519+0000", 2024-04-19T01:05:01.383 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9720999999999999e-05 2024-04-19T01:05:01.383 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.383 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.383 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:01.383 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.699239+0000", 2024-04-19T01:05:01.383 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.040719568999999997 2024-04-19T01:05:01.383 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.384 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.384 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:01.384 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.699314+0000", 2024-04-19T01:05:01.384 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5061000000000005e-05 2024-04-19T01:05:01.384 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.384 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:01.384 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.384 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.384 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.384 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:605 3.4s0 3:219bff74:::smithi15528762-16:head [append 0~671744] snapc 1a=[19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-19T01:05:01.384 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:49.674586+0000", 2024-04-19T01:05:01.384 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.663165679, 2024-04-19T01:05:01.384 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.041485808999999998, 2024-04-19T01:05:01.384 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:01.384 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:01.385 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:01.385 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:05:01.385 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:05:01.385 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:01.385 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 605 2024-04-19T01:05:01.385 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.385 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:01.385 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.385 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:01.385 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.674586+0000", 2024-04-19T01:05:01.385 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.385 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.385 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.385 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:01.386 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.674586+0000", 2024-04-19T01:05:01.386 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.386 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.386 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.386 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:01.386 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.674588+0000", 2024-04-19T01:05:01.386 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7779999999999999e-06 2024-04-19T01:05:01.386 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.386 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.386 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:01.386 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.676436+0000", 2024-04-19T01:05:01.387 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0018479060000000001 2024-04-19T01:05:01.387 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.387 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.387 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:01.387 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.676485+0000", 2024-04-19T01:05:01.387 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8797000000000001e-05 2024-04-19T01:05:01.387 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.387 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.387 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:01.387 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.676509+0000", 2024-04-19T01:05:01.387 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3811999999999998e-05 2024-04-19T01:05:01.388 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.388 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.388 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:01.388 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.677380+0000", 2024-04-19T01:05:01.388 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00087160999999999998 2024-04-19T01:05:01.388 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.388 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.388 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:05:01.388 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.677579+0000", 2024-04-19T01:05:01.388 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000198402 2024-04-19T01:05:01.388 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.388 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.388 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:01.388 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.680409+0000", 2024-04-19T01:05:01.388 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0028303960000000002 2024-04-19T01:05:01.389 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.389 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.389 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:01.389 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.711082+0000", 2024-04-19T01:05:01.389 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.030672228999999999 2024-04-19T01:05:01.389 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.389 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.389 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-04-19T01:05:01.389 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.715856+0000", 2024-04-19T01:05:01.389 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0047740589999999998 2024-04-19T01:05:01.389 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.389 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.389 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:01.389 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.716071+0000", 2024-04-19T01:05:01.389 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021582 2024-04-19T01:05:01.390 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.390 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:01.390 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.390 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.390 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.390 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:606 3.4s0 3:219bff74:::smithi15528762-16:head [append 0~753664] snapc 1a=[19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-19T01:05:01.390 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:49.682434+0000", 2024-04-19T01:05:01.390 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.655317673999999, 2024-04-19T01:05:01.390 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.037255644999999997, 2024-04-19T01:05:01.390 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:01.390 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:01.390 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:01.390 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:05:01.390 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:05:01.391 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:01.391 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 606 2024-04-19T01:05:01.391 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.391 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:01.391 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.391 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:01.391 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.682434+0000", 2024-04-19T01:05:01.391 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.391 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.391 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.391 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:01.391 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.682434+0000", 2024-04-19T01:05:01.391 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.391 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.391 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.392 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:01.392 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.682441+0000", 2024-04-19T01:05:01.392 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5989999999999996e-06 2024-04-19T01:05:01.392 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.392 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.392 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:01.392 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.683247+0000", 2024-04-19T01:05:01.392 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00080545400000000004 2024-04-19T01:05:01.392 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.392 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.392 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:01.392 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.683274+0000", 2024-04-19T01:05:01.392 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7827999999999999e-05 2024-04-19T01:05:01.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.393 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.393 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:01.393 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.683292+0000", 2024-04-19T01:05:01.393 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7958999999999999e-05 2024-04-19T01:05:01.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.393 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.393 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:01.393 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.697174+0000", 2024-04-19T01:05:01.393 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013881881 2024-04-19T01:05:01.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.393 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.394 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:05:01.394 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.697346+0000", 2024-04-19T01:05:01.394 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000171898 2024-04-19T01:05:01.394 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.394 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.395 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:01.395 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.700021+0000", 2024-04-19T01:05:01.395 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0026752099999999999 2024-04-19T01:05:01.395 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.395 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.395 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:01.395 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.715555+0000", 2024-04-19T01:05:01.395 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.01553331 2024-04-19T01:05:01.395 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.395 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.395 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-04-19T01:05:01.395 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.719523+0000", 2024-04-19T01:05:01.395 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003968465 2024-04-19T01:05:01.395 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.396 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.396 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:01.396 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.719689+0000", 2024-04-19T01:05:01.396 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016604099999999999 2024-04-19T01:05:01.396 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.396 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:01.396 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.396 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.396 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.396 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:609 3.4s0 3:219bff74:::smithi15528762-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e79)", 2024-04-19T01:05:01.396 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:49.689050+0000", 2024-04-19T01:05:01.396 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.648701163, 2024-04-19T01:05:01.397 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.033842161000000003, 2024-04-19T01:05:01.397 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:01.397 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:01.397 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-04-19T01:05:01.397 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:05:01.397 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:05:01.397 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:01.397 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 609 2024-04-19T01:05:01.397 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.397 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:01.397 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.397 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:01.397 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.689050+0000", 2024-04-19T01:05:01.398 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.398 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.398 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.398 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:01.398 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.689050+0000", 2024-04-19T01:05:01.398 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.398 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.398 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.398 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:01.399 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.689051+0000", 2024-04-19T01:05:01.399 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2270000000000001e-06 2024-04-19T01:05:01.399 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.399 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.399 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:01.399 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.689056+0000", 2024-04-19T01:05:01.399 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9060000000000001e-06 2024-04-19T01:05:01.399 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.399 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.399 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:01.399 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.689072+0000", 2024-04-19T01:05:01.399 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6033999999999999e-05 2024-04-19T01:05:01.399 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.399 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.399 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:01.400 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.689084+0000", 2024-04-19T01:05:01.400 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2004000000000001e-05 2024-04-19T01:05:01.400 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.400 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.400 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:01.400 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.718329+0000", 2024-04-19T01:05:01.400 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.029244405000000001 2024-04-19T01:05:01.400 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.400 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.400 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-04-19T01:05:01.400 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.718558+0000", 2024-04-19T01:05:01.400 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000228874 2024-04-19T01:05:01.400 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.400 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.401 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:01.401 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.721092+0000", 2024-04-19T01:05:01.401 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002534503 2024-04-19T01:05:01.401 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.401 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.401 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:05:01.402 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.721251+0000", 2024-04-19T01:05:01.402 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0001594 2024-04-19T01:05:01.402 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.402 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.402 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:01.402 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.722892+0000", 2024-04-19T01:05:01.402 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0016408079999999999 2024-04-19T01:05:01.402 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.403 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:01.403 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.403 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.403 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.403 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:618 3.4s0 3:24fe125c:::smithi15528762-3:head [append 0~720896] snapc 1c=[1c,1b,19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-19T01:05:01.403 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:51.527932+0000", 2024-04-19T01:05:01.403 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 9.809819354, 2024-04-19T01:05:01.403 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.045645045000000002, 2024-04-19T01:05:01.403 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:01.403 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:01.403 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:01.403 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:05:01.403 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:05:01.404 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:01.404 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 618 2024-04-19T01:05:01.404 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.404 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:01.404 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.404 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:01.404 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.527932+0000", 2024-04-19T01:05:01.404 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.404 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.404 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.404 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:01.404 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.527932+0000", 2024-04-19T01:05:01.404 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.404 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.405 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.405 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:01.405 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.527935+0000", 2024-04-19T01:05:01.405 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.32e-06 2024-04-19T01:05:01.405 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.405 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.405 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:01.405 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.529172+0000", 2024-04-19T01:05:01.405 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0012371229999999999 2024-04-19T01:05:01.405 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.405 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.405 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:01.405 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.529205+0000", 2024-04-19T01:05:01.405 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3148999999999997e-05 2024-04-19T01:05:01.405 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.406 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.406 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:01.406 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.529229+0000", 2024-04-19T01:05:01.406 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3118000000000001e-05 2024-04-19T01:05:01.406 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.406 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.406 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:01.406 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.529364+0000", 2024-04-19T01:05:01.406 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000135833 2024-04-19T01:05:01.406 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.406 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.406 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op must wait for map", 2024-04-19T01:05:01.407 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.529378+0000", 2024-04-19T01:05:01.407 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3349999999999999e-05 2024-04-19T01:05:01.407 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.407 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.407 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:01.407 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.534487+0000", 2024-04-19T01:05:01.407 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0051091069999999999 2024-04-19T01:05:01.407 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.407 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.407 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:05:01.407 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.534645+0000", 2024-04-19T01:05:01.407 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000158579 2024-04-19T01:05:01.407 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.407 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.407 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:01.408 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.536865+0000", 2024-04-19T01:05:01.408 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022194939999999998 2024-04-19T01:05:01.408 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.408 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.408 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:01.408 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.572978+0000", 2024-04-19T01:05:01.408 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.036112559000000002 2024-04-19T01:05:01.408 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.408 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.408 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-04-19T01:05:01.408 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.573320+0000", 2024-04-19T01:05:01.408 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00034209900000000001 2024-04-19T01:05:01.408 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.408 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.408 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:01.409 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.573577+0000", 2024-04-19T01:05:01.409 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00025731400000000002 2024-04-19T01:05:01.409 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.409 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:01.409 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.409 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.409 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.409 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:619 3.4s0 3:24fe125c:::smithi15528762-3:head [append 0~802816] snapc 1c=[1c,1b,19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-19T01:05:01.409 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:51.540275+0000", 2024-04-19T01:05:01.409 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 9.7974758340000001, 2024-04-19T01:05:01.409 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.034131375999999998, 2024-04-19T01:05:01.409 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:01.409 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:01.409 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:01.409 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:05:01.410 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:05:01.410 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:01.410 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 619 2024-04-19T01:05:01.410 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.410 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:01.410 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.410 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:01.410 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.540275+0000", 2024-04-19T01:05:01.410 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.410 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.410 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.410 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:01.410 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.540275+0000", 2024-04-19T01:05:01.410 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.411 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.411 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.411 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:01.411 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.540278+0000", 2024-04-19T01:05:01.411 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7300000000000001e-06 2024-04-19T01:05:01.411 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.411 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.411 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:01.411 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.541861+0000", 2024-04-19T01:05:01.411 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0015829939999999999 2024-04-19T01:05:01.414 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.417 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.417 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:01.417 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.541909+0000", 2024-04-19T01:05:01.417 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8016999999999998e-05 2024-04-19T01:05:01.417 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.417 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.417 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:01.417 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.541932+0000", 2024-04-19T01:05:01.417 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3125e-05 2024-04-19T01:05:01.417 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.418 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.418 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:01.418 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.554751+0000", 2024-04-19T01:05:01.418 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012818942999999999 2024-04-19T01:05:01.418 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.418 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.418 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:05:01.418 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.555019+0000", 2024-04-19T01:05:01.418 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026781 2024-04-19T01:05:01.418 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.419 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.419 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:01.419 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.557960+0000", 2024-04-19T01:05:01.419 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0029404679999999999 2024-04-19T01:05:01.419 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.419 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.419 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:01.419 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.573846+0000", 2024-04-19T01:05:01.419 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015886324 2024-04-19T01:05:01.419 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.419 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.419 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-04-19T01:05:01.419 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.574282+0000", 2024-04-19T01:05:01.420 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00043590099999999999 2024-04-19T01:05:01.420 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.420 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.421 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:01.421 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.574407+0000", 2024-04-19T01:05:01.421 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012506400000000001 2024-04-19T01:05:01.421 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.421 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:01.421 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.421 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.421 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.421 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:620 3.4s0 3:24fe125c:::smithi15528762-3:head [append 0~360448] snapc 1c=[1c,1b,19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-19T01:05:01.421 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:51.547149+0000", 2024-04-19T01:05:01.421 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 9.7906019519999994, 2024-04-19T01:05:01.422 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.040862640999999998, 2024-04-19T01:05:01.422 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:01.422 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:01.422 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:01.422 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:05:01.422 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:05:01.422 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:01.422 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 620 2024-04-19T01:05:01.422 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.423 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:01.423 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.423 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:01.423 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.547149+0000", 2024-04-19T01:05:01.423 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.423 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.423 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.424 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:01.424 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.547149+0000", 2024-04-19T01:05:01.424 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.424 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.424 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.424 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:01.424 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.547152+0000", 2024-04-19T01:05:01.424 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0790000000000001e-06 2024-04-19T01:05:01.424 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.424 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.424 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:01.424 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.547839+0000", 2024-04-19T01:05:01.424 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00068704599999999997 2024-04-19T01:05:01.424 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.425 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.425 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:01.425 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.547872+0000", 2024-04-19T01:05:01.425 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3045e-05 2024-04-19T01:05:01.425 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.425 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.425 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:01.425 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.547894+0000", 2024-04-19T01:05:01.426 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1515999999999999e-05 2024-04-19T01:05:01.426 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.426 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.426 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:01.426 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.575146+0000", 2024-04-19T01:05:01.426 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.027252327 2024-04-19T01:05:01.426 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.426 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.426 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:05:01.426 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.575446+0000", 2024-04-19T01:05:01.426 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029948799999999999 2024-04-19T01:05:01.426 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.426 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.427 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:01.427 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.576902+0000", 2024-04-19T01:05:01.427 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0014561730000000001 2024-04-19T01:05:01.427 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.427 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.427 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:01.427 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.585278+0000", 2024-04-19T01:05:01.427 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0083762349999999992 2024-04-19T01:05:01.428 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.428 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.428 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-04-19T01:05:01.428 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.587851+0000", 2024-04-19T01:05:01.428 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025729759999999998 2024-04-19T01:05:01.428 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.428 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.428 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:01.428 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.588012+0000", 2024-04-19T01:05:01.428 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016075599999999999 2024-04-19T01:05:01.428 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.428 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:01.429 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.429 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.429 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.429 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:621 3.4s0 3:24fe125c:::smithi15528762-3:head [setxattr _header (56) in=7b] snapc 1c=[1c,1b,19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-19T01:05:01.429 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:51.547930+0000", 2024-04-19T01:05:01.429 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 9.7898209919999992, 2024-04-19T01:05:01.429 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.043519543000000001, 2024-04-19T01:05:01.429 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:01.429 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:01.429 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:01.429 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:05:01.429 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:05:01.429 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:01.429 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 621 2024-04-19T01:05:01.430 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.430 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:01.430 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.430 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:01.430 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.547930+0000", 2024-04-19T01:05:01.430 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.430 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.431 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.431 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:01.431 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.547930+0000", 2024-04-19T01:05:01.431 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.431 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.431 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.431 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:01.431 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.547932+0000", 2024-04-19T01:05:01.431 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.894e-06 2024-04-19T01:05:01.432 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.432 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.432 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:01.432 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.547942+0000", 2024-04-19T01:05:01.432 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0332e-05 2024-04-19T01:05:01.432 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.432 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.432 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:01.432 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.547964+0000", 2024-04-19T01:05:01.432 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1268999999999999e-05 2024-04-19T01:05:01.432 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.434 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.434 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:01.434 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.547979+0000", 2024-04-19T01:05:01.434 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5722e-05 2024-04-19T01:05:01.434 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.434 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.434 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:01.434 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.583319+0000", 2024-04-19T01:05:01.434 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.035339684000000003 2024-04-19T01:05:01.434 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.434 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.435 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:05:01.435 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.583593+0000", 2024-04-19T01:05:01.435 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00027369299999999998 2024-04-19T01:05:01.435 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.435 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.436 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:01.436 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.584632+0000", 2024-04-19T01:05:01.436 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001039531 2024-04-19T01:05:01.436 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.436 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.436 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:01.436 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.585936+0000", 2024-04-19T01:05:01.436 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001303294 2024-04-19T01:05:01.436 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.436 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.436 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-04-19T01:05:01.436 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.591233+0000", 2024-04-19T01:05:01.436 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005296935 2024-04-19T01:05:01.437 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.437 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.437 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:01.437 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.591450+0000", 2024-04-19T01:05:01.437 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021718900000000001 2024-04-19T01:05:01.437 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.437 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:01.437 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.437 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.437 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.438 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:622 3.4s0 3:24fe125c:::smithi15528762-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e81)", 2024-04-19T01:05:01.438 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:51.548002+0000", 2024-04-19T01:05:01.438 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 9.7897490620000003, 2024-04-19T01:05:01.438 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.049427448999999998, 2024-04-19T01:05:01.438 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:01.438 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:01.438 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-04-19T01:05:01.438 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:05:01.438 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:05:01.438 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:01.439 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 622 2024-04-19T01:05:01.439 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.439 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:01.439 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.439 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:01.439 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.548002+0000", 2024-04-19T01:05:01.439 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.439 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.439 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.439 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:01.439 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.548002+0000", 2024-04-19T01:05:01.439 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.439 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.439 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.440 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:01.440 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.548004+0000", 2024-04-19T01:05:01.440 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.46e-06 2024-04-19T01:05:01.440 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.440 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.440 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:01.440 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.548010+0000", 2024-04-19T01:05:01.440 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7630000000000003e-06 2024-04-19T01:05:01.440 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.440 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.440 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:01.440 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.548037+0000", 2024-04-19T01:05:01.441 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6492e-05 2024-04-19T01:05:01.441 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.441 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.441 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:01.441 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.548053+0000", 2024-04-19T01:05:01.441 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6436e-05 2024-04-19T01:05:01.441 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.441 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.442 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:01.442 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.586552+0000", 2024-04-19T01:05:01.442 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.038498903000000001 2024-04-19T01:05:01.442 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.442 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.442 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-04-19T01:05:01.442 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.586701+0000", 2024-04-19T01:05:01.442 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014879300000000001 2024-04-19T01:05:01.443 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.443 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.443 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:01.443 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.592222+0000", 2024-04-19T01:05:01.443 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0055211629999999999 2024-04-19T01:05:01.443 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.443 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.443 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:05:01.443 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.592488+0000", 2024-04-19T01:05:01.444 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026554200000000001 2024-04-19T01:05:01.444 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.444 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.444 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:01.444 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.597430+0000", 2024-04-19T01:05:01.444 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0049418969999999998 2024-04-19T01:05:01.444 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.444 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:01.444 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.444 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.445 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.445 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 81/31 ECSubWrite(tid=122, reqid=client.4604.0:626, at_version=81'96, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:05:01.445 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:51.594389+0000", 2024-04-19T01:05:01.445 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 9.7433624529999996, 2024-04-19T01:05:01.445 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.064667589999999997, 2024-04-19T01:05:01.445 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:01.445 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:01.445 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:01.445 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:01.445 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.446 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:01.446 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.594389+0000", 2024-04-19T01:05:01.446 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.446 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.446 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.446 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:01.447 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.594389+0000", 2024-04-19T01:05:01.447 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.447 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.447 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.447 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:01.447 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.594389+0000", 2024-04-19T01:05:01.447 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0900000000000002e-07 2024-04-19T01:05:01.447 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.447 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.447 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:01.447 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.595258+0000", 2024-04-19T01:05:01.448 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000868705 2024-04-19T01:05:01.448 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.448 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.448 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:01.448 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.595294+0000", 2024-04-19T01:05:01.448 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5685000000000001e-05 2024-04-19T01:05:01.448 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.448 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.448 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:01.448 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.595317+0000", 2024-04-19T01:05:01.448 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3527999999999999e-05 2024-04-19T01:05:01.448 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.449 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.449 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:01.449 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.595472+0000", 2024-04-19T01:05:01.449 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015491300000000001 2024-04-19T01:05:01.449 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.449 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.449 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:01.449 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.595497+0000", 2024-04-19T01:05:01.449 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4510000000000001e-05 2024-04-19T01:05:01.449 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.449 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.449 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:01.449 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.658996+0000", 2024-04-19T01:05:01.449 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.063499261000000001 2024-04-19T01:05:01.450 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.450 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.450 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:01.450 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.659056+0000", 2024-04-19T01:05:01.450 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0578999999999997e-05 2024-04-19T01:05:01.450 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.450 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:01.450 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.450 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.450 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.450 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 81/31 ECSubWrite(tid=123, reqid=client.4604.0:627, at_version=81'97, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:05:01.450 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:51.601189+0000", 2024-04-19T01:05:01.450 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 9.7365626340000002, 2024-04-19T01:05:01.450 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.057910206999999998, 2024-04-19T01:05:01.451 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:01.451 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:01.451 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:01.451 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:01.451 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.451 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:01.451 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.601189+0000", 2024-04-19T01:05:01.452 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.452 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.452 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.452 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:01.452 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.601189+0000", 2024-04-19T01:05:01.452 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.452 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.452 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.453 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:01.453 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.601191+0000", 2024-04-19T01:05:01.453 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0200000000000001e-06 2024-04-19T01:05:01.453 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.453 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.453 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:01.453 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.601493+0000", 2024-04-19T01:05:01.453 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000302625 2024-04-19T01:05:01.453 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.453 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.453 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:01.454 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.601543+0000", 2024-04-19T01:05:01.454 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0016999999999999e-05 2024-04-19T01:05:01.454 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.454 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.454 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:01.455 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.601578+0000", 2024-04-19T01:05:01.455 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5079000000000001e-05 2024-04-19T01:05:01.455 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.455 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.455 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:01.455 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.609356+0000", 2024-04-19T01:05:01.455 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0077777990000000002 2024-04-19T01:05:01.455 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.455 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.455 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:01.455 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.609389+0000", 2024-04-19T01:05:01.455 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2662e-05 2024-04-19T01:05:01.455 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.455 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.456 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:01.456 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.659064+0000", 2024-04-19T01:05:01.456 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.049674836999999999 2024-04-19T01:05:01.456 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.456 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.458 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:01.458 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.659099+0000", 2024-04-19T01:05:01.458 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5167999999999999e-05 2024-04-19T01:05:01.458 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.458 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:01.458 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.458 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.458 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.458 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 81/31 ECSubWrite(tid=124, reqid=client.4604.0:628, at_version=81'98, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:05:01.458 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:51.606908+0000", 2024-04-19T01:05:01.459 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 9.73084302, 2024-04-19T01:05:01.459 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.052296042000000001, 2024-04-19T01:05:01.459 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:01.459 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:01.459 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:01.459 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:01.459 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.459 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:01.459 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.606908+0000", 2024-04-19T01:05:01.459 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.460 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.460 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.460 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:01.460 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.606908+0000", 2024-04-19T01:05:01.460 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.460 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.460 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.460 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:01.460 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.606910+0000", 2024-04-19T01:05:01.461 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1229999999999998e-06 2024-04-19T01:05:01.461 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.461 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.461 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:01.461 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.607105+0000", 2024-04-19T01:05:01.461 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000194293 2024-04-19T01:05:01.461 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.461 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.461 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:01.461 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.607144+0000", 2024-04-19T01:05:01.462 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9415000000000001e-05 2024-04-19T01:05:01.462 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.462 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.462 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:01.462 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.607170+0000", 2024-04-19T01:05:01.462 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5819000000000001e-05 2024-04-19T01:05:01.463 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.463 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.463 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:01.463 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.621613+0000", 2024-04-19T01:05:01.463 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014443123 2024-04-19T01:05:01.463 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.463 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.463 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:01.463 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.621631+0000", 2024-04-19T01:05:01.464 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8343e-05 2024-04-19T01:05:01.464 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.464 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.464 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:01.464 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.659115+0000", 2024-04-19T01:05:01.464 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.037483797999999999 2024-04-19T01:05:01.464 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.465 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.465 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:01.465 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.659204+0000", 2024-04-19T01:05:01.465 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9128000000000005e-05 2024-04-19T01:05:01.465 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.465 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:01.465 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.465 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.466 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.466 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 81/31 ECSubWrite(tid=125, reqid=client.4604.0:629, at_version=81'99, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:05:01.466 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:51.610690+0000", 2024-04-19T01:05:01.466 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 9.7270611850000002, 2024-04-19T01:05:01.466 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.048566986999999999, 2024-04-19T01:05:01.466 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:01.467 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:01.467 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:01.467 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:01.467 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.467 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:01.468 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.610690+0000", 2024-04-19T01:05:01.468 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.468 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.468 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.468 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:01.468 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.610690+0000", 2024-04-19T01:05:01.468 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.468 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.468 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.468 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:01.468 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.610692+0000", 2024-04-19T01:05:01.468 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9640000000000002e-06 2024-04-19T01:05:01.468 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.468 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.468 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:01.469 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.610737+0000", 2024-04-19T01:05:01.469 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5290999999999997e-05 2024-04-19T01:05:01.469 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.469 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.469 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:01.469 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.610780+0000", 2024-04-19T01:05:01.469 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2633999999999999e-05 2024-04-19T01:05:01.469 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.469 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.469 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:01.470 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.610803+0000", 2024-04-19T01:05:01.470 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2792000000000001e-05 2024-04-19T01:05:01.470 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.470 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.470 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:01.470 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.632193+0000", 2024-04-19T01:05:01.470 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.021390273000000001 2024-04-19T01:05:01.470 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.470 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.470 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:01.471 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.632216+0000", 2024-04-19T01:05:01.471 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3011e-05 2024-04-19T01:05:01.471 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.471 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.471 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:01.471 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.659221+0000", 2024-04-19T01:05:01.471 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.027004759 2024-04-19T01:05:01.471 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.471 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.471 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:01.471 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.659257+0000", 2024-04-19T01:05:01.471 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6263000000000002e-05 2024-04-19T01:05:01.471 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.472 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:01.472 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.472 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.472 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.472 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 81/31 ECSubWrite(tid=126, reqid=client.4604.0:631, at_version=81'101, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:05:01.472 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:51.620071+0000", 2024-04-19T01:05:01.472 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 9.7176805260000005, 2024-04-19T01:05:01.472 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.039719863000000001, 2024-04-19T01:05:01.472 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:01.472 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:01.472 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:01.472 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:01.472 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.473 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:01.473 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.620071+0000", 2024-04-19T01:05:01.473 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.473 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.473 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.473 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:01.473 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.620071+0000", 2024-04-19T01:05:01.473 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.473 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.473 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.473 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:01.473 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.620073+0000", 2024-04-19T01:05:01.473 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8419999999999999e-06 2024-04-19T01:05:01.473 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.473 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.474 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:01.474 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.620690+0000", 2024-04-19T01:05:01.474 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00061762799999999995 2024-04-19T01:05:01.474 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.474 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.474 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:01.474 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.620724+0000", 2024-04-19T01:05:01.474 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3422e-05 2024-04-19T01:05:01.474 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.474 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.474 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:01.474 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.620743+0000", 2024-04-19T01:05:01.474 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9405000000000001e-05 2024-04-19T01:05:01.475 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.475 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.475 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:01.475 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.632724+0000", 2024-04-19T01:05:01.475 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011981014999999999 2024-04-19T01:05:01.475 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.475 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.475 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:01.476 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.632750+0000", 2024-04-19T01:05:01.476 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6293000000000001e-05 2024-04-19T01:05:01.476 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.476 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.476 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:01.476 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.659730+0000", 2024-04-19T01:05:01.476 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.026979876999999999 2024-04-19T01:05:01.476 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.476 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.476 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:01.476 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.659791+0000", 2024-04-19T01:05:01.477 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0380999999999997e-05 2024-04-19T01:05:01.477 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.477 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:01.477 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.477 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.477 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.477 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:652 3.4s0 3:232ae717:::smithi15528762-15:head [delete] snapc 1d=[1d,1c,1b,14] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86)", 2024-04-19T01:05:01.478 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:56.523328+0000", 2024-04-19T01:05:01.478 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.8144236669999998, 2024-04-19T01:05:01.478 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.033525841000000001, 2024-04-19T01:05:01.478 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:01.478 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:01.478 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:01.478 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:05:01.478 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:05:01.478 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:01.478 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 652 2024-04-19T01:05:01.478 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.478 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:01.478 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.478 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:01.479 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.523328+0000", 2024-04-19T01:05:01.479 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.479 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.479 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.479 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:01.479 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.523328+0000", 2024-04-19T01:05:01.479 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.479 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.479 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.479 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:01.479 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.523331+0000", 2024-04-19T01:05:01.479 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4230000000000001e-06 2024-04-19T01:05:01.479 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.479 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.480 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:01.480 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.523345+0000", 2024-04-19T01:05:01.480 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3886999999999999e-05 2024-04-19T01:05:01.480 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.480 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.480 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:01.480 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.523372+0000", 2024-04-19T01:05:01.480 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6991e-05 2024-04-19T01:05:01.480 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.480 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.480 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:01.480 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.523388+0000", 2024-04-19T01:05:01.480 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.632e-05 2024-04-19T01:05:01.480 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.480 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.481 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:01.481 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.523546+0000", 2024-04-19T01:05:01.481 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015807899999999999 2024-04-19T01:05:01.481 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.481 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.481 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op must wait for map", 2024-04-19T01:05:01.481 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.523566+0000", 2024-04-19T01:05:01.481 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0208999999999999e-05 2024-04-19T01:05:01.481 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.481 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.482 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:01.482 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.538203+0000", 2024-04-19T01:05:01.482 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014636929 2024-04-19T01:05:01.482 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.482 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.482 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:05:01.483 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.538529+0000", 2024-04-19T01:05:01.483 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00032558000000000001 2024-04-19T01:05:01.483 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.483 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.483 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:01.483 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.545616+0000", 2024-04-19T01:05:01.483 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0070871880000000003 2024-04-19T01:05:01.483 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.483 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.484 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:01.484 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.555376+0000", 2024-04-19T01:05:01.484 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0097596060000000005 2024-04-19T01:05:01.484 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.484 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.484 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-04-19T01:05:01.484 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.556606+0000", 2024-04-19T01:05:01.484 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0012300340000000001 2024-04-19T01:05:01.484 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.484 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.484 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:01.484 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.556853+0000", 2024-04-19T01:05:01.484 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00024759499999999998 2024-04-19T01:05:01.484 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.485 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:01.485 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.485 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.485 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:01.485 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:05:01.487 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-04-19T01:05:01.488 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:05:01.485+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:05:01.517 INFO:tasks.rados.rados.0.smithi155.stdout:283: done (0 left) 2024-04-19T01:05:01.517 INFO:tasks.rados.rados.0.smithi155.stdout:284: read oid 3 snap -1 2024-04-19T01:05:01.517 INFO:tasks.rados.rados.0.smithi155.stdout:284: expect (ObjNum 112 snap 27 seq_num 112) 2024-04-19T01:05:01.517 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 30 current snap is 30 2024-04-19T01:05:01.518 INFO:tasks.rados.rados.0.smithi155.stdout:285: seq_num 119 ranges {3702784=458752,4161536=507904,4669440=507904,5177344=458752,5636096=81920} 2024-04-19T01:05:01.579 INFO:tasks.rados.rados.0.smithi155.stdout:285: writing smithi15528762-30 from 3702784 to 4161536 tid 1 2024-04-19T01:05:01.584 INFO:tasks.rados.rados.0.smithi155.stdout:285: writing smithi15528762-30 from 4161536 to 4669440 tid 2 2024-04-19T01:05:01.588 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:05:01.588+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:05:01.589 INFO:tasks.rados.rados.0.smithi155.stdout:285: writing smithi15528762-30 from 4669440 to 5177344 tid 3 2024-04-19T01:05:01.594 INFO:tasks.rados.rados.0.smithi155.stdout:285: writing smithi15528762-30 from 5177344 to 5636096 tid 4 2024-04-19T01:05:01.595 INFO:tasks.rados.rados.0.smithi155.stdout:285: writing smithi15528762-30 from 5636096 to 5718016 tid 5 2024-04-19T01:05:01.595 INFO:tasks.rados.rados.0.smithi155.stdout:286: snap_create 2024-04-19T01:05:01.595 INFO:tasks.rados.rados.0.smithi155.stdout:284: expect (ObjNum 112 snap 27 seq_num 112) 2024-04-19T01:05:01.626 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-19T01:05:01.628 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:05:01.628 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-04-19T01:05:01.629 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-04-19T01:05:01.629 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:05:01.646 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-04-19T01:05:01.648 INFO:tasks.rados.rados.0.smithi155.stdout:285: finishing write tid 1 to smithi15528762-30 2024-04-19T01:05:01.648 INFO:tasks.rados.rados.0.smithi155.stdout:285: finishing write tid 2 to smithi15528762-30 2024-04-19T01:05:01.648 INFO:tasks.rados.rados.0.smithi155.stdout:285: finishing write tid 3 to smithi15528762-30 2024-04-19T01:05:01.648 INFO:tasks.rados.rados.0.smithi155.stdout:285: finishing write tid 4 to smithi15528762-30 2024-04-19T01:05:01.648 INFO:tasks.rados.rados.0.smithi155.stdout:285: finishing write tid 5 to smithi15528762-30 2024-04-19T01:05:01.648 INFO:tasks.rados.rados.0.smithi155.stdout:285: finishing write tid 6 to smithi15528762-30 2024-04-19T01:05:01.649 INFO:tasks.rados.rados.0.smithi155.stdout:285: finishing write tid 7 to smithi15528762-30 2024-04-19T01:05:01.649 INFO:tasks.rados.rados.0.smithi155.stdout:285: oid 30 updating version 0 to 89 2024-04-19T01:05:01.649 INFO:tasks.rados.rados.0.smithi155.stdout:285: oid 30 updating version 89 to 94 2024-04-19T01:05:01.649 INFO:tasks.rados.rados.0.smithi155.stdout:285: oid 30 version 94 is already newer than 92 2024-04-19T01:05:01.649 INFO:tasks.rados.rados.0.smithi155.stdout:285: oid 30 version 94 is already newer than 90 2024-04-19T01:05:01.649 INFO:tasks.rados.rados.0.smithi155.stdout:285: oid 30 version 94 is already newer than 93 2024-04-19T01:05:01.649 INFO:tasks.rados.rados.0.smithi155.stdout:285: oid 30 version 94 is already newer than 91 2024-04-19T01:05:01.650 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 30 v 94 (ObjNum 119 snap 30 seq_num 119) dirty exists 2024-04-19T01:05:01.650 INFO:tasks.rados.rados.0.smithi155.stdout:285: left oid 30 (ObjNum 119 snap 30 seq_num 119) 2024-04-19T01:05:01.695 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:05:01.695+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:05:01.769 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:05:01.769 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-04-19T01:05:01.769 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-04-19T01:05:01.769 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-04-19T01:05:01.769 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:05:01.783 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-04-19T01:05:01.827 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:05:01.827+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:05:01.891 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:05:01.888+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:05:01.957 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:05:01.957 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-04-19T01:05:01.957 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-04-19T01:05:01.957 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-04-19T01:05:01.957 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.957 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s2 44/31 ECSubWrite(tid=82, reqid=client.4604.0:280, at_version=44'50, trim_to=0'0, roll_forward_to=43'48))", 2024-04-19T01:05:01.957 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.882222+0000", 2024-04-19T01:05:01.957 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 47.070991020999998, 2024-04-19T01:05:01.958 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.035827673999999997, 2024-04-19T01:05:01.958 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:01.958 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:01.958 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:01.958 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:01.958 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.958 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:01.958 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882222+0000", 2024-04-19T01:05:01.958 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.958 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.958 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.958 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:01.958 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882222+0000", 2024-04-19T01:05:01.958 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.959 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.959 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.959 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:01.959 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882222+0000", 2024-04-19T01:05:01.959 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7999999999999995e-07 2024-04-19T01:05:01.959 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.959 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.959 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:01.959 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882709+0000", 2024-04-19T01:05:01.959 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00048730799999999999 2024-04-19T01:05:01.959 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.959 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.959 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:01.959 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882742+0000", 2024-04-19T01:05:01.960 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2428000000000002e-05 2024-04-19T01:05:01.960 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.960 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.960 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:01.960 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882765+0000", 2024-04-19T01:05:01.960 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2693000000000001e-05 2024-04-19T01:05:01.960 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.960 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.960 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:01.960 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.883062+0000", 2024-04-19T01:05:01.960 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029777100000000001 2024-04-19T01:05:01.960 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.960 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.961 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:01.961 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.883088+0000", 2024-04-19T01:05:01.961 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6018999999999999e-05 2024-04-19T01:05:01.961 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.961 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.961 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:01.961 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.917995+0000", 2024-04-19T01:05:01.961 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.034906988999999999 2024-04-19T01:05:01.961 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.961 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.961 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:01.962 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.918049+0000", 2024-04-19T01:05:01.962 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3885999999999999e-05 2024-04-19T01:05:01.962 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.962 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:01.962 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.964 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.964 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.964 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 59/31 ECSubWrite(tid=66, reqid=client.4604.0:423, at_version=59'51, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:05:01.964 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.541618+0000", 2024-04-19T01:05:01.964 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 32.411594938999997, 2024-04-19T01:05:01.964 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.040796200999999997, 2024-04-19T01:05:01.964 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:01.964 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:01.964 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:01.964 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:01.965 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.965 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:01.965 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.541618+0000", 2024-04-19T01:05:01.965 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.965 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.965 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.965 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:01.965 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.541618+0000", 2024-04-19T01:05:01.978 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.978 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.978 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.979 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:01.979 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.541618+0000", 2024-04-19T01:05:01.979 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9699999999999995e-07 2024-04-19T01:05:01.979 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.979 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.979 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:01.979 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.542215+0000", 2024-04-19T01:05:01.979 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00059629300000000002 2024-04-19T01:05:01.979 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.979 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.979 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:01.979 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.542247+0000", 2024-04-19T01:05:01.979 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2339000000000003e-05 2024-04-19T01:05:01.979 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.980 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.980 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:01.980 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.542272+0000", 2024-04-19T01:05:01.980 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4973e-05 2024-04-19T01:05:01.980 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.980 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.980 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:01.980 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.542445+0000", 2024-04-19T01:05:01.980 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017274599999999999 2024-04-19T01:05:01.980 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.980 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.980 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:01.980 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.542468+0000", 2024-04-19T01:05:01.980 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3689e-05 2024-04-19T01:05:01.981 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.981 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.981 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:01.981 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.582330+0000", 2024-04-19T01:05:01.981 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.039861946000000002 2024-04-19T01:05:01.981 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.981 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.981 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:01.981 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.582414+0000", 2024-04-19T01:05:01.981 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3517999999999996e-05 2024-04-19T01:05:01.981 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.981 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:01.981 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.981 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.981 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.987 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 59/31 ECSubWrite(tid=67, reqid=client.4604.0:424, at_version=59'52, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:05:01.987 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.549371+0000", 2024-04-19T01:05:01.987 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 32.403841116000002, 2024-04-19T01:05:01.987 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.033134352999999998, 2024-04-19T01:05:01.987 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:01.987 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:01.987 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:01.988 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:01.988 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.988 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:01.988 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549371+0000", 2024-04-19T01:05:01.988 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.990 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.990 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.990 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:01.990 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549371+0000", 2024-04-19T01:05:01.990 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.990 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.990 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.990 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:01.990 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549375+0000", 2024-04-19T01:05:01.991 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0869999999999998e-06 2024-04-19T01:05:01.991 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.991 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.991 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:01.991 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549729+0000", 2024-04-19T01:05:01.991 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000354225 2024-04-19T01:05:01.991 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.991 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.991 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:01.991 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549769+0000", 2024-04-19T01:05:01.991 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0488999999999998e-05 2024-04-19T01:05:01.991 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.991 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.991 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:01.991 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549793+0000", 2024-04-19T01:05:01.992 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3702000000000001e-05 2024-04-19T01:05:01.992 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.992 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.992 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:01.992 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.561556+0000", 2024-04-19T01:05:01.992 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011763018 2024-04-19T01:05:01.992 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.992 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.992 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:01.992 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.561583+0000", 2024-04-19T01:05:01.992 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7243e-05 2024-04-19T01:05:01.992 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.992 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.992 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:01.993 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.582445+0000", 2024-04-19T01:05:01.993 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020862084999999999 2024-04-19T01:05:01.993 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.993 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.993 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:01.993 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.582506+0000", 2024-04-19T01:05:01.993 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0504000000000002e-05 2024-04-19T01:05:01.993 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.993 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:01.993 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.993 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.993 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.993 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 59/31 ECSubWrite(tid=68, reqid=client.4604.0:425, at_version=59'53, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:05:01.993 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.556915+0000", 2024-04-19T01:05:01.993 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 32.396297984999997, 2024-04-19T01:05:01.994 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.027043356000000001, 2024-04-19T01:05:01.994 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:01.994 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:01.995 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:01.995 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:01.995 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.995 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:01.995 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556915+0000", 2024-04-19T01:05:01.995 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.995 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.995 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.995 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:01.995 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556915+0000", 2024-04-19T01:05:01.995 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:01.995 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.996 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.996 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:01.996 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556917+0000", 2024-04-19T01:05:01.996 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9769999999999999e-06 2024-04-19T01:05:01.996 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.996 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.996 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:01.996 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.557048+0000", 2024-04-19T01:05:01.996 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013188699999999999 2024-04-19T01:05:01.996 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.997 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.997 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:01.997 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.557079+0000", 2024-04-19T01:05:01.997 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0318999999999999e-05 2024-04-19T01:05:01.997 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.997 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.997 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:01.997 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.557139+0000", 2024-04-19T01:05:01.997 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0469e-05 2024-04-19T01:05:01.997 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.998 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.998 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:01.998 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.579036+0000", 2024-04-19T01:05:01.998 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.021896863999999999 2024-04-19T01:05:01.998 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.998 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.998 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:01.998 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.579065+0000", 2024-04-19T01:05:01.998 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8846000000000001e-05 2024-04-19T01:05:01.998 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.998 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.998 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:01.998 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.583896+0000", 2024-04-19T01:05:01.999 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0048308159999999999 2024-04-19T01:05:01.999 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.999 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.999 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:01.999 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.583958+0000", 2024-04-19T01:05:01.999 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2178e-05 2024-04-19T01:05:01.999 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.999 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:01.999 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:01.999 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:01.999 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:01.999 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.4s2 71/31 ECSubWrite(tid=89, reqid=client.4604.0:518, at_version=71'22, trim_to=0'0, roll_forward_to=52'20))", 2024-04-19T01:05:01.999 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:41.545110+0000", 2024-04-19T01:05:01.999 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.408102164999999, 2024-04-19T01:05:02.000 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.049869387000000001, 2024-04-19T01:05:02.000 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:02.000 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:02.002 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:02.002 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:02.002 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.002 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:02.002 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.545110+0000", 2024-04-19T01:05:02.002 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:02.002 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.002 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.003 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:02.003 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.545110+0000", 2024-04-19T01:05:02.003 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:02.003 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.003 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.003 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:02.003 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.545111+0000", 2024-04-19T01:05:02.003 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3399999999999999e-07 2024-04-19T01:05:02.004 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.004 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.004 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:02.004 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.546538+0000", 2024-04-19T01:05:02.004 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0014274330000000001 2024-04-19T01:05:02.004 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.004 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.004 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:02.004 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.546582+0000", 2024-04-19T01:05:02.005 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3155000000000003e-05 2024-04-19T01:05:02.005 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.005 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.005 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:02.005 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.546610+0000", 2024-04-19T01:05:02.005 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8807e-05 2024-04-19T01:05:02.005 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.005 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.006 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:02.006 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.546756+0000", 2024-04-19T01:05:02.006 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014540399999999999 2024-04-19T01:05:02.006 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.006 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.006 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:02.006 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.546778+0000", 2024-04-19T01:05:02.006 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1722999999999999e-05 2024-04-19T01:05:02.006 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.006 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.006 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:02.007 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.594901+0000", 2024-04-19T01:05:02.007 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.048123725999999999 2024-04-19T01:05:02.007 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.007 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.007 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:02.007 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.594980+0000", 2024-04-19T01:05:02.007 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8604999999999997e-05 2024-04-19T01:05:02.007 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:02.007 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:02.007 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:02.007 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.007 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.007 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 76/31 ECSubWrite(tid=97, reqid=client.4604.0:568, at_version=76'74, trim_to=0'0, roll_forward_to=74'72))", 2024-04-19T01:05:02.007 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:46.580144+0000", 2024-04-19T01:05:02.008 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.373068524000001, 2024-04-19T01:05:02.008 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.068404570999999997, 2024-04-19T01:05:02.009 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:02.009 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:02.009 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:02.009 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:02.009 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.009 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:02.009 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.580144+0000", 2024-04-19T01:05:02.009 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:02.009 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.010 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.010 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:02.010 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.580144+0000", 2024-04-19T01:05:02.010 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:02.010 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.010 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.010 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:02.010 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.580145+0000", 2024-04-19T01:05:02.010 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8599999999999998e-07 2024-04-19T01:05:02.011 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.011 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.011 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:02.011 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.580959+0000", 2024-04-19T01:05:02.011 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00081373299999999997 2024-04-19T01:05:02.011 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.011 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.011 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:02.011 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.581006+0000", 2024-04-19T01:05:02.011 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7481999999999997e-05 2024-04-19T01:05:02.011 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.012 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.012 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:02.012 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.581034+0000", 2024-04-19T01:05:02.012 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8293000000000002e-05 2024-04-19T01:05:02.012 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.012 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.012 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:02.012 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.581198+0000", 2024-04-19T01:05:02.012 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016329100000000001 2024-04-19T01:05:02.012 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.012 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.013 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:02.013 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.581223+0000", 2024-04-19T01:05:02.013 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5656e-05 2024-04-19T01:05:02.013 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.013 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.013 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:02.013 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.648474+0000", 2024-04-19T01:05:02.013 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.067250402000000001 2024-04-19T01:05:02.013 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.013 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.013 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:02.014 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.648549+0000", 2024-04-19T01:05:02.014 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5028000000000001e-05 2024-04-19T01:05:02.014 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:02.014 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:02.014 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:02.014 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.014 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.014 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 76/31 ECSubWrite(tid=98, reqid=client.4604.0:569, at_version=76'75, trim_to=0'0, roll_forward_to=74'72))", 2024-04-19T01:05:02.014 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:46.587805+0000", 2024-04-19T01:05:02.014 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.365407146000001, 2024-04-19T01:05:02.014 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.060794462000000001, 2024-04-19T01:05:02.014 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:02.014 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:02.015 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:02.015 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:02.015 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.015 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:02.015 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.587805+0000", 2024-04-19T01:05:02.015 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:02.016 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.016 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.016 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:02.016 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.587805+0000", 2024-04-19T01:05:02.016 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:02.016 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.016 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.016 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:02.016 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.587808+0000", 2024-04-19T01:05:02.016 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8399999999999999e-06 2024-04-19T01:05:02.016 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.016 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.016 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:02.017 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.588154+0000", 2024-04-19T01:05:02.017 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00034547400000000002 2024-04-19T01:05:02.017 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.017 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.017 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:02.017 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.588204+0000", 2024-04-19T01:05:02.017 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0482000000000002e-05 2024-04-19T01:05:02.017 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.017 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.017 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:02.017 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.588227+0000", 2024-04-19T01:05:02.017 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2339000000000001e-05 2024-04-19T01:05:02.017 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.017 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.018 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:02.018 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.596418+0000", 2024-04-19T01:05:02.018 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0081914320000000002 2024-04-19T01:05:02.018 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.018 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.018 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:02.018 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.596444+0000", 2024-04-19T01:05:02.018 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5953e-05 2024-04-19T01:05:02.018 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.018 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.018 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:02.020 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.648559+0000", 2024-04-19T01:05:02.020 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.052114797999999997 2024-04-19T01:05:02.020 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.020 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.020 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:02.020 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.648600+0000", 2024-04-19T01:05:02.021 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1143999999999998e-05 2024-04-19T01:05:02.021 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:02.021 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:02.021 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:02.021 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.021 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.021 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 76/31 ECSubWrite(tid=99, reqid=client.4604.0:570, at_version=76'76, trim_to=0'0, roll_forward_to=74'72))", 2024-04-19T01:05:02.021 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:46.594492+0000", 2024-04-19T01:05:02.021 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.358720755, 2024-04-19T01:05:02.022 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.054401448999999998, 2024-04-19T01:05:02.022 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:02.022 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:02.022 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:02.022 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:02.022 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.022 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:02.022 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.594492+0000", 2024-04-19T01:05:02.022 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:02.022 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.022 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.023 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:02.023 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.594492+0000", 2024-04-19T01:05:02.023 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:02.023 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.023 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.023 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:02.023 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.594495+0000", 2024-04-19T01:05:02.023 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6769999999999999e-06 2024-04-19T01:05:02.023 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.023 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.023 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:02.023 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.594850+0000", 2024-04-19T01:05:02.023 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000355247 2024-04-19T01:05:02.024 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.024 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.024 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:02.024 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.594886+0000", 2024-04-19T01:05:02.024 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6545999999999999e-05 2024-04-19T01:05:02.024 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.024 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.025 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:02.025 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.594911+0000", 2024-04-19T01:05:02.025 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4366e-05 2024-04-19T01:05:02.025 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.025 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.025 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:02.025 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.613951+0000", 2024-04-19T01:05:02.025 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019039865999999999 2024-04-19T01:05:02.025 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.025 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.026 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:02.026 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.613968+0000", 2024-04-19T01:05:02.026 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7825999999999998e-05 2024-04-19T01:05:02.026 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.026 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.026 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:02.026 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.648836+0000", 2024-04-19T01:05:02.027 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.034867926 2024-04-19T01:05:02.027 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.027 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.027 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:02.027 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.648893+0000", 2024-04-19T01:05:02.027 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6994999999999999e-05 2024-04-19T01:05:02.027 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:02.027 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:02.027 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:02.027 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.027 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.027 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 76/31 ECSubWrite(tid=100, reqid=client.4604.0:571, at_version=76'77, trim_to=0'0, roll_forward_to=74'72))", 2024-04-19T01:05:02.027 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:46.600988+0000", 2024-04-19T01:05:02.027 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.352224707, 2024-04-19T01:05:02.028 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.047952294999999999, 2024-04-19T01:05:02.028 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:02.028 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:02.028 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:02.028 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:02.028 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.028 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:02.028 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.600988+0000", 2024-04-19T01:05:02.028 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:02.028 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.028 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.028 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:02.029 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.600988+0000", 2024-04-19T01:05:02.029 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:02.029 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.029 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.029 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:02.029 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.600990+0000", 2024-04-19T01:05:02.029 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3439999999999999e-06 2024-04-19T01:05:02.029 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.029 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.029 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:02.029 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.601037+0000", 2024-04-19T01:05:02.030 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6548999999999998e-05 2024-04-19T01:05:02.030 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.030 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.040 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:02.041 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.601082+0000", 2024-04-19T01:05:02.041 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4978999999999997e-05 2024-04-19T01:05:02.041 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.041 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.041 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:02.041 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.601105+0000", 2024-04-19T01:05:02.041 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3519999999999998e-05 2024-04-19T01:05:02.041 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.041 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.041 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:02.042 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.621093+0000", 2024-04-19T01:05:02.042 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0199878 2024-04-19T01:05:02.042 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.042 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.042 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:02.043 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.621110+0000", 2024-04-19T01:05:02.044 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7316000000000002e-05 2024-04-19T01:05:02.044 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.044 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.044 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:02.044 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.648899+0000", 2024-04-19T01:05:02.044 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.027788891 2024-04-19T01:05:02.044 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.044 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.044 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:02.044 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.648940+0000", 2024-04-19T01:05:02.044 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0896000000000003e-05 2024-04-19T01:05:02.044 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:02.044 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:02.044 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:02.045 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.045 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.045 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 76/31 ECSubWrite(tid=101, reqid=client.4604.0:573, at_version=76'79, trim_to=0'0, roll_forward_to=74'72))", 2024-04-19T01:05:02.045 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:46.612179+0000", 2024-04-19T01:05:02.045 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.341033364999999, 2024-04-19T01:05:02.045 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.036832701000000002, 2024-04-19T01:05:02.045 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:02.045 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:02.045 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:02.046 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:02.046 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.046 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:02.046 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.612179+0000", 2024-04-19T01:05:02.046 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:02.046 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.046 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.046 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:02.046 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.612179+0000", 2024-04-19T01:05:02.047 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:02.047 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.048 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.048 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:02.048 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.612181+0000", 2024-04-19T01:05:02.048 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8759999999999999e-06 2024-04-19T01:05:02.048 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.048 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.048 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:02.048 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.612643+0000", 2024-04-19T01:05:02.049 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00046178400000000002 2024-04-19T01:05:02.049 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.049 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.049 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:02.049 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.612676+0000", 2024-04-19T01:05:02.049 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3339e-05 2024-04-19T01:05:02.049 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.049 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.049 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:02.049 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.612694+0000", 2024-04-19T01:05:02.049 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7655999999999998e-05 2024-04-19T01:05:02.050 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.050 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.050 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:02.050 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.621352+0000", 2024-04-19T01:05:02.050 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0086576210000000008 2024-04-19T01:05:02.050 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.050 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.050 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:02.050 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.621360+0000", 2024-04-19T01:05:02.050 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4009999999999997e-06 2024-04-19T01:05:02.050 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.050 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.050 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:02.050 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.648969+0000", 2024-04-19T01:05:02.051 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.027608669999999998 2024-04-19T01:05:02.051 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.051 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.051 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:02.051 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.649012+0000", 2024-04-19T01:05:02.051 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3353999999999999e-05 2024-04-19T01:05:02.051 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:02.051 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:02.051 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:02.051 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.051 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.052 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 76/31 ECSubWrite(tid=102, reqid=client.4604.0:574, at_version=76'80, trim_to=0'0, roll_forward_to=74'72))", 2024-04-19T01:05:02.052 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:46.619419+0000", 2024-04-19T01:05:02.052 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.333793175, 2024-04-19T01:05:02.052 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.029644113, 2024-04-19T01:05:02.052 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:02.052 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:02.052 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:02.052 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:02.052 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.052 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:02.052 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.619419+0000", 2024-04-19T01:05:02.052 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:02.052 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.053 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.053 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:02.053 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.619419+0000", 2024-04-19T01:05:02.053 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:02.053 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.053 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.053 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:02.053 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.619421+0000", 2024-04-19T01:05:02.053 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6759999999999999e-06 2024-04-19T01:05:02.053 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.053 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.053 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:02.053 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.619861+0000", 2024-04-19T01:05:02.053 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00043962599999999998 2024-04-19T01:05:02.054 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.054 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.054 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:02.054 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.619890+0000", 2024-04-19T01:05:02.054 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8872000000000001e-05 2024-04-19T01:05:02.054 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.054 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.054 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:02.054 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.619905+0000", 2024-04-19T01:05:02.054 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5857000000000001e-05 2024-04-19T01:05:02.054 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.054 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.054 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:02.055 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.628845+0000", 2024-04-19T01:05:02.055 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0089394179999999993 2024-04-19T01:05:02.055 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.055 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.055 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:02.055 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.628861+0000", 2024-04-19T01:05:02.055 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6398000000000001e-05 2024-04-19T01:05:02.055 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.055 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.055 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:02.055 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.649021+0000", 2024-04-19T01:05:02.055 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020159664000000001 2024-04-19T01:05:02.055 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.056 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.056 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:02.056 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.649064+0000", 2024-04-19T01:05:02.056 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2602000000000003e-05 2024-04-19T01:05:02.056 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:02.056 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:02.056 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:02.056 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.056 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.056 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 79/31 ECSubWrite(tid=115, reqid=client.4604.0:591, at_version=79'90, trim_to=0'0, roll_forward_to=79'86))", 2024-04-19T01:05:02.056 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:49.657905+0000", 2024-04-19T01:05:02.056 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.295307611, 2024-04-19T01:05:02.056 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.043682177000000003, 2024-04-19T01:05:02.056 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:02.057 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:02.057 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:02.057 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:02.057 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.057 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:02.057 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.657905+0000", 2024-04-19T01:05:02.058 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:02.059 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.059 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.059 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:02.059 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.657905+0000", 2024-04-19T01:05:02.059 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:02.059 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.059 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.059 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:02.059 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.657907+0000", 2024-04-19T01:05:02.059 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2869999999999999e-06 2024-04-19T01:05:02.059 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.059 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.059 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:02.060 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658833+0000", 2024-04-19T01:05:02.060 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00092593299999999999 2024-04-19T01:05:02.060 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.060 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.060 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:02.060 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658867+0000", 2024-04-19T01:05:02.060 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4048999999999999e-05 2024-04-19T01:05:02.060 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.060 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.060 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:02.060 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658883+0000", 2024-04-19T01:05:02.060 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5502000000000002e-05 2024-04-19T01:05:02.060 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.061 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.061 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:02.061 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.659042+0000", 2024-04-19T01:05:02.061 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015962800000000001 2024-04-19T01:05:02.061 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.061 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.061 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:02.061 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.659073+0000", 2024-04-19T01:05:02.061 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0232000000000001e-05 2024-04-19T01:05:02.061 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.061 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.061 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:02.061 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.701480+0000", 2024-04-19T01:05:02.061 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.042407621999999999 2024-04-19T01:05:02.062 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.062 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.062 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:02.062 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.701587+0000", 2024-04-19T01:05:02.062 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010692399999999999 2024-04-19T01:05:02.062 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:02.062 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:02.062 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:02.062 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.062 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.062 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 79/31 ECSubWrite(tid=116, reqid=client.4604.0:592, at_version=79'91, trim_to=0'0, roll_forward_to=79'88))", 2024-04-19T01:05:02.062 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:49.666213+0000", 2024-04-19T01:05:02.062 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.286999718000001, 2024-04-19T01:05:02.063 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.035451589999999998, 2024-04-19T01:05:02.063 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:02.063 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:02.063 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:02.063 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:02.063 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.063 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:02.063 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.666213+0000", 2024-04-19T01:05:02.063 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:02.063 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.063 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.063 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:02.063 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.666213+0000", 2024-04-19T01:05:02.064 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:02.064 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.064 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.064 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:02.064 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.666215+0000", 2024-04-19T01:05:02.064 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.95e-06 2024-04-19T01:05:02.064 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.064 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.064 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:02.064 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.666782+0000", 2024-04-19T01:05:02.064 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00056714299999999999 2024-04-19T01:05:02.064 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.064 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.064 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:02.065 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.666830+0000", 2024-04-19T01:05:02.065 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7787000000000002e-05 2024-04-19T01:05:02.065 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.065 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.065 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:02.065 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.666852+0000", 2024-04-19T01:05:02.065 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2303e-05 2024-04-19T01:05:02.065 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.065 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.065 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:02.065 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.677400+0000", 2024-04-19T01:05:02.065 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010547503999999999 2024-04-19T01:05:02.066 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.066 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.066 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:02.066 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.677425+0000", 2024-04-19T01:05:02.066 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5644999999999999e-05 2024-04-19T01:05:02.066 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.066 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.066 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:02.066 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.701599+0000", 2024-04-19T01:05:02.066 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024174079000000001 2024-04-19T01:05:02.066 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.066 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.066 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:02.067 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.701664+0000", 2024-04-19T01:05:02.067 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5178999999999994e-05 2024-04-19T01:05:02.067 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:02.067 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:02.067 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:02.067 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.067 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.067 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 79/31 ECSubWrite(tid=117, reqid=client.4604.0:593, at_version=79'92, trim_to=0'0, roll_forward_to=79'88))", 2024-04-19T01:05:02.067 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:49.674069+0000", 2024-04-19T01:05:02.067 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.279143229000001, 2024-04-19T01:05:02.067 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.028967645, 2024-04-19T01:05:02.067 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:02.067 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:02.068 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:02.070 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:02.070 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.070 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:02.070 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.674069+0000", 2024-04-19T01:05:02.070 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:02.070 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.070 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.070 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:02.070 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.674069+0000", 2024-04-19T01:05:02.070 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:02.070 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.070 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.071 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:02.071 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.674072+0000", 2024-04-19T01:05:02.071 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4329999999999998e-06 2024-04-19T01:05:02.071 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.071 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.071 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:02.071 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.674528+0000", 2024-04-19T01:05:02.071 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00045595899999999998 2024-04-19T01:05:02.071 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.071 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.071 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:02.071 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.674586+0000", 2024-04-19T01:05:02.071 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7896999999999998e-05 2024-04-19T01:05:02.071 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.072 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.072 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:02.072 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.674610+0000", 2024-04-19T01:05:02.072 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4442000000000001e-05 2024-04-19T01:05:02.072 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.072 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.072 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:02.072 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.690138+0000", 2024-04-19T01:05:02.072 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015528192999999999 2024-04-19T01:05:02.072 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.072 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.072 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:02.072 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.690157+0000", 2024-04-19T01:05:02.072 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9134e-05 2024-04-19T01:05:02.073 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.073 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.073 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:02.073 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.702949+0000", 2024-04-19T01:05:02.073 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012791494 2024-04-19T01:05:02.073 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.073 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.073 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:02.073 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.703037+0000", 2024-04-19T01:05:02.073 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8092999999999998e-05 2024-04-19T01:05:02.073 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:02.073 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:02.073 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:02.073 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.074 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.074 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.4s2 79/31 ECSubWrite(tid=111, reqid=client.4604.0:605, at_version=79'36, trim_to=0'0, roll_forward_to=79'34))", 2024-04-19T01:05:02.074 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:49.681460+0000", 2024-04-19T01:05:02.074 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.271752789000001, 2024-04-19T01:05:02.074 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.032313512000000003, 2024-04-19T01:05:02.074 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:02.074 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:02.074 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:02.074 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:02.074 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.074 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:02.074 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.681460+0000", 2024-04-19T01:05:02.074 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:02.075 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.075 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.075 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:02.075 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.681460+0000", 2024-04-19T01:05:02.075 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:02.075 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.075 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.075 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:02.075 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.681462+0000", 2024-04-19T01:05:02.075 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8950000000000001e-06 2024-04-19T01:05:02.075 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.075 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.075 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:02.076 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.682022+0000", 2024-04-19T01:05:02.076 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00055996400000000003 2024-04-19T01:05:02.076 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.076 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.076 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:02.076 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.682075+0000", 2024-04-19T01:05:02.076 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3100000000000003e-05 2024-04-19T01:05:02.076 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.076 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.076 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:02.076 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.682096+0000", 2024-04-19T01:05:02.076 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0973000000000001e-05 2024-04-19T01:05:02.076 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.077 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.077 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:02.077 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.682234+0000", 2024-04-19T01:05:02.077 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013790499999999999 2024-04-19T01:05:02.077 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.077 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.077 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:02.077 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.682258+0000", 2024-04-19T01:05:02.077 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4528e-05 2024-04-19T01:05:02.077 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.077 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.077 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:02.077 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.713716+0000", 2024-04-19T01:05:02.077 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.03145746 2024-04-19T01:05:02.078 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.078 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.078 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:02.078 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.713773+0000", 2024-04-19T01:05:02.078 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7686999999999998e-05 2024-04-19T01:05:02.078 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:02.079 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:02.079 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:02.079 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.079 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.079 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 81/31 ECSubWrite(tid=122, reqid=client.4604.0:626, at_version=81'96, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:05:02.079 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:51.594239+0000", 2024-04-19T01:05:02.079 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.358973595, 2024-04-19T01:05:02.080 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.043938814999999999, 2024-04-19T01:05:02.080 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:02.080 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:02.080 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:02.080 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:02.081 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.081 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:02.081 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.594239+0000", 2024-04-19T01:05:02.082 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:02.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.082 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.082 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:02.082 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.594239+0000", 2024-04-19T01:05:02.082 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:02.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.082 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.082 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:02.082 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.594240+0000", 2024-04-19T01:05:02.082 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6300000000000005e-07 2024-04-19T01:05:02.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.082 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.082 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:02.083 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.595111+0000", 2024-04-19T01:05:02.083 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00087121599999999996 2024-04-19T01:05:02.083 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.083 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.083 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:02.083 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.595145+0000", 2024-04-19T01:05:02.083 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3698000000000002e-05 2024-04-19T01:05:02.083 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.083 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.083 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:02.083 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.595166+0000", 2024-04-19T01:05:02.083 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1081999999999999e-05 2024-04-19T01:05:02.083 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.083 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.084 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:02.084 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.595312+0000", 2024-04-19T01:05:02.084 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014611299999999999 2024-04-19T01:05:02.084 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.084 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.084 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:02.084 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.595336+0000", 2024-04-19T01:05:02.085 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4057999999999999e-05 2024-04-19T01:05:02.085 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.085 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.085 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:02.085 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.638124+0000", 2024-04-19T01:05:02.085 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.042788228999999997 2024-04-19T01:05:02.085 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.085 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.085 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:02.085 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.638178+0000", 2024-04-19T01:05:02.085 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3755999999999998e-05 2024-04-19T01:05:02.085 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:02.085 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:02.085 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:02.086 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.086 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.086 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 81/31 ECSubWrite(tid=123, reqid=client.4604.0:627, at_version=81'97, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:05:02.086 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:51.600586+0000", 2024-04-19T01:05:02.086 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.352626602999999, 2024-04-19T01:05:02.086 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.037646446, 2024-04-19T01:05:02.086 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:02.086 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:02.086 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:02.086 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:02.086 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.086 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:02.086 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.600586+0000", 2024-04-19T01:05:02.086 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:02.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.087 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:02.087 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.600586+0000", 2024-04-19T01:05:02.087 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:02.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.087 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:02.087 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.600588+0000", 2024-04-19T01:05:02.087 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0999999999999998e-06 2024-04-19T01:05:02.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.087 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:02.087 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.600966+0000", 2024-04-19T01:05:02.087 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000377878 2024-04-19T01:05:02.088 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.088 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.088 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:02.088 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.601011+0000", 2024-04-19T01:05:02.088 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4920999999999999e-05 2024-04-19T01:05:02.088 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.088 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.088 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:02.088 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.601032+0000", 2024-04-19T01:05:02.088 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1192999999999999e-05 2024-04-19T01:05:02.088 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.088 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.088 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:02.088 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.613182+0000", 2024-04-19T01:05:02.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012149900999999999 2024-04-19T01:05:02.090 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.090 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.090 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:02.090 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.613204+0000", 2024-04-19T01:05:02.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1761999999999999e-05 2024-04-19T01:05:02.090 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.090 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.091 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:02.091 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.638194+0000", 2024-04-19T01:05:02.091 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024990413 2024-04-19T01:05:02.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.091 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.091 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:02.091 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.638232+0000", 2024-04-19T01:05:02.091 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8278000000000002e-05 2024-04-19T01:05:02.091 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:02.091 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:02.091 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:02.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.091 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.092 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 81/31 ECSubWrite(tid=124, reqid=client.4604.0:628, at_version=81'98, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:05:02.092 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:51.606371+0000", 2024-04-19T01:05:02.092 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.346841733, 2024-04-19T01:05:02.092 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.031905021999999998, 2024-04-19T01:05:02.092 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:02.092 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:02.092 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:02.092 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:02.092 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.092 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:02.092 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.606371+0000", 2024-04-19T01:05:02.092 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:02.093 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.093 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.093 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:02.093 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.606371+0000", 2024-04-19T01:05:02.093 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:02.093 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.093 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.093 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:02.093 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.606373+0000", 2024-04-19T01:05:02.093 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.108e-06 2024-04-19T01:05:02.093 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.093 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.094 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:02.094 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.606762+0000", 2024-04-19T01:05:02.094 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000389398 2024-04-19T01:05:02.094 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.094 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.094 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:02.094 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.606802+0000", 2024-04-19T01:05:02.094 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0098000000000001e-05 2024-04-19T01:05:02.094 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.095 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.095 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:02.095 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.606825+0000", 2024-04-19T01:05:02.095 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.247e-05 2024-04-19T01:05:02.095 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.096 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.096 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:02.096 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.626279+0000", 2024-04-19T01:05:02.096 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019454050000000001 2024-04-19T01:05:02.097 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.097 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.097 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:02.097 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.626301+0000", 2024-04-19T01:05:02.097 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2011e-05 2024-04-19T01:05:02.097 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.097 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.097 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:02.097 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.638239+0000", 2024-04-19T01:05:02.097 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011937830999999999 2024-04-19T01:05:02.098 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.098 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.098 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:02.098 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.638276+0000", 2024-04-19T01:05:02.098 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7055999999999999e-05 2024-04-19T01:05:02.098 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:02.098 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:02.098 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:02.098 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.098 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.099 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 81/31 ECSubWrite(tid=125, reqid=client.4604.0:629, at_version=81'99, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:05:02.099 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:51.610678+0000", 2024-04-19T01:05:02.099 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.342534694999999, 2024-04-19T01:05:02.099 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.046619226999999999, 2024-04-19T01:05:02.099 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:02.099 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:02.099 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:02.099 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:02.099 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.099 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:02.099 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.610678+0000", 2024-04-19T01:05:02.099 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:02.099 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.099 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.099 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:02.100 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.610678+0000", 2024-04-19T01:05:02.100 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:02.100 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.100 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.100 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:02.100 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.610680+0000", 2024-04-19T01:05:02.100 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0499999999999999e-06 2024-04-19T01:05:02.101 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.101 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.101 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:02.101 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.610729+0000", 2024-04-19T01:05:02.101 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8906999999999998e-05 2024-04-19T01:05:02.101 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.101 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.101 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:02.102 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.610776+0000", 2024-04-19T01:05:02.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7250999999999999e-05 2024-04-19T01:05:02.102 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.102 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.102 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:02.102 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.610800+0000", 2024-04-19T01:05:02.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4094999999999999e-05 2024-04-19T01:05:02.102 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.102 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.102 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:02.102 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.637697+0000", 2024-04-19T01:05:02.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.026896798 2024-04-19T01:05:02.102 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.103 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:02.104 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.637715+0000", 2024-04-19T01:05:02.104 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7923000000000001e-05 2024-04-19T01:05:02.104 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.104 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.104 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:02.104 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.657204+0000", 2024-04-19T01:05:02.104 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.01948861 2024-04-19T01:05:02.104 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.104 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.104 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:02.104 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.657297+0000", 2024-04-19T01:05:02.104 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3592999999999997e-05 2024-04-19T01:05:02.105 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:02.105 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:02.105 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:02.105 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.105 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.105 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 81/31 ECSubWrite(tid=126, reqid=client.4604.0:631, at_version=81'101, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:05:02.105 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:51.619126+0000", 2024-04-19T01:05:02.105 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.334086210000001, 2024-04-19T01:05:02.105 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.038912264000000002, 2024-04-19T01:05:02.105 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:02.105 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:02.105 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:02.105 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:02.105 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.105 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:02.106 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.619126+0000", 2024-04-19T01:05:02.106 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:02.106 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.106 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.106 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:02.106 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.619126+0000", 2024-04-19T01:05:02.106 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:02.106 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.106 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.106 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:02.107 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.619128+0000", 2024-04-19T01:05:02.107 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0889999999999998e-06 2024-04-19T01:05:02.107 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.107 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.107 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:02.107 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.619962+0000", 2024-04-19T01:05:02.107 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00083304099999999997 2024-04-19T01:05:02.107 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.107 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.107 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:02.107 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.620004+0000", 2024-04-19T01:05:02.107 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2035e-05 2024-04-19T01:05:02.107 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.107 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.108 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:02.108 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.620029+0000", 2024-04-19T01:05:02.108 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5429e-05 2024-04-19T01:05:02.108 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.108 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.108 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:02.108 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.638344+0000", 2024-04-19T01:05:02.108 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018315498999999999 2024-04-19T01:05:02.108 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.108 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.108 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:02.108 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.638358+0000", 2024-04-19T01:05:02.108 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3392e-05 2024-04-19T01:05:02.108 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.108 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.109 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:02.109 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.657967+0000", 2024-04-19T01:05:02.109 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019608961000000001 2024-04-19T01:05:02.109 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:02.109 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:02.109 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:02.109 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.658039+0000", 2024-04-19T01:05:02.109 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1817999999999996e-05 2024-04-19T01:05:02.109 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:02.109 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:02.109 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:02.109 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:02.109 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:02.109 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:05:02.111 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-19T01:05:02.112 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:05:02.111+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:05:02.212 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:05:02.209+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:05:02.245 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:05:02.245 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:05:02.245 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-04-19T01:05:02.245 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:05:02.258 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-19T01:05:02.317 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:05:02.319+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:05:02.377 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-19T01:05:02.381 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:05:02.381 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:05:02.381 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-04-19T01:05:02.381 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-04-19T01:05:02.381 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:05:02.396 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-19T01:05:02.420 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:05:02.413+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:05:02.516 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:05:02.516 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-04-19T01:05:02.516 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-04-19T01:05:02.516 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-04-19T01:05:02.517 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.517 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.0s3 47/31 ECSubWrite(tid=32, reqid=client.4604.0:308, at_version=47'29, trim_to=0'0, roll_forward_to=47'28))", 2024-04-19T01:05:02.517 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:17.565539+0000", 2024-04-19T01:05:02.517 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 44.949916835000003, 2024-04-19T01:05:02.517 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.018109618000000001, 2024-04-19T01:05:02.517 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:02.517 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:02.517 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:02.517 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:02.517 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.517 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:02.517 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.565539+0000", 2024-04-19T01:05:02.517 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:02.518 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.518 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.518 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:02.518 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.565539+0000", 2024-04-19T01:05:02.518 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:02.518 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.518 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.518 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:02.518 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.565541+0000", 2024-04-19T01:05:02.518 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.04e-06 2024-04-19T01:05:02.518 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.518 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.519 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:02.519 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.566554+0000", 2024-04-19T01:05:02.519 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0010129029999999999 2024-04-19T01:05:02.519 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.519 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.519 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:02.519 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.566591+0000", 2024-04-19T01:05:02.519 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7061999999999999e-05 2024-04-19T01:05:02.519 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.519 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.519 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:02.519 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.566613+0000", 2024-04-19T01:05:02.520 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.145e-05 2024-04-19T01:05:02.520 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.520 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.520 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:02.526 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.566752+0000", 2024-04-19T01:05:02.526 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000138868 2024-04-19T01:05:02.526 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.526 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.526 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:02.527 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.566783+0000", 2024-04-19T01:05:02.527 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0772000000000003e-05 2024-04-19T01:05:02.527 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.527 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.527 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:02.527 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.583581+0000", 2024-04-19T01:05:02.527 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.016798124000000001 2024-04-19T01:05:02.527 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.527 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.527 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:02.527 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.583649+0000", 2024-04-19T01:05:02.527 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.8398999999999993e-05 2024-04-19T01:05:02.527 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:02.527 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:02.527 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:02.528 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.528 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.528 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.0s3 47/31 ECSubWrite(tid=33, reqid=client.4604.0:309, at_version=47'30, trim_to=0'0, roll_forward_to=47'28))", 2024-04-19T01:05:02.528 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:17.573407+0000", 2024-04-19T01:05:02.528 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 44.942049228000002, 2024-04-19T01:05:02.528 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.010864425, 2024-04-19T01:05:02.528 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:02.528 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:02.528 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:02.528 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:02.528 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.528 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:02.528 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.573407+0000", 2024-04-19T01:05:02.529 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:02.529 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.529 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.529 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:02.529 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.573407+0000", 2024-04-19T01:05:02.529 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:02.529 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.529 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.529 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:02.529 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.573408+0000", 2024-04-19T01:05:02.529 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4300000000000001e-06 2024-04-19T01:05:02.529 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.529 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.530 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:02.530 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.573474+0000", 2024-04-19T01:05:02.530 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5711999999999997e-05 2024-04-19T01:05:02.530 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.530 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.530 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:02.530 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.573506+0000", 2024-04-19T01:05:02.530 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1445999999999997e-05 2024-04-19T01:05:02.530 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.530 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.530 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:02.530 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.573526+0000", 2024-04-19T01:05:02.530 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9919999999999999e-05 2024-04-19T01:05:02.530 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.531 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.531 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:02.531 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.582034+0000", 2024-04-19T01:05:02.531 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.008508837 2024-04-19T01:05:02.531 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.531 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.531 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:02.531 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.582053+0000", 2024-04-19T01:05:02.531 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8816000000000001e-05 2024-04-19T01:05:02.531 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.531 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.531 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:02.531 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.584163+0000", 2024-04-19T01:05:02.532 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0021099439999999999 2024-04-19T01:05:02.532 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.532 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.532 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:02.532 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.584271+0000", 2024-04-19T01:05:02.532 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010832000000000001 2024-04-19T01:05:02.532 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:02.532 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:02.532 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:02.532 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.532 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.532 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.0s3 47/31 ECSubWrite(tid=34, reqid=client.4604.0:310, at_version=47'31, trim_to=0'0, roll_forward_to=47'28))", 2024-04-19T01:05:02.533 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:17.574889+0000", 2024-04-19T01:05:02.533 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 44.940567715, 2024-04-19T01:05:02.534 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0098864950000000004, 2024-04-19T01:05:02.534 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:02.534 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:02.534 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:02.534 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:02.535 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.535 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:02.535 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.574889+0000", 2024-04-19T01:05:02.535 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:02.535 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.535 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.535 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:02.535 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.574889+0000", 2024-04-19T01:05:02.535 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:02.535 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.535 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.535 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:02.535 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.574890+0000", 2024-04-19T01:05:02.535 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.457e-06 2024-04-19T01:05:02.535 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.536 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.536 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:02.536 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.574924+0000", 2024-04-19T01:05:02.536 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4181999999999999e-05 2024-04-19T01:05:02.536 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.536 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.536 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:02.536 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.574953+0000", 2024-04-19T01:05:02.536 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9226000000000001e-05 2024-04-19T01:05:02.536 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.536 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.536 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:02.536 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.574974+0000", 2024-04-19T01:05:02.536 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0273000000000001e-05 2024-04-19T01:05:02.536 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.537 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.537 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:02.537 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.583276+0000", 2024-04-19T01:05:02.537 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.008302495 2024-04-19T01:05:02.537 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.537 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.537 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:02.537 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.583290+0000", 2024-04-19T01:05:02.537 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3696999999999999e-05 2024-04-19T01:05:02.537 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.537 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.537 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:02.537 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.584690+0000", 2024-04-19T01:05:02.537 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001400047 2024-04-19T01:05:02.537 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.538 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.538 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:02.538 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.584775+0000", 2024-04-19T01:05:02.538 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5117999999999994e-05 2024-04-19T01:05:02.538 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:02.538 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:02.538 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:02.538 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.538 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.538 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as3 48/31 ECSubWrite(tid=71, reqid=client.4604.0:354, at_version=48'10, trim_to=0'0, roll_forward_to=34'8))", 2024-04-19T01:05:02.538 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.556344+0000", 2024-04-19T01:05:02.538 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 43.959112070000003, 2024-04-19T01:05:02.538 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.030027702, 2024-04-19T01:05:02.538 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:02.539 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:02.539 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:02.539 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:02.539 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.539 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:02.539 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.556344+0000", 2024-04-19T01:05:02.539 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:02.539 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.539 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.539 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:02.540 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.556344+0000", 2024-04-19T01:05:02.540 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:02.540 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.540 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.540 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:02.540 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.556345+0000", 2024-04-19T01:05:02.540 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1199999999999999e-07 2024-04-19T01:05:02.540 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.540 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.540 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:02.541 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.557285+0000", 2024-04-19T01:05:02.541 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00094096299999999996 2024-04-19T01:05:02.541 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.541 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.541 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:02.541 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.557335+0000", 2024-04-19T01:05:02.541 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9669000000000002e-05 2024-04-19T01:05:02.541 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.541 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.541 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:02.541 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.557358+0000", 2024-04-19T01:05:02.541 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3099999999999999e-05 2024-04-19T01:05:02.541 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.541 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.541 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:02.542 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.557490+0000", 2024-04-19T01:05:02.542 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013213400000000001 2024-04-19T01:05:02.542 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.542 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.542 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:02.542 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.557512+0000", 2024-04-19T01:05:02.542 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1603999999999998e-05 2024-04-19T01:05:02.543 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.543 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.543 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:02.545 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.586316+0000", 2024-04-19T01:05:02.545 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.028803714000000001 2024-04-19T01:05:02.545 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.545 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.545 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:02.545 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.586372+0000", 2024-04-19T01:05:02.546 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6205999999999997e-05 2024-04-19T01:05:02.546 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:02.546 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:02.546 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:02.546 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.546 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.546 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as3 65/31 ECSubWrite(tid=107, reqid=client.4604.0:478, at_version=65'26, trim_to=0'0, roll_forward_to=64'24))", 2024-04-19T01:05:02.546 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:35.638966+0000", 2024-04-19T01:05:02.546 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 26.876490573000002, 2024-04-19T01:05:02.546 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.01558386, 2024-04-19T01:05:02.546 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:02.546 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:02.546 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:02.546 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:02.547 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.547 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:02.547 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.638966+0000", 2024-04-19T01:05:02.547 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:02.547 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.547 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.547 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:02.547 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.638966+0000", 2024-04-19T01:05:02.547 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:02.547 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.547 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.547 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:02.547 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.638967+0000", 2024-04-19T01:05:02.548 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2500000000000001e-06 2024-04-19T01:05:02.548 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.548 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.548 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:02.548 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.639799+0000", 2024-04-19T01:05:02.548 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00083235399999999999 2024-04-19T01:05:02.548 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.548 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.548 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:02.548 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.639828+0000", 2024-04-19T01:05:02.548 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8513e-05 2024-04-19T01:05:02.548 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.548 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.549 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:02.549 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.639854+0000", 2024-04-19T01:05:02.549 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5769999999999999e-05 2024-04-19T01:05:02.549 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.549 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.549 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:02.549 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.640035+0000", 2024-04-19T01:05:02.549 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000181714 2024-04-19T01:05:02.549 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.549 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.549 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:02.549 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.640070+0000", 2024-04-19T01:05:02.549 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5179000000000003e-05 2024-04-19T01:05:02.552 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.552 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.552 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:02.552 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.654477+0000", 2024-04-19T01:05:02.552 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.014406402 2024-04-19T01:05:02.552 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.552 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.552 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:02.552 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.654550+0000", 2024-04-19T01:05:02.552 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2677999999999998e-05 2024-04-19T01:05:02.552 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:02.553 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:02.553 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:02.553 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.553 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.553 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as3 65/31 ECSubWrite(tid=108, reqid=client.4604.0:479, at_version=65'27, trim_to=0'0, roll_forward_to=64'24))", 2024-04-19T01:05:02.553 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:35.655152+0000", 2024-04-19T01:05:02.553 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 26.860304278000001, 2024-04-19T01:05:02.553 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.010020193, 2024-04-19T01:05:02.553 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:02.553 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:02.553 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:02.553 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:02.553 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.554 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:02.554 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.655152+0000", 2024-04-19T01:05:02.555 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:02.555 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.555 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.555 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:02.555 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.655152+0000", 2024-04-19T01:05:02.555 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:02.555 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.555 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.555 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:02.555 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.655154+0000", 2024-04-19T01:05:02.555 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.883e-06 2024-04-19T01:05:02.555 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.556 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.556 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:02.556 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.655409+0000", 2024-04-19T01:05:02.556 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000254827 2024-04-19T01:05:02.556 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.556 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.556 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:02.556 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.655437+0000", 2024-04-19T01:05:02.556 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8085999999999998e-05 2024-04-19T01:05:02.556 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.556 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.556 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:02.556 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.655455+0000", 2024-04-19T01:05:02.556 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8413000000000001e-05 2024-04-19T01:05:02.556 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.557 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.557 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:02.557 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.655610+0000", 2024-04-19T01:05:02.557 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000155005 2024-04-19T01:05:02.557 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.557 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.557 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:02.557 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.655636+0000", 2024-04-19T01:05:02.557 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6118999999999998e-05 2024-04-19T01:05:02.557 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.557 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.557 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:02.558 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.665068+0000", 2024-04-19T01:05:02.558 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0094318170000000003 2024-04-19T01:05:02.558 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.558 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.558 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:02.558 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.665172+0000", 2024-04-19T01:05:02.558 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010404300000000001 2024-04-19T01:05:02.558 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:02.558 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:02.558 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:02.558 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.558 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.558 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.2s5 79/31 ECSubWrite(tid=252, reqid=client.4604.0:600, at_version=79'23, trim_to=0'0, roll_forward_to=79'21))", 2024-04-19T01:05:02.558 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:49.675205+0000", 2024-04-19T01:05:02.559 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 12.84025113, 2024-04-19T01:05:02.559 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.010261670000000001, 2024-04-19T01:05:02.559 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:02.559 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:02.559 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:02.559 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:02.559 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.559 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:02.559 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.675205+0000", 2024-04-19T01:05:02.559 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:02.559 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.559 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.559 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:02.560 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.675205+0000", 2024-04-19T01:05:02.560 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:02.560 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.560 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.560 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:02.560 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.675208+0000", 2024-04-19T01:05:02.560 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9739999999999998e-06 2024-04-19T01:05:02.560 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.560 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.560 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:02.560 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.675577+0000", 2024-04-19T01:05:02.560 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00036901499999999999 2024-04-19T01:05:02.560 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.561 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.561 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:02.561 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.675649+0000", 2024-04-19T01:05:02.561 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2331999999999998e-05 2024-04-19T01:05:02.561 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.561 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.561 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:02.561 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.675674+0000", 2024-04-19T01:05:02.561 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4326999999999999e-05 2024-04-19T01:05:02.561 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.561 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.561 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:02.561 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.675827+0000", 2024-04-19T01:05:02.561 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015302800000000001 2024-04-19T01:05:02.561 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.562 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.562 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:02.562 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.675854+0000", 2024-04-19T01:05:02.562 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6746000000000001e-05 2024-04-19T01:05:02.562 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.562 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.563 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:02.563 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.685384+0000", 2024-04-19T01:05:02.563 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0095306990000000001 2024-04-19T01:05:02.563 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.563 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.563 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:02.563 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.685467+0000", 2024-04-19T01:05:02.563 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.2548999999999999e-05 2024-04-19T01:05:02.563 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:02.563 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:02.563 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:02.563 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.563 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.564 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.4s3 79/31 ECSubWrite(tid=111, reqid=client.4604.0:605, at_version=79'36, trim_to=0'0, roll_forward_to=79'34))", 2024-04-19T01:05:02.564 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:49.680617+0000", 2024-04-19T01:05:02.564 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 12.834839518000001, 2024-04-19T01:05:02.564 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.010979739000000001, 2024-04-19T01:05:02.564 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:02.564 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:02.564 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:02.564 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:02.564 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.565 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:02.565 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.680617+0000", 2024-04-19T01:05:02.565 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:02.565 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.565 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.565 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:02.565 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.680617+0000", 2024-04-19T01:05:02.565 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:02.565 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.565 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.565 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:02.566 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.680619+0000", 2024-04-19T01:05:02.566 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1399999999999998e-06 2024-04-19T01:05:02.566 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.566 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.566 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:02.566 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.681174+0000", 2024-04-19T01:05:02.566 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00055461900000000001 2024-04-19T01:05:02.566 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.566 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.566 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:02.566 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.681195+0000", 2024-04-19T01:05:02.566 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1804999999999999e-05 2024-04-19T01:05:02.566 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.567 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.567 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:02.567 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.681205+0000", 2024-04-19T01:05:02.567 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6679999999999996e-06 2024-04-19T01:05:02.567 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.567 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.567 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:02.567 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.681312+0000", 2024-04-19T01:05:02.567 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000107293 2024-04-19T01:05:02.567 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.567 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.567 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:02.567 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.681327+0000", 2024-04-19T01:05:02.568 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.49e-05 2024-04-19T01:05:02.568 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.568 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.568 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:02.568 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.691558+0000", 2024-04-19T01:05:02.568 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.010231153 2024-04-19T01:05:02.568 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.568 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.568 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:02.568 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.691596+0000", 2024-04-19T01:05:02.568 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8161000000000003e-05 2024-04-19T01:05:02.568 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:02.568 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:02.568 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:02.569 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.569 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.569 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.4s3 81/31 ECSubWrite(tid=117, reqid=client.4604.0:618, at_version=81'41, trim_to=0'0, roll_forward_to=79'39))", 2024-04-19T01:05:02.569 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:51.536744+0000", 2024-04-19T01:05:02.569 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.978712514, 2024-04-19T01:05:02.569 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.018283939999999999, 2024-04-19T01:05:02.569 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:02.569 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:02.569 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:02.569 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:02.569 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.569 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:02.569 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.536744+0000", 2024-04-19T01:05:02.569 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:02.570 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.570 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.570 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:02.570 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.536744+0000", 2024-04-19T01:05:02.570 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:02.570 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.570 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.570 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:02.570 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.536744+0000", 2024-04-19T01:05:02.571 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6600000000000002e-07 2024-04-19T01:05:02.571 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.571 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.571 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:02.571 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.537473+0000", 2024-04-19T01:05:02.571 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00072840000000000003 2024-04-19T01:05:02.571 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.571 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.571 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:02.571 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.537505+0000", 2024-04-19T01:05:02.571 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2746000000000001e-05 2024-04-19T01:05:02.571 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.571 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.572 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:02.572 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.537536+0000", 2024-04-19T01:05:02.572 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0562000000000003e-05 2024-04-19T01:05:02.572 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.572 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.572 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:02.572 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.537733+0000", 2024-04-19T01:05:02.572 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019705800000000001 2024-04-19T01:05:02.572 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.573 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.573 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:02.573 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.537762+0000", 2024-04-19T01:05:02.573 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8958999999999998e-05 2024-04-19T01:05:02.573 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.573 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.573 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:02.573 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.554926+0000", 2024-04-19T01:05:02.573 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.017164338000000001 2024-04-19T01:05:02.573 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.574 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.574 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:02.574 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.555028+0000", 2024-04-19T01:05:02.574 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000101511 2024-04-19T01:05:02.574 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:02.574 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:02.574 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:02.574 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.574 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.574 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.4s3 81/31 ECSubWrite(tid=120, reqid=client.4604.0:619, at_version=81'42, trim_to=0'0, roll_forward_to=79'39))", 2024-04-19T01:05:02.574 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:51.557348+0000", 2024-04-19T01:05:02.574 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.95810833, 2024-04-19T01:05:02.574 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.015514271, 2024-04-19T01:05:02.575 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:02.575 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:02.575 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:02.575 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:02.575 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.575 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:02.575 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.557348+0000", 2024-04-19T01:05:02.576 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:02.576 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.576 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.576 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:02.576 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.557348+0000", 2024-04-19T01:05:02.576 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:02.576 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.576 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.576 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:02.576 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.557350+0000", 2024-04-19T01:05:02.576 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6279999999999999e-06 2024-04-19T01:05:02.576 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.576 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.577 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:02.577 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.557622+0000", 2024-04-19T01:05:02.577 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00027276399999999999 2024-04-19T01:05:02.577 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.577 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.577 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:02.577 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.557649+0000", 2024-04-19T01:05:02.577 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6412999999999999e-05 2024-04-19T01:05:02.577 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.577 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.577 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:02.577 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.557663+0000", 2024-04-19T01:05:02.577 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4472e-05 2024-04-19T01:05:02.577 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.578 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.578 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:02.578 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.557936+0000", 2024-04-19T01:05:02.578 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00027278099999999999 2024-04-19T01:05:02.578 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.581 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.581 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:02.581 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.557957+0000", 2024-04-19T01:05:02.581 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0625999999999999e-05 2024-04-19T01:05:02.581 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.581 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.581 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:02.581 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.572801+0000", 2024-04-19T01:05:02.581 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.014844375999999999 2024-04-19T01:05:02.582 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.582 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.582 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:02.582 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.572862+0000", 2024-04-19T01:05:02.582 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1210999999999994e-05 2024-04-19T01:05:02.582 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:02.582 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:02.582 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:02.582 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.582 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.582 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.4s3 81/31 ECSubWrite(tid=123, reqid=client.4604.0:620, at_version=81'43, trim_to=0'0, roll_forward_to=81'42))", 2024-04-19T01:05:02.582 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:51.576489+0000", 2024-04-19T01:05:02.588 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.938967133, 2024-04-19T01:05:02.588 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.010317349999999999, 2024-04-19T01:05:02.588 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:02.588 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:02.588 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:02.588 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:02.589 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.589 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:02.589 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.576489+0000", 2024-04-19T01:05:02.589 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:02.589 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.589 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.589 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:02.589 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.576489+0000", 2024-04-19T01:05:02.589 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:02.590 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.590 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.590 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:02.590 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.576491+0000", 2024-04-19T01:05:02.590 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4899999999999999e-06 2024-04-19T01:05:02.590 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.590 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.590 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:02.590 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.576704+0000", 2024-04-19T01:05:02.590 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021347099999999999 2024-04-19T01:05:02.590 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.590 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.591 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:02.591 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.576730+0000", 2024-04-19T01:05:02.591 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5499000000000001e-05 2024-04-19T01:05:02.591 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.591 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.591 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:02.591 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.576746+0000", 2024-04-19T01:05:02.591 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6036e-05 2024-04-19T01:05:02.591 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.591 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.591 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:02.591 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.576887+0000", 2024-04-19T01:05:02.591 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014133599999999999 2024-04-19T01:05:02.591 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.592 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.594 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:02.594 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.576915+0000", 2024-04-19T01:05:02.594 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7847999999999998e-05 2024-04-19T01:05:02.595 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.595 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.599 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:02.599 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.586737+0000", 2024-04-19T01:05:02.599 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0098220990000000008 2024-04-19T01:05:02.599 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.599 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.599 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:02.599 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.586806+0000", 2024-04-19T01:05:02.599 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.9571000000000002e-05 2024-04-19T01:05:02.599 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:02.599 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:02.599 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:02.599 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.600 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.600 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.4s3 84/31 ECSubWrite(tid=136, reqid=osd.10.0:136, at_version=84'56, trim_to=0'0, roll_forward_to=84'52))", 2024-04-19T01:05:02.600 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:54.564645+0000", 2024-04-19T01:05:02.600 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 7.950810851, 2024-04-19T01:05:02.600 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0098734219999999998, 2024-04-19T01:05:02.600 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:02.600 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:02.600 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:02.600 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:02.600 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.601 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:02.601 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:54.564645+0000", 2024-04-19T01:05:02.601 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:02.601 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.601 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.601 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:02.601 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:54.564645+0000", 2024-04-19T01:05:02.601 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:02.602 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.602 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.602 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:02.602 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:54.564646+0000", 2024-04-19T01:05:02.602 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.0999999999999997e-07 2024-04-19T01:05:02.602 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.602 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.602 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:02.602 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:54.564685+0000", 2024-04-19T01:05:02.602 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8532000000000003e-05 2024-04-19T01:05:02.602 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.602 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.603 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:02.603 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:54.564705+0000", 2024-04-19T01:05:02.603 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9873000000000001e-05 2024-04-19T01:05:02.603 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.603 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.603 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:02.603 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:54.564722+0000", 2024-04-19T01:05:02.603 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7348999999999999e-05 2024-04-19T01:05:02.603 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.604 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.604 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:02.604 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:54.565141+0000", 2024-04-19T01:05:02.604 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000418983 2024-04-19T01:05:02.604 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.604 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.605 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:02.605 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:54.565167+0000", 2024-04-19T01:05:02.605 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6403000000000001e-05 2024-04-19T01:05:02.605 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.605 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.605 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:02.605 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:54.574426+0000", 2024-04-19T01:05:02.605 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0092584420000000004 2024-04-19T01:05:02.605 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.605 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.605 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:02.606 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:54.574519+0000", 2024-04-19T01:05:02.606 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2930000000000006e-05 2024-04-19T01:05:02.606 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:02.606 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:02.606 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:02.606 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.606 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.606 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as3 84/31 ECSubWrite(tid=173, reqid=osd.12.0:173, at_version=84'76, trim_to=0'0, roll_forward_to=84'74))", 2024-04-19T01:05:02.606 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:54.565367+0000", 2024-04-19T01:05:02.606 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 7.95008933, 2024-04-19T01:05:02.606 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0092910179999999998, 2024-04-19T01:05:02.606 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:02.606 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:02.607 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:02.607 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:02.607 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.607 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:02.607 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:54.565367+0000", 2024-04-19T01:05:02.607 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:02.607 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.607 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.607 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:02.607 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:54.565367+0000", 2024-04-19T01:05:02.607 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:02.607 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.608 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.608 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:02.608 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:54.565369+0000", 2024-04-19T01:05:02.608 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.59e-06 2024-04-19T01:05:02.608 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.608 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.608 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:02.608 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:54.565413+0000", 2024-04-19T01:05:02.608 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4703000000000002e-05 2024-04-19T01:05:02.608 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.608 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.608 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:02.608 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:54.565439+0000", 2024-04-19T01:05:02.609 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5888999999999999e-05 2024-04-19T01:05:02.609 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.609 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.609 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:02.609 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:54.565462+0000", 2024-04-19T01:05:02.609 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3215999999999999e-05 2024-04-19T01:05:02.610 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.610 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.610 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:02.610 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:54.565535+0000", 2024-04-19T01:05:02.610 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2327000000000001e-05 2024-04-19T01:05:02.610 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.610 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.610 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:02.610 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:54.565544+0000", 2024-04-19T01:05:02.610 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1200000000000008e-06 2024-04-19T01:05:02.610 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.611 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.611 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:02.611 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:54.574542+0000", 2024-04-19T01:05:02.611 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0089979729999999994 2024-04-19T01:05:02.611 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.611 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.611 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:02.611 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:54.574658+0000", 2024-04-19T01:05:02.611 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011620000000000001 2024-04-19T01:05:02.611 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:02.611 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:02.611 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:02.612 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.612 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.612 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.4s3 85/31 ECSubWrite(tid=138, reqid=osd.10.0:138, at_version=85'58, trim_to=0'0, roll_forward_to=84'56))", 2024-04-19T01:05:02.612 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:55.549556+0000", 2024-04-19T01:05:02.612 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 6.9659005230000002, 2024-04-19T01:05:02.612 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.010353438, 2024-04-19T01:05:02.612 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:02.612 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:02.612 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:02.612 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:02.612 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.612 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:02.612 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:55.549556+0000", 2024-04-19T01:05:02.613 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:02.613 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.613 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.613 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:02.613 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:55.549556+0000", 2024-04-19T01:05:02.613 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:02.613 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.613 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.613 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:02.613 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:55.549556+0000", 2024-04-19T01:05:02.613 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0299999999999999e-07 2024-04-19T01:05:02.613 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.613 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.613 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:02.614 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:55.549613+0000", 2024-04-19T01:05:02.614 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6502999999999998e-05 2024-04-19T01:05:02.614 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.614 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.614 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:02.614 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:55.549644+0000", 2024-04-19T01:05:02.614 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0769000000000003e-05 2024-04-19T01:05:02.615 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.621 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.621 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:02.621 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:55.549671+0000", 2024-04-19T01:05:02.621 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7158999999999999e-05 2024-04-19T01:05:02.621 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.621 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.621 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:02.621 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:55.549854+0000", 2024-04-19T01:05:02.622 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00018351500000000001 2024-04-19T01:05:02.622 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.622 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.622 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:02.622 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:55.549881+0000", 2024-04-19T01:05:02.624 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6846e-05 2024-04-19T01:05:02.624 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.624 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.624 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:02.625 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:55.559865+0000", 2024-04-19T01:05:02.625 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.009983713 2024-04-19T01:05:02.625 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.625 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.625 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:02.625 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:55.559909+0000", 2024-04-19T01:05:02.625 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4329999999999997e-05 2024-04-19T01:05:02.625 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:02.626 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:02.626 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:02.626 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.626 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.626 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.1s2 85/31 ECSubWrite(tid=42, reqid=osd.6.0:42, at_version=85'36, trim_to=0'0, roll_forward_to=84'32))", 2024-04-19T01:05:02.626 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:55.550182+0000", 2024-04-19T01:05:02.626 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 6.9652738430000003, 2024-04-19T01:05:02.626 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0098646900000000006, 2024-04-19T01:05:02.626 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:02.626 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:02.627 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:02.627 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:02.627 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.627 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:02.627 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:55.550182+0000", 2024-04-19T01:05:02.627 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:02.627 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.627 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.627 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:02.627 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:55.550182+0000", 2024-04-19T01:05:02.627 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:02.627 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.627 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.627 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:02.627 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:55.550185+0000", 2024-04-19T01:05:02.628 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4399999999999999e-06 2024-04-19T01:05:02.628 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.628 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.628 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:02.628 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:55.550308+0000", 2024-04-19T01:05:02.628 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000123063 2024-04-19T01:05:02.628 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.628 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.628 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:02.628 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:55.550336+0000", 2024-04-19T01:05:02.628 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7875999999999999e-05 2024-04-19T01:05:02.628 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.628 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.628 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:02.629 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:55.550360+0000", 2024-04-19T01:05:02.629 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4136999999999999e-05 2024-04-19T01:05:02.629 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.629 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.629 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:02.629 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:55.550520+0000", 2024-04-19T01:05:02.629 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015959500000000001 2024-04-19T01:05:02.629 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.629 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.629 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:02.629 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:55.550546+0000", 2024-04-19T01:05:02.629 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6253000000000001e-05 2024-04-19T01:05:02.629 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.629 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.629 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:02.630 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:55.559954+0000", 2024-04-19T01:05:02.630 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0094078860000000007 2024-04-19T01:05:02.630 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.630 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.630 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:02.630 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:55.560047+0000", 2024-04-19T01:05:02.630 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.344e-05 2024-04-19T01:05:02.630 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:02.630 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:02.630 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:02.630 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.630 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.630 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.2s5 86/31 ECSubWrite(tid=265, reqid=client.4604.0:655, at_version=86'35, trim_to=0'0, roll_forward_to=86'33))", 2024-04-19T01:05:02.630 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:56.580820+0000", 2024-04-19T01:05:02.631 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.9346364860000005, 2024-04-19T01:05:02.631 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.030714183999999999, 2024-04-19T01:05:02.631 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:02.631 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:02.631 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:02.631 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:02.631 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.631 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:02.631 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.580820+0000", 2024-04-19T01:05:02.631 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:02.631 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.631 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.632 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:02.632 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.580820+0000", 2024-04-19T01:05:02.632 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:02.632 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.632 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.632 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:02.632 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.580823+0000", 2024-04-19T01:05:02.632 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8090000000000001e-06 2024-04-19T01:05:02.632 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.632 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.632 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:02.632 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.581340+0000", 2024-04-19T01:05:02.633 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00051707599999999999 2024-04-19T01:05:02.633 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.633 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.633 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:02.633 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.581384+0000", 2024-04-19T01:05:02.633 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4369999999999997e-05 2024-04-19T01:05:02.633 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.633 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.633 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:02.633 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.581409+0000", 2024-04-19T01:05:02.633 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4913999999999999e-05 2024-04-19T01:05:02.633 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.633 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.634 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:02.634 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.581604+0000", 2024-04-19T01:05:02.634 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019476500000000001 2024-04-19T01:05:02.634 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.634 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.634 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:02.634 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.581631+0000", 2024-04-19T01:05:02.634 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6979e-05 2024-04-19T01:05:02.634 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.634 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.634 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:02.634 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.611484+0000", 2024-04-19T01:05:02.634 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.029853356000000001 2024-04-19T01:05:02.635 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.635 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.635 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:02.635 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.611534+0000", 2024-04-19T01:05:02.635 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9914999999999999e-05 2024-04-19T01:05:02.635 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:02.635 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:02.635 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:02.635 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.635 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.635 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.2s5 86/31 ECSubWrite(tid=266, reqid=client.4604.0:656, at_version=86'36, trim_to=0'0, roll_forward_to=86'33))", 2024-04-19T01:05:02.635 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:56.598798+0000", 2024-04-19T01:05:02.635 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.9166583790000002, 2024-04-19T01:05:02.635 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.014313559, 2024-04-19T01:05:02.636 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:02.636 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:02.636 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:02.636 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:02.636 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.636 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:02.637 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.598798+0000", 2024-04-19T01:05:02.637 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:02.637 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.637 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.637 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:02.637 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.598798+0000", 2024-04-19T01:05:02.637 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:02.637 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.637 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.637 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:02.637 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.598800+0000", 2024-04-19T01:05:02.637 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5919999999999999e-06 2024-04-19T01:05:02.637 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.638 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.638 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:02.638 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.598988+0000", 2024-04-19T01:05:02.638 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00018729799999999999 2024-04-19T01:05:02.638 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.638 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.638 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:02.638 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.599030+0000", 2024-04-19T01:05:02.638 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2592999999999997e-05 2024-04-19T01:05:02.638 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.638 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.638 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:02.639 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.599056+0000", 2024-04-19T01:05:02.639 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5684999999999999e-05 2024-04-19T01:05:02.639 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.639 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.639 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:02.639 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.601825+0000", 2024-04-19T01:05:02.639 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0027694180000000001 2024-04-19T01:05:02.639 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.639 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.639 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:02.639 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.601849+0000", 2024-04-19T01:05:02.639 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.315e-05 2024-04-19T01:05:02.639 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.640 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.640 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:02.640 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.613040+0000", 2024-04-19T01:05:02.640 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.011191134 2024-04-19T01:05:02.640 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.640 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.640 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:02.640 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.613111+0000", 2024-04-19T01:05:02.640 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.1688999999999998e-05 2024-04-19T01:05:02.640 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:02.640 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:02.640 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:02.640 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.640 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.641 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.0s3 91/31 ECSubWrite(tid=96, reqid=client.4604.0:683, at_version=91'89, trim_to=0'0, roll_forward_to=89'87))", 2024-04-19T01:05:02.641 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:01.594726+0000", 2024-04-19T01:05:02.641 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.92073008999999995, 2024-04-19T01:05:02.641 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.011665662, 2024-04-19T01:05:02.641 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:02.641 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:02.641 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:02.641 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:02.641 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.641 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:02.641 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.594726+0000", 2024-04-19T01:05:02.641 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:02.641 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.642 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.642 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:02.642 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.594726+0000", 2024-04-19T01:05:02.642 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:02.642 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.642 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.642 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:02.642 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.594726+0000", 2024-04-19T01:05:02.642 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7300000000000002e-07 2024-04-19T01:05:02.642 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.642 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.642 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:02.642 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.595219+0000", 2024-04-19T01:05:02.642 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00049298400000000002 2024-04-19T01:05:02.643 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.643 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.643 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:02.643 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.595243+0000", 2024-04-19T01:05:02.643 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3645000000000001e-05 2024-04-19T01:05:02.643 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.643 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.643 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:02.643 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.595260+0000", 2024-04-19T01:05:02.643 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6739e-05 2024-04-19T01:05:02.643 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.643 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.643 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:02.644 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.595428+0000", 2024-04-19T01:05:02.644 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016768600000000001 2024-04-19T01:05:02.644 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.644 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.644 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:02.644 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.595461+0000", 2024-04-19T01:05:02.644 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3949999999999999e-05 2024-04-19T01:05:02.644 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.644 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.644 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:02.644 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.606319+0000", 2024-04-19T01:05:02.644 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.010858003 2024-04-19T01:05:02.644 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.644 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.644 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:02.645 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.606392+0000", 2024-04-19T01:05:02.645 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2381999999999999e-05 2024-04-19T01:05:02.645 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:02.645 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:02.645 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:02.645 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.645 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.645 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.0s3 91/31 ECSubWrite(tid=97, reqid=client.4604.0:684, at_version=91'90, trim_to=0'0, roll_forward_to=89'87))", 2024-04-19T01:05:02.645 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:01.606875+0000", 2024-04-19T01:05:02.645 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.90858099299999995, 2024-04-19T01:05:02.645 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0099147720000000005, 2024-04-19T01:05:02.645 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:02.645 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:02.646 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:02.646 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:02.646 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.646 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:02.646 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.606875+0000", 2024-04-19T01:05:02.646 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:02.646 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.646 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.646 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:02.647 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.606875+0000", 2024-04-19T01:05:02.647 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:02.647 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.647 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.647 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:02.647 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.606877+0000", 2024-04-19T01:05:02.647 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.53e-06 2024-04-19T01:05:02.647 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.647 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.647 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:02.648 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.607151+0000", 2024-04-19T01:05:02.648 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00027466399999999998 2024-04-19T01:05:02.648 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.648 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.648 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:02.648 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.607180+0000", 2024-04-19T01:05:02.648 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8659000000000001e-05 2024-04-19T01:05:02.648 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.648 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.648 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:02.648 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.607199+0000", 2024-04-19T01:05:02.648 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9213999999999999e-05 2024-04-19T01:05:02.648 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.648 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.648 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:02.649 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.607336+0000", 2024-04-19T01:05:02.649 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013616999999999999 2024-04-19T01:05:02.649 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.649 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.649 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:02.649 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.607354+0000", 2024-04-19T01:05:02.649 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8462e-05 2024-04-19T01:05:02.649 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.649 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.649 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:02.649 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.616712+0000", 2024-04-19T01:05:02.649 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0093582219999999994 2024-04-19T01:05:02.649 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.649 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.650 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:02.650 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.616790+0000", 2024-04-19T01:05:02.651 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7850999999999997e-05 2024-04-19T01:05:02.651 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:02.652 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:02.652 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:02.652 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.652 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.653 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.0s3 91/31 ECSubWrite(tid=98, reqid=client.4604.0:685, at_version=91'91, trim_to=0'0, roll_forward_to=89'87))", 2024-04-19T01:05:02.653 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:01.618671+0000", 2024-04-19T01:05:02.653 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.89678546699999995, 2024-04-19T01:05:02.653 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.011554114000000001, 2024-04-19T01:05:02.653 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:02.653 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:02.653 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:02.653 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:02.653 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.653 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:02.653 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.618671+0000", 2024-04-19T01:05:02.653 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:02.653 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.653 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.653 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:02.654 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.618671+0000", 2024-04-19T01:05:02.654 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:02.654 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.654 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.654 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:02.654 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.618673+0000", 2024-04-19T01:05:02.654 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7439999999999999e-06 2024-04-19T01:05:02.654 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.654 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.654 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:02.654 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.618909+0000", 2024-04-19T01:05:02.654 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00023606699999999999 2024-04-19T01:05:02.654 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.654 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.655 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:02.655 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.618937+0000", 2024-04-19T01:05:02.655 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8163000000000001e-05 2024-04-19T01:05:02.655 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.655 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.655 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:02.655 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.618952+0000", 2024-04-19T01:05:02.656 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5173e-05 2024-04-19T01:05:02.656 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.656 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.656 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:02.656 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.619083+0000", 2024-04-19T01:05:02.656 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013083699999999999 2024-04-19T01:05:02.656 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.656 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.656 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:02.656 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.619102+0000", 2024-04-19T01:05:02.657 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8836999999999999e-05 2024-04-19T01:05:02.657 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.657 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.657 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:02.657 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.630158+0000", 2024-04-19T01:05:02.657 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.011056755 2024-04-19T01:05:02.657 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:02.657 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:02.657 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:02.657 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.630225+0000", 2024-04-19T01:05:02.657 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6538000000000006e-05 2024-04-19T01:05:02.657 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:02.657 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:02.657 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:02.657 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:02.658 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:02.658 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:05:02.659 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-04-19T01:05:02.660 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T01:05:02.661 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-04-19T01:05:02.661 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T01:05:02.661 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-19T01:05:02.662 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-04-19T01:05:02.663 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T01:05:02.663 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2024-04-19T01:05:02.663 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-04-19T01:05:02.664 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T01:05:02.665 INFO:tasks.rados.rados.0.smithi155.stdout:284: done (2 left) 2024-04-19T01:05:02.665 INFO:tasks.rados.rados.0.smithi155.stdout:285: done (1 left) 2024-04-19T01:05:02.665 INFO:tasks.rados.rados.0.smithi155.stdout:286: done (0 left) 2024-04-19T01:05:02.665 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 44 current snap is 31 2024-04-19T01:05:02.665 INFO:tasks.rados.rados.0.smithi155.stdout:287: seq_num 120 ranges {2441216=475136,2916352=655360,3571712=622592,4194304=278528} 2024-04-19T01:05:02.666 INFO:tasks.rados.rados.0.smithi155.stdout:287: writing smithi15528762-44 from 2441216 to 2916352 tid 1 2024-04-19T01:05:02.666 INFO:tasks.rados.rados.0.smithi155.stdout:287: writing smithi15528762-44 from 2916352 to 3571712 tid 2 2024-04-19T01:05:02.666 INFO:tasks.rados.rados.0.smithi155.stdout:287: writing smithi15528762-44 from 3571712 to 4194304 tid 3 2024-04-19T01:05:02.666 INFO:tasks.rados.rados.0.smithi155.stdout:287: writing smithi15528762-44 from 4194304 to 4472832 tid 4 2024-04-19T01:05:02.666 INFO:tasks.rados.rados.0.smithi155.stdout:288: delete oid 38 current snap is 31 2024-04-19T01:05:02.666 INFO:tasks.rados.rados.0.smithi155.stdout:288: done (1 left) 2024-04-19T01:05:02.666 INFO:tasks.rados.rados.0.smithi155.stdout:289: rollback oid 22 current snap is 31 2024-04-19T01:05:02.666 INFO:tasks.rados.rados.0.smithi155.stdout:rollback oid 22 to 26 2024-04-19T01:05:02.666 INFO:tasks.rados.rados.0.smithi155.stdout:290: copy_from oid 19 from oid 42 current snap is 31 2024-04-19T01:05:02.666 INFO:tasks.rados.rados.0.smithi155.stdout:291: setattr oid 18 current snap is 31 2024-04-19T01:05:02.666 INFO:tasks.rados.rados.0.smithi155.stdout:292: rollback oid 36 current snap is 31 2024-04-19T01:05:02.666 INFO:tasks.rados.rados.0.smithi155.stdout:rollback oid 36 to 26 2024-04-19T01:05:02.666 INFO:tasks.rados.rados.0.smithi155.stdout:293: snap_remove snap 28 2024-04-19T01:05:02.667 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-04-19T01:05:02.667 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:05:02.656+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:05:02.668 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T01:05:02.668 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-04-19T01:05:02.668 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T01:05:02.687 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-04-19T01:05:02.687 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T01:05:02.716 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-04-19T01:05:02.718 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T01:05:02.738 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2024-04-19T01:05:02.738 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T01:05:02.760 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:05:02.759+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:05:02.762 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2024-04-19T01:05:02.763 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T01:05:02.787 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2024-04-19T01:05:02.787 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T01:05:02.808 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2024-04-19T01:05:02.809 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T01:05:02.831 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2024-04-19T01:05:02.831 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-19T01:05:02.838 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:02.838 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:02.859 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2024-04-19T01:05:02.859 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-19T01:05:02.860 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-19T01:05:02.861 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:05:02.860+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:05:02.878 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2024-04-19T01:05:02.878 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-19T01:05:02.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:05:02.897+0000 7f908b7fe640 0 -- 172.21.15.33:0/1280582129 >> [v2:172.21.15.155:6807/741132602,v1:172.21.15.155:6810/741132602] conn(0x7f908c226410 msgr2=0x7f908c228800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-19T01:05:02.900 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2024-04-19T01:05:02.901 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-19T01:05:02.963 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:05:02.961+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:05:02.979 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:02.980 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:02.999 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-19T01:05:03.064 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:05:03.061+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:05:03.116 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:03.117 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:03.132 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-19T01:05:03.164 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:05:03.161+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:05:03.255 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:03.255 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:03.266 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:05:03.267+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:05:03.269 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-19T01:05:03.366 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:05:03.364+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:05:03.385 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:03.385 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:03.402 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-19T01:05:03.470 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:05:03.465+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:05:03.564 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:03.564 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:03.569 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:05:03.568+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:05:03.578 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-04-19T01:05:03.603 INFO:tasks.rados.rados.0.smithi155.stdout:293: done (5 left) 2024-04-19T01:05:03.603 INFO:tasks.rados.rados.0.smithi155.stdout:294: rollback oid 40 current snap is 31 2024-04-19T01:05:03.603 INFO:tasks.rados.rados.0.smithi155.stdout:rollback oid 40 to 26 2024-04-19T01:05:03.603 INFO:tasks.rados.rados.0.smithi155.stdout:292: finishing rollback tid 0 to smithi15528762-36 2024-04-19T01:05:03.603 INFO:tasks.rados.rados.0.smithi155.stdout:292: finishing rollback tid 1 to smithi15528762-36 2024-04-19T01:05:03.603 INFO:tasks.rados.rados.0.smithi155.stdout:292: finishing rollback tid 2 to smithi15528762-36 2024-04-19T01:05:03.603 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 36 v 63 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-04-19T01:05:03.604 INFO:tasks.rados.rados.0.smithi155.stdout:289: finishing rollback tid 0 to smithi15528762-22 2024-04-19T01:05:03.604 INFO:tasks.rados.rados.0.smithi155.stdout:292: done (5 left) 2024-04-19T01:05:03.604 INFO:tasks.rados.rados.0.smithi155.stdout:295: read oid 43 snap -1 2024-04-19T01:05:03.604 INFO:tasks.rados.rados.0.smithi155.stdout:295: expect (ObjNum 12 snap 0 seq_num 12) 2024-04-19T01:05:03.604 INFO:tasks.rados.rados.0.smithi155.stdout:296: delete oid 41 current snap is 31 2024-04-19T01:05:03.604 INFO:tasks.rados.rados.0.smithi155.stdout:289: finishing rollback tid 1 to smithi15528762-22 2024-04-19T01:05:03.604 INFO:tasks.rados.rados.0.smithi155.stdout:289: finishing rollback tid 2 to smithi15528762-22 2024-04-19T01:05:03.604 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 22 v 155 (ObjNum 105 snap 24 seq_num 105) dirty exists 2024-04-19T01:05:03.604 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 18 v 157 (ObjNum 56 snap 5 seq_num 56) dirty exists 2024-04-19T01:05:03.604 INFO:tasks.rados.rados.0.smithi155.stdout:287: finishing write tid 1 to smithi15528762-44 2024-04-19T01:05:03.604 INFO:tasks.rados.rados.0.smithi155.stdout:287: finishing write tid 2 to smithi15528762-44 2024-04-19T01:05:03.604 INFO:tasks.rados.rados.0.smithi155.stdout:287: finishing write tid 3 to smithi15528762-44 2024-04-19T01:05:03.604 INFO:tasks.rados.rados.0.smithi155.stdout:287: finishing write tid 4 to smithi15528762-44 2024-04-19T01:05:03.604 INFO:tasks.rados.rados.0.smithi155.stdout:287: finishing write tid 5 to smithi15528762-44 2024-04-19T01:05:03.605 INFO:tasks.rados.rados.0.smithi155.stdout:287: finishing write tid 6 to smithi15528762-44 2024-04-19T01:05:03.605 INFO:tasks.rados.rados.0.smithi155.stdout:287: oid 44 updating version 0 to 125 2024-04-19T01:05:03.605 INFO:tasks.rados.rados.0.smithi155.stdout:287: oid 44 updating version 125 to 128 2024-04-19T01:05:03.605 INFO:tasks.rados.rados.0.smithi155.stdout:287: oid 44 version 128 is already newer than 124 2024-04-19T01:05:03.605 INFO:tasks.rados.rados.0.smithi155.stdout:287: oid 44 version 128 is already newer than 126 2024-04-19T01:05:03.605 INFO:tasks.rados.rados.0.smithi155.stdout:287: oid 44 version 128 is already newer than 127 2024-04-19T01:05:03.605 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 44 v 128 (ObjNum 120 snap 31 seq_num 120) dirty exists 2024-04-19T01:05:03.605 INFO:tasks.rados.rados.0.smithi155.stdout:287: left oid 44 (ObjNum 120 snap 31 seq_num 120) 2024-04-19T01:05:03.605 INFO:tasks.rados.rados.0.smithi155.stdout:290: finishing copy_from racing read to smithi15528762-19 2024-04-19T01:05:03.605 INFO:tasks.rados.rados.0.smithi155.stdout:290: finishing copy_from to smithi15528762-19 2024-04-19T01:05:03.605 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 19 v 30 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-04-19T01:05:03.621 INFO:tasks.rados.rados.0.smithi155.stdout:295: expect (ObjNum 12 snap 0 seq_num 12) 2024-04-19T01:05:03.628 INFO:tasks.rados.rados.0.smithi155.stdout:294: finishing rollback tid 0 to smithi15528762-40 2024-04-19T01:05:03.628 INFO:tasks.rados.rados.0.smithi155.stdout:294: finishing rollback tid 1 to smithi15528762-40 2024-04-19T01:05:03.628 INFO:tasks.rados.rados.0.smithi155.stdout:294: finishing rollback tid 2 to smithi15528762-40 2024-04-19T01:05:03.628 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 40 v 50 (ObjNum 55 snap 5 seq_num 55) dirty exists 2024-04-19T01:05:03.629 INFO:tasks.rados.rados.0.smithi155.stdout:287: done (6 left) 2024-04-19T01:05:03.629 INFO:tasks.rados.rados.0.smithi155.stdout:289: done (5 left) 2024-04-19T01:05:03.629 INFO:tasks.rados.rados.0.smithi155.stdout:290: done (4 left) 2024-04-19T01:05:03.630 INFO:tasks.rados.rados.0.smithi155.stdout:291: done (3 left) 2024-04-19T01:05:03.630 INFO:tasks.rados.rados.0.smithi155.stdout:294: done (2 left) 2024-04-19T01:05:03.630 INFO:tasks.rados.rados.0.smithi155.stdout:295: done (1 left) 2024-04-19T01:05:03.630 INFO:tasks.rados.rados.0.smithi155.stdout:296: done (0 left) 2024-04-19T01:05:03.630 INFO:tasks.rados.rados.0.smithi155.stdout:297: snap_remove snap 30 2024-04-19T01:05:03.669 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:05:03.667+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:05:03.721 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:03.721 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:03.735 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-04-19T01:05:03.770 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:05:03.767+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:05:03.871 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:05:03.871+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:05:03.900 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:03.901 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:03.915 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-04-19T01:05:03.971 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:05:03.972+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:05:04.037 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:04.037 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:04.051 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-19T01:05:04.073 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:05:04.073+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:05:04.174 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:05:04.172+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:05:04.190 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:04.190 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:04.204 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-19T01:05:04.275 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:05:04.275+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:05:04.328 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:04.329 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:04.343 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-19T01:05:04.376 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:05:04.373+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:05:04.474 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:04.474 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:04.477 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:05:04.473+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:05:04.489 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-19T01:05:04.578 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:05:04.576+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:05:04.607 INFO:tasks.rados.rados.0.smithi155.stdout:297: done (0 left) 2024-04-19T01:05:04.607 INFO:tasks.rados.rados.0.smithi155.stdout:298: snap_create 2024-04-19T01:05:04.619 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-04-19T01:05:04.619 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-19T01:05:04.619 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] 2024-04-19T01:05:04.619 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 0 2024-04-19T01:05:04.623 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:04.624 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:04.643 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-19T01:05:04.667 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-19T01:05:04.678 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:05:04.673+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:05:04.780 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:05:04.780+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:05:04.824 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:04.825 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:04.840 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-19T01:05:04.881 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:05:04.880+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:05:04.922 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-19T01:05:04.947 INFO:tasks.ceph.mon.a.smithi033.stderr:2024-04-19T01:05:04.945+0000 7f9b08596640 -1 mon.a@0(leader).osd e94 definitely_dead 0 2024-04-19T01:05:04.958 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:04.958 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:04.973 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-19T01:05:04.982 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:05:04.980+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:05:05.068 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-04-19T01:05:05.069 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T01:05:05.084 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-04-19T01:05:05.085 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:05:05.083+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:05:05.085 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T01:05:05.105 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-04-19T01:05:05.106 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T01:05:05.126 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-04-19T01:05:05.126 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T01:05:05.136 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:05.137 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:05.144 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-04-19T01:05:05.144 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T01:05:05.151 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-19T01:05:05.165 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-04-19T01:05:05.166 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T01:05:05.185 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:05:05.183+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:05:05.187 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-04-19T01:05:05.187 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T01:05:05.210 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-04-19T01:05:05.210 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T01:05:05.230 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2024-04-19T01:05:05.230 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T01:05:05.251 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2024-04-19T01:05:05.251 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T01:05:05.272 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2024-04-19T01:05:05.272 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T01:05:05.283 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:05:05.283 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:05:05.283 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2024-04-19T01:05:05.284 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2024-04-19T01:05:05.284 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:05:05.286 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:05:05.284+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:05:05.293 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2024-04-19T01:05:05.293 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T01:05:05.297 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-19T01:05:05.312 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2024-04-19T01:05:05.312 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-19T01:05:05.334 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2024-04-19T01:05:05.335 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-19T01:05:05.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:05:05.333+0000 7f89feffd640 0 -- 172.21.15.33:0/831047266 >> [v2:172.21.15.118:6800/506416463,v1:172.21.15.118:6801/506416463] conn(0x7f8a00223dd0 msgr2=0x7f8a002261c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-19T01:05:05.387 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:05:05.388+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:05:05.434 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:05:05.434 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2024-04-19T01:05:05.434 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2024-04-19T01:05:05.434 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-04-19T01:05:05.434 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.434 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:423 3.5s0 3:a03c5c89:::smithi15528762-37:head [append 0~655360] snapc e=[d] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-19T01:05:05.435 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:29.537904+0000", 2024-04-19T01:05:05.435 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 35.893524470999999, 2024-04-19T01:05:05.435 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.049520382000000002, 2024-04-19T01:05:05.435 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:05.435 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:05.435 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:05.435 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:05.435 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:05.435 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:05.435 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 423 2024-04-19T01:05:05.435 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.435 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:05.435 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.435 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:05.436 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.537904+0000", 2024-04-19T01:05:05.436 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:05.436 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.436 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.436 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:05.436 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.537904+0000", 2024-04-19T01:05:05.436 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:05.436 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.436 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.436 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:05.436 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.537906+0000", 2024-04-19T01:05:05.436 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7859999999999999e-06 2024-04-19T01:05:05.436 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.436 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.437 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:05.437 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.538813+0000", 2024-04-19T01:05:05.437 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00090743099999999999 2024-04-19T01:05:05.437 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.437 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.437 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:05.437 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.538828+0000", 2024-04-19T01:05:05.437 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5206e-05 2024-04-19T01:05:05.437 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.437 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.437 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:05.437 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.538838+0000", 2024-04-19T01:05:05.437 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.3070000000000008e-06 2024-04-19T01:05:05.437 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.438 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.438 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:05.438 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.538910+0000", 2024-04-19T01:05:05.438 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2206000000000006e-05 2024-04-19T01:05:05.438 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.438 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.438 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:05.438 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.539016+0000", 2024-04-19T01:05:05.438 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010665199999999999 2024-04-19T01:05:05.438 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.438 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.438 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:05.438 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.540080+0000", 2024-04-19T01:05:05.439 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001063273 2024-04-19T01:05:05.439 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.439 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.439 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:05.439 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.545973+0000", 2024-04-19T01:05:05.439 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0058930570000000002 2024-04-19T01:05:05.439 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.439 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.439 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:05.439 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.587359+0000", 2024-04-19T01:05:05.439 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.041386128000000001 2024-04-19T01:05:05.439 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.439 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.440 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:05.440 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.587424+0000", 2024-04-19T01:05:05.440 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5335999999999999e-05 2024-04-19T01:05:05.440 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:05.440 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:05.440 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:05.440 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.440 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.440 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:424 3.5s0 3:a03c5c89:::smithi15528762-37:head [append 0~638976] snapc e=[d] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-19T01:05:05.440 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:29.545481+0000", 2024-04-19T01:05:05.440 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 35.885947483000002, 2024-04-19T01:05:05.440 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.042187482999999998, 2024-04-19T01:05:05.440 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:05.440 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:05.441 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:05.441 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:05.441 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:05.441 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:05.441 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 424 2024-04-19T01:05:05.441 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.441 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:05.441 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.441 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:05.441 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.545481+0000", 2024-04-19T01:05:05.441 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:05.441 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.441 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.441 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:05.442 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.545481+0000", 2024-04-19T01:05:05.442 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:05.442 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.442 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.442 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:05.442 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.545483+0000", 2024-04-19T01:05:05.442 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3879999999999998e-06 2024-04-19T01:05:05.442 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.442 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.442 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:05.442 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.546298+0000", 2024-04-19T01:05:05.442 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00081462099999999996 2024-04-19T01:05:05.442 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.442 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.443 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:05.443 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.546314+0000", 2024-04-19T01:05:05.443 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5722e-05 2024-04-19T01:05:05.443 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.443 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.443 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:05.443 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.546320+0000", 2024-04-19T01:05:05.443 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0939999999999996e-06 2024-04-19T01:05:05.443 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.443 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.443 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:05.443 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.546381+0000", 2024-04-19T01:05:05.443 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1696e-05 2024-04-19T01:05:05.443 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.443 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.444 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:05.444 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.546497+0000", 2024-04-19T01:05:05.444 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000115903 2024-04-19T01:05:05.444 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.444 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.444 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:05.444 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547522+0000", 2024-04-19T01:05:05.444 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010249289999999999 2024-04-19T01:05:05.444 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.444 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.444 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:05.444 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.556940+0000", 2024-04-19T01:05:05.444 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0094178899999999999 2024-04-19T01:05:05.444 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.587578+0000", 2024-04-19T01:05:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.030638021000000001 2024-04-19T01:05:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.587668+0000", 2024-04-19T01:05:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.0218999999999998e-05 2024-04-19T01:05:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:425 3.5s0 3:a03c5c89:::smithi15528762-37:head [append 0~131072] snapc e=[d] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-19T01:05:05.446 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:29.547018+0000", 2024-04-19T01:05:05.446 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 35.884410041999999, 2024-04-19T01:05:05.446 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.040774228000000003, 2024-04-19T01:05:05.446 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:05.446 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:05.446 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:05.446 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:05.446 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:05.446 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:05.446 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 425 2024-04-19T01:05:05.446 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.446 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:05.446 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.446 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:05.447 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547018+0000", 2024-04-19T01:05:05.447 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:05.447 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.447 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.447 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:05.447 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547018+0000", 2024-04-19T01:05:05.447 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:05.447 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.447 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.447 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:05.447 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547020+0000", 2024-04-19T01:05:05.447 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3710000000000001e-06 2024-04-19T01:05:05.447 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.447 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.447 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:05.448 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547148+0000", 2024-04-19T01:05:05.448 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012848699999999999 2024-04-19T01:05:05.448 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.448 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.448 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:05.448 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547162+0000", 2024-04-19T01:05:05.448 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4248e-05 2024-04-19T01:05:05.448 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.448 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.448 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:05.448 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547170+0000", 2024-04-19T01:05:05.448 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2069999999999998e-06 2024-04-19T01:05:05.448 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.448 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.449 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:05.449 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.554554+0000", 2024-04-19T01:05:05.449 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0073842159999999999 2024-04-19T01:05:05.449 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.449 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.449 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:05.449 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.554666+0000", 2024-04-19T01:05:05.449 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011222699999999999 2024-04-19T01:05:05.449 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.449 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.449 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:05.449 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.555080+0000", 2024-04-19T01:05:05.449 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00041430500000000001 2024-04-19T01:05:05.449 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.450 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.450 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:05.450 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.557075+0000", 2024-04-19T01:05:05.450 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001994198 2024-04-19T01:05:05.450 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.450 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.450 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:05.450 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.587746+0000", 2024-04-19T01:05:05.450 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.030671032000000001 2024-04-19T01:05:05.450 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.453 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.453 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:05.453 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.587793+0000", 2024-04-19T01:05:05.453 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6937000000000001e-05 2024-04-19T01:05:05.453 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:05.453 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:05.453 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:05.454 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.454 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.454 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:426 3.5s0 3:a03c5c89:::smithi15528762-37:head [setxattr _header (57) in=7b] snapc e=[d] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-19T01:05:05.454 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:29.547183+0000", 2024-04-19T01:05:05.454 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 35.884245206999999, 2024-04-19T01:05:05.454 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.040837638000000002, 2024-04-19T01:05:05.454 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:05.454 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:05.454 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:05.454 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:05.454 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:05.454 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:05.454 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 426 2024-04-19T01:05:05.454 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.455 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:05.455 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.455 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:05.455 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547183+0000", 2024-04-19T01:05:05.455 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:05.455 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.455 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.455 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:05.455 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547183+0000", 2024-04-19T01:05:05.456 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:05.456 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.456 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.456 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:05.456 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547184+0000", 2024-04-19T01:05:05.456 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.6899999999999997e-07 2024-04-19T01:05:05.456 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.456 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.456 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:05.456 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547188+0000", 2024-04-19T01:05:05.456 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4150000000000001e-06 2024-04-19T01:05:05.456 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.456 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.456 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:05.457 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547196+0000", 2024-04-19T01:05:05.457 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.8660000000000006e-06 2024-04-19T01:05:05.457 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.457 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.457 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:05.457 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547203+0000", 2024-04-19T01:05:05.457 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.0550000000000002e-06 2024-04-19T01:05:05.457 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.457 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.457 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:05.457 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.556074+0000", 2024-04-19T01:05:05.457 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0088710260000000006 2024-04-19T01:05:05.457 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.457 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.458 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:05.458 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.556179+0000", 2024-04-19T01:05:05.458 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000104947 2024-04-19T01:05:05.458 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.458 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.458 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:05.458 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.556694+0000", 2024-04-19T01:05:05.458 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000514563 2024-04-19T01:05:05.458 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.458 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.459 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:05.459 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.557301+0000", 2024-04-19T01:05:05.459 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00060763899999999999 2024-04-19T01:05:05.459 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.459 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.459 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:05.459 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.587923+0000", 2024-04-19T01:05:05.459 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.030621770999999999 2024-04-19T01:05:05.459 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.459 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.459 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:05.459 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.588021+0000", 2024-04-19T01:05:05.459 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7687000000000002e-05 2024-04-19T01:05:05.459 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:05.460 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:05.460 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:05.460 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.460 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.460 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:427 3.5s0 3:a03c5c89:::smithi15528762-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e59)", 2024-04-19T01:05:05.460 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:29.547214+0000", 2024-04-19T01:05:05.460 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 35.884214692999997, 2024-04-19T01:05:05.460 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.043029057000000002, 2024-04-19T01:05:05.460 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:05.460 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:05.460 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:05:05.460 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:05.460 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:05.460 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:05.461 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 427 2024-04-19T01:05:05.461 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.461 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:05.461 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.462 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:05.462 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547214+0000", 2024-04-19T01:05:05.462 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:05.462 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.462 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.462 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:05.462 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547214+0000", 2024-04-19T01:05:05.462 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:05.462 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.462 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.462 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:05.462 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547214+0000", 2024-04-19T01:05:05.462 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3799999999999998e-07 2024-04-19T01:05:05.463 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.463 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.463 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:05.463 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547216+0000", 2024-04-19T01:05:05.463 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.131e-06 2024-04-19T01:05:05.463 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.463 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.463 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:05.463 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547223+0000", 2024-04-19T01:05:05.463 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2790000000000004e-06 2024-04-19T01:05:05.463 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.463 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.464 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:05.464 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547226+0000", 2024-04-19T01:05:05.464 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4529999999999998e-06 2024-04-19T01:05:05.464 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.464 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.464 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:05.464 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.556968+0000", 2024-04-19T01:05:05.464 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0097420590000000008 2024-04-19T01:05:05.464 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.464 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.464 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:05:05.465 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.557060+0000", 2024-04-19T01:05:05.465 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.1807999999999999e-05 2024-04-19T01:05:05.465 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.465 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.465 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:05.465 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.588480+0000", 2024-04-19T01:05:05.465 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.031420333000000002 2024-04-19T01:05:05.465 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.465 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.465 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:05.465 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.588538+0000", 2024-04-19T01:05:05.465 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.7652000000000003e-05 2024-04-19T01:05:05.466 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.466 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.466 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:05.466 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.590243+0000", 2024-04-19T01:05:05.466 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0017049039999999999 2024-04-19T01:05:05.466 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:05.466 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:05.466 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:05.466 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.466 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.466 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:543 3.5s0 3:a8ddfdee:::smithi15528762-22:head [append 0~524288] snapc 17=[17,15,14,12] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-04-19T01:05:05.466 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:44.548241+0000", 2024-04-19T01:05:05.466 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 20.883187384999999, 2024-04-19T01:05:05.467 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.049103814000000003, 2024-04-19T01:05:05.467 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:05.467 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:05.467 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:05.467 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:05.467 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:05.467 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:05.468 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 543 2024-04-19T01:05:05.468 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.468 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:05.468 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.468 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:05.468 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.548241+0000", 2024-04-19T01:05:05.468 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:05.468 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.468 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.468 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:05.468 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.548241+0000", 2024-04-19T01:05:05.469 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:05.469 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.469 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.469 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:05.469 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.548244+0000", 2024-04-19T01:05:05.469 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.1609999999999999e-06 2024-04-19T01:05:05.469 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.469 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.469 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:05.469 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.548890+0000", 2024-04-19T01:05:05.469 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00064546500000000001 2024-04-19T01:05:05.469 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.469 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.469 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:05.470 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.548916+0000", 2024-04-19T01:05:05.470 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6563999999999998e-05 2024-04-19T01:05:05.470 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.470 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.470 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:05.470 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.548934+0000", 2024-04-19T01:05:05.470 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7351e-05 2024-04-19T01:05:05.470 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.470 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.470 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:05.470 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.549031+0000", 2024-04-19T01:05:05.470 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7528000000000005e-05 2024-04-19T01:05:05.471 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.471 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.471 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op must wait for map", 2024-04-19T01:05:05.471 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.549037+0000", 2024-04-19T01:05:05.471 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.7590000000000003e-06 2024-04-19T01:05:05.471 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.471 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.471 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:05.471 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.551497+0000", 2024-04-19T01:05:05.471 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0024606580000000001 2024-04-19T01:05:05.471 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.472 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.472 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:05.472 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.551597+0000", 2024-04-19T01:05:05.472 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.9525999999999995e-05 2024-04-19T01:05:05.472 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.472 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.472 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:05.472 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.552924+0000", 2024-04-19T01:05:05.472 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001327356 2024-04-19T01:05:05.472 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.472 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.472 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:05.472 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.559310+0000", 2024-04-19T01:05:05.473 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0063859900000000002 2024-04-19T01:05:05.473 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.473 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.473 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:05.473 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.597268+0000", 2024-04-19T01:05:05.473 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.037957492000000002 2024-04-19T01:05:05.473 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.473 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.473 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:05.473 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.597345+0000", 2024-04-19T01:05:05.473 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6964000000000004e-05 2024-04-19T01:05:05.473 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:05.473 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:05.473 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:05.474 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.474 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.474 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:591 3.5s0 3:a626fd0d:::smithi15528762-17:head [append 0~655360] snapc 1a=[19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-19T01:05:05.474 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:49.652801+0000", 2024-04-19T01:05:05.474 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 15.778627686, 2024-04-19T01:05:05.474 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.051043682999999999, 2024-04-19T01:05:05.474 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:05.474 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:05.474 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:05.474 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:05.474 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:05.474 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:05.474 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 591 2024-04-19T01:05:05.474 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.475 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:05.475 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.475 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:05.475 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.652801+0000", 2024-04-19T01:05:05.475 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:05.475 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.475 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.475 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:05.475 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.652801+0000", 2024-04-19T01:05:05.475 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:05.475 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.475 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.475 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:05.476 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.652826+0000", 2024-04-19T01:05:05.476 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5695e-05 2024-04-19T01:05:05.476 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.476 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.476 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:05.476 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.654116+0000", 2024-04-19T01:05:05.476 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001289807 2024-04-19T01:05:05.476 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.476 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.476 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:05.476 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.654135+0000", 2024-04-19T01:05:05.476 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8734999999999999e-05 2024-04-19T01:05:05.476 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.476 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.477 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:05.477 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.654147+0000", 2024-04-19T01:05:05.477 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2140000000000001e-05 2024-04-19T01:05:05.477 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.477 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.477 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:05.477 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.654229+0000", 2024-04-19T01:05:05.477 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1798000000000005e-05 2024-04-19T01:05:05.477 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.477 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.477 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:05.477 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.654371+0000", 2024-04-19T01:05:05.477 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000141817 2024-04-19T01:05:05.477 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.478 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.478 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:05.478 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.656095+0000", 2024-04-19T01:05:05.478 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00172386 2024-04-19T01:05:05.478 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.478 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.478 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:05.478 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.679643+0000", 2024-04-19T01:05:05.479 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.023548158999999999 2024-04-19T01:05:05.479 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.479 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.479 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:05.479 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.703771+0000", 2024-04-19T01:05:05.479 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.024128437999999999 2024-04-19T01:05:05.479 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.479 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.479 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:05.479 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.703844+0000", 2024-04-19T01:05:05.479 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.3233999999999997e-05 2024-04-19T01:05:05.480 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:05.480 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:05.480 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:05.480 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.480 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.480 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:592 3.5s0 3:a626fd0d:::smithi15528762-17:head [append 0~606208] snapc 1a=[19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-19T01:05:05.480 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:49.658117+0000", 2024-04-19T01:05:05.480 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 15.773311219, 2024-04-19T01:05:05.480 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.047890332000000001, 2024-04-19T01:05:05.480 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:05.481 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:05.481 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:05.481 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:05.481 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:05.481 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:05.481 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 592 2024-04-19T01:05:05.481 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.481 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:05.481 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.481 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:05.481 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.658117+0000", 2024-04-19T01:05:05.481 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:05.481 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.481 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.481 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:05.482 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.658117+0000", 2024-04-19T01:05:05.482 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:05.482 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.482 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.482 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:05.482 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.658120+0000", 2024-04-19T01:05:05.482 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9129999999999999e-06 2024-04-19T01:05:05.482 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.482 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.482 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:05.482 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.658931+0000", 2024-04-19T01:05:05.482 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00081110999999999998 2024-04-19T01:05:05.482 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.483 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.483 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:05.483 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.658965+0000", 2024-04-19T01:05:05.483 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4001999999999997e-05 2024-04-19T01:05:05.483 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.483 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.483 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:05.483 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.658981+0000", 2024-04-19T01:05:05.483 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5443000000000001e-05 2024-04-19T01:05:05.483 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.483 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.483 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:05.483 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663331+0000", 2024-04-19T01:05:05.483 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0043501240000000004 2024-04-19T01:05:05.484 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.484 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.484 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:05.484 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663457+0000", 2024-04-19T01:05:05.484 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000125857 2024-04-19T01:05:05.484 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.484 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.484 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:05.484 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.664259+0000", 2024-04-19T01:05:05.484 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00080283700000000004 2024-04-19T01:05:05.484 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.484 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.484 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:05.484 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.679757+0000", 2024-04-19T01:05:05.485 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015497100999999999 2024-04-19T01:05:05.485 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.485 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.485 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:05.485 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.705944+0000", 2024-04-19T01:05:05.485 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.026187892000000001 2024-04-19T01:05:05.485 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.485 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.485 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:05.485 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.706007+0000", 2024-04-19T01:05:05.485 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3052999999999994e-05 2024-04-19T01:05:05.485 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:05.485 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:05.485 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:05.486 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.486 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.486 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:593 3.5s0 3:a626fd0d:::smithi15528762-17:head [append 0~475136] snapc 1a=[19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-19T01:05:05.486 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:49.662459+0000", 2024-04-19T01:05:05.486 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 15.768969185, 2024-04-19T01:05:05.486 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.043664481999999998, 2024-04-19T01:05:05.486 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:05.486 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:05.486 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:05.486 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:05.486 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:05.486 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:05.486 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 593 2024-04-19T01:05:05.486 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.487 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:05.487 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.487 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:05.487 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.662459+0000", 2024-04-19T01:05:05.487 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:05.487 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.487 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.487 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:05.487 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.662459+0000", 2024-04-19T01:05:05.487 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:05.488 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.488 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.488 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:05.488 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.662465+0000", 2024-04-19T01:05:05.488 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2029999999999997e-06 2024-04-19T01:05:05.488 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.488 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.488 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:05.488 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663095+0000", 2024-04-19T01:05:05.488 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00062950400000000002 2024-04-19T01:05:05.488 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.488 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.488 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:05.488 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663191+0000", 2024-04-19T01:05:05.489 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.5866999999999995e-05 2024-04-19T01:05:05.489 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.489 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.489 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:05.489 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663213+0000", 2024-04-19T01:05:05.489 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1832e-05 2024-04-19T01:05:05.489 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.489 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.489 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:05.489 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.671032+0000", 2024-04-19T01:05:05.489 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0078190310000000006 2024-04-19T01:05:05.489 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.489 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.489 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:05.490 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.671178+0000", 2024-04-19T01:05:05.490 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014599400000000001 2024-04-19T01:05:05.490 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.490 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.490 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:05.490 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.672264+0000", 2024-04-19T01:05:05.490 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001086652 2024-04-19T01:05:05.490 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.490 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.490 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:05.490 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.679758+0000", 2024-04-19T01:05:05.490 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0074936059999999999 2024-04-19T01:05:05.490 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.490 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.491 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:05.491 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.706080+0000", 2024-04-19T01:05:05.491 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.026322534000000002 2024-04-19T01:05:05.491 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.491 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.491 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:05.491 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.706124+0000", 2024-04-19T01:05:05.491 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3259e-05 2024-04-19T01:05:05.491 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:05.491 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:05.491 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:05.491 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.491 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.491 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:594 3.5s0 3:a626fd0d:::smithi15528762-17:head [append 0~81920] snapc 1a=[19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-19T01:05:05.492 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:49.663273+0000", 2024-04-19T01:05:05.492 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 15.76815517, 2024-04-19T01:05:05.492 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.043572570999999997, 2024-04-19T01:05:05.492 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:05.492 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:05.492 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:05.492 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:05.492 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:05.492 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:05.492 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 594 2024-04-19T01:05:05.492 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.492 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:05.492 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.492 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:05.492 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663273+0000", 2024-04-19T01:05:05.493 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:05.493 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.493 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.493 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:05.493 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663273+0000", 2024-04-19T01:05:05.493 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:05.493 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.493 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.493 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:05.493 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663275+0000", 2024-04-19T01:05:05.493 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.421e-06 2024-04-19T01:05:05.493 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.493 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.493 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:05.494 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663327+0000", 2024-04-19T01:05:05.494 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2669e-05 2024-04-19T01:05:05.494 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.494 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.494 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:05.494 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663342+0000", 2024-04-19T01:05:05.494 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5046e-05 2024-04-19T01:05:05.494 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.494 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.494 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:05.494 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663351+0000", 2024-04-19T01:05:05.494 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.0610000000000002e-06 2024-04-19T01:05:05.494 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.494 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.495 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:05.495 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.676918+0000", 2024-04-19T01:05:05.495 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013566147000000001 2024-04-19T01:05:05.495 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.495 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.495 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:05.495 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.677079+0000", 2024-04-19T01:05:05.495 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000161683 2024-04-19T01:05:05.495 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.495 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.495 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:05.495 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.678015+0000", 2024-04-19T01:05:05.495 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00093627399999999996 2024-04-19T01:05:05.495 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.495 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.496 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:05.496 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.681601+0000", 2024-04-19T01:05:05.496 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003585464 2024-04-19T01:05:05.496 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.496 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.496 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:05.496 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.706786+0000", 2024-04-19T01:05:05.496 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.025185553999999999 2024-04-19T01:05:05.496 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.497 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.497 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:05.497 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.706846+0000", 2024-04-19T01:05:05.497 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.9252000000000001e-05 2024-04-19T01:05:05.497 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:05.497 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:05.497 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:05.497 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.497 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.497 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:595 3.5s0 3:a626fd0d:::smithi15528762-17:head [setxattr _header (57) in=7b] snapc 1a=[19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-19T01:05:05.497 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:49.663370+0000", 2024-04-19T01:05:05.498 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 15.768058712, 2024-04-19T01:05:05.498 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.044174650000000003, 2024-04-19T01:05:05.498 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:05.498 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:05.498 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:05.498 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:05.498 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:05.498 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:05.498 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 595 2024-04-19T01:05:05.498 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.498 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:05.498 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.498 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:05.498 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663370+0000", 2024-04-19T01:05:05.499 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:05.499 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.499 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.499 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:05.499 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663370+0000", 2024-04-19T01:05:05.499 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:05.499 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.499 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.499 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:05.499 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663371+0000", 2024-04-19T01:05:05.499 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.3699999999999999e-07 2024-04-19T01:05:05.499 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.499 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.499 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:05.500 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663376+0000", 2024-04-19T01:05:05.500 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6879999999999996e-06 2024-04-19T01:05:05.500 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.500 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.500 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:05.500 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663386+0000", 2024-04-19T01:05:05.500 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7829999999999993e-06 2024-04-19T01:05:05.500 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.500 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.500 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:05.500 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663393+0000", 2024-04-19T01:05:05.500 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2540000000000003e-06 2024-04-19T01:05:05.500 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.501 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.507 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:05.507 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.679885+0000", 2024-04-19T01:05:05.507 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.016492168000000001 2024-04-19T01:05:05.516 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.516 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.516 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:05.516 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.679999+0000", 2024-04-19T01:05:05.516 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000113638 2024-04-19T01:05:05.516 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.517 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.517 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:05.517 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.680695+0000", 2024-04-19T01:05:05.517 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00069550999999999999 2024-04-19T01:05:05.517 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.518 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.519 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:05.519 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.681605+0000", 2024-04-19T01:05:05.519 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00091078000000000003 2024-04-19T01:05:05.519 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.519 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.519 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:05.519 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.707485+0000", 2024-04-19T01:05:05.519 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.025879491000000001 2024-04-19T01:05:05.519 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.519 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.519 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:05.519 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.707544+0000", 2024-04-19T01:05:05.519 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.9401000000000003e-05 2024-04-19T01:05:05.520 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:05.520 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:05.520 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:05.520 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.520 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.520 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:596 3.5s0 3:a626fd0d:::smithi15528762-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e79)", 2024-04-19T01:05:05.520 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:49.663401+0000", 2024-04-19T01:05:05.520 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 15.768027001, 2024-04-19T01:05:05.520 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.045794460000000002, 2024-04-19T01:05:05.520 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:05.520 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:05.520 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:05:05.520 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:05.520 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:05.520 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:05.521 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 596 2024-04-19T01:05:05.521 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.521 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:05.521 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.521 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:05.521 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663401+0000", 2024-04-19T01:05:05.521 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:05.521 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.521 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.521 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:05.521 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663401+0000", 2024-04-19T01:05:05.521 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:05.521 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.521 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.522 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:05.522 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663402+0000", 2024-04-19T01:05:05.522 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2800000000000002e-07 2024-04-19T01:05:05.522 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.522 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.522 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:05.522 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663406+0000", 2024-04-19T01:05:05.522 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2830000000000003e-06 2024-04-19T01:05:05.522 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.522 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.522 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:05.522 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663412+0000", 2024-04-19T01:05:05.522 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.784e-06 2024-04-19T01:05:05.522 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.523 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.523 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:05.523 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663416+0000", 2024-04-19T01:05:05.523 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1339999999999997e-06 2024-04-19T01:05:05.523 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.523 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.523 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:05.523 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.681057+0000", 2024-04-19T01:05:05.523 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017640841000000001 2024-04-19T01:05:05.523 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.523 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.523 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:05:05.523 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.681119+0000", 2024-04-19T01:05:05.523 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2592e-05 2024-04-19T01:05:05.524 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.524 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.524 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:05.524 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.707942+0000", 2024-04-19T01:05:05.524 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.026822915999999999 2024-04-19T01:05:05.524 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.524 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.524 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:05.525 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.708008+0000", 2024-04-19T01:05:05.525 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.6088999999999997e-05 2024-04-19T01:05:05.525 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.525 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.525 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:05.525 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.709196+0000", 2024-04-19T01:05:05.525 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0011873929999999999 2024-04-19T01:05:05.525 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:05.525 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:05.525 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:05.525 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.526 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.526 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:626 3.5s0 3:ab059e23:::smithi15528762-38:head [append 0~737280] snapc 1c=[1c,1b,19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-19T01:05:05.526 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:51.587820+0000", 2024-04-19T01:05:05.526 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 13.843608794, 2024-04-19T01:05:05.526 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.070148642999999997, 2024-04-19T01:05:05.526 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:05.526 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:05.526 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:05.526 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:05.526 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:05.526 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:05.526 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 626 2024-04-19T01:05:05.526 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.527 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:05.527 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.527 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:05.527 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.587820+0000", 2024-04-19T01:05:05.527 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:05.527 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.527 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.527 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:05.527 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.587820+0000", 2024-04-19T01:05:05.527 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:05.527 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.527 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.527 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:05.528 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.587822+0000", 2024-04-19T01:05:05.528 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5150000000000001e-06 2024-04-19T01:05:05.528 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.528 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.528 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:05.528 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.588850+0000", 2024-04-19T01:05:05.528 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010283029999999999 2024-04-19T01:05:05.528 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.528 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.528 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:05.528 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.588877+0000", 2024-04-19T01:05:05.528 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6593000000000001e-05 2024-04-19T01:05:05.528 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.529 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.529 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:05.529 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.588891+0000", 2024-04-19T01:05:05.529 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4245e-05 2024-04-19T01:05:05.529 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.529 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.529 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:05.529 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.588997+0000", 2024-04-19T01:05:05.529 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000106252 2024-04-19T01:05:05.529 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.529 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.529 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op must wait for map", 2024-04-19T01:05:05.529 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.589004+0000", 2024-04-19T01:05:05.530 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.8079999999999999e-06 2024-04-19T01:05:05.530 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.530 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.530 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:05.530 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.591485+0000", 2024-04-19T01:05:05.530 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0024811970000000001 2024-04-19T01:05:05.530 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.530 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.530 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:05.530 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.591587+0000", 2024-04-19T01:05:05.530 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000101387 2024-04-19T01:05:05.530 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.530 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.530 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:05.530 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.592588+0000", 2024-04-19T01:05:05.531 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010014850000000001 2024-04-19T01:05:05.531 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.531 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.531 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:05.531 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.602598+0000", 2024-04-19T01:05:05.531 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010010023999999999 2024-04-19T01:05:05.531 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.531 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.531 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:05.531 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.657845+0000", 2024-04-19T01:05:05.531 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.055246157999999997 2024-04-19T01:05:05.531 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.531 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.531 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:05.532 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.657968+0000", 2024-04-19T01:05:05.532 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000123676 2024-04-19T01:05:05.532 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:05.532 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:05.532 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:05.532 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.532 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.532 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:627 3.5s0 3:ab059e23:::smithi15528762-38:head [append 0~557056] snapc 1c=[1c,1b,19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-19T01:05:05.532 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:51.594492+0000", 2024-04-19T01:05:05.532 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 13.836936182000001, 2024-04-19T01:05:05.532 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.06373906, 2024-04-19T01:05:05.532 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:05.532 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:05.532 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:05.533 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:05.533 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:05.533 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:05.533 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 627 2024-04-19T01:05:05.533 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.533 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:05.533 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.533 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:05.533 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.594492+0000", 2024-04-19T01:05:05.533 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:05.533 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.533 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.533 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:05.534 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.594492+0000", 2024-04-19T01:05:05.534 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:05.534 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.534 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.534 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:05.534 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.594494+0000", 2024-04-19T01:05:05.534 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8899999999999999e-06 2024-04-19T01:05:05.534 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.534 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.534 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:05.534 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.595085+0000", 2024-04-19T01:05:05.534 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00059048800000000002 2024-04-19T01:05:05.534 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.535 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.535 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:05.535 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.595102+0000", 2024-04-19T01:05:05.535 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7515000000000001e-05 2024-04-19T01:05:05.535 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.535 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.535 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:05.535 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.595114+0000", 2024-04-19T01:05:05.535 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1678000000000001e-05 2024-04-19T01:05:05.535 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.535 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.535 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:05.535 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.598103+0000", 2024-04-19T01:05:05.535 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0029891789999999998 2024-04-19T01:05:05.535 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.536 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.536 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:05.536 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.598183+0000", 2024-04-19T01:05:05.536 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.9659000000000004e-05 2024-04-19T01:05:05.536 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.536 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.536 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:05.536 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.598883+0000", 2024-04-19T01:05:05.536 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00069997600000000005 2024-04-19T01:05:05.536 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.537 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.537 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:05.537 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.608539+0000", 2024-04-19T01:05:05.537 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0096562739999999994 2024-04-19T01:05:05.537 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.537 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.537 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:05.537 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.658124+0000", 2024-04-19T01:05:05.537 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.049585021 2024-04-19T01:05:05.537 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.537 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.537 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:05.537 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.658231+0000", 2024-04-19T01:05:05.538 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010738000000000001 2024-04-19T01:05:05.538 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:05.538 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:05.538 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:05.538 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.538 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.538 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:626 3.5s0 3:ab059e23:::smithi15528762-38:head [append 0~737280] snapc 1c=[1c,1b,19,18,17,14,12] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e81)", 2024-04-19T01:05:05.538 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:51.601568+0000", 2024-04-19T01:05:05.538 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 13.829860293999999, 2024-04-19T01:05:05.538 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.056289715999999997, 2024-04-19T01:05:05.538 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:05.538 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:05.538 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "waiting for ondisk", 2024-04-19T01:05:05.538 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:05.539 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:05.539 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:05.539 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 626 2024-04-19T01:05:05.539 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.539 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:05.539 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.539 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:05.539 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.601568+0000", 2024-04-19T01:05:05.539 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:05.539 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.539 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.539 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:05.539 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.601568+0000", 2024-04-19T01:05:05.539 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:05.540 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.540 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.540 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:05.540 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.601569+0000", 2024-04-19T01:05:05.540 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.9999999999999997e-07 2024-04-19T01:05:05.540 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.540 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.540 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:05.540 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.602838+0000", 2024-04-19T01:05:05.540 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0012697349999999999 2024-04-19T01:05:05.540 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.540 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.540 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:05.540 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.602848+0000", 2024-04-19T01:05:05.540 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.6970000000000005e-06 2024-04-19T01:05:05.541 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.541 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.541 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:05.541 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.602858+0000", 2024-04-19T01:05:05.541 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.482e-06 2024-04-19T01:05:05.541 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.541 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.541 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:05.541 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.602901+0000", 2024-04-19T01:05:05.541 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3711999999999998e-05 2024-04-19T01:05:05.541 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.541 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.541 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for ondisk", 2024-04-19T01:05:05.541 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.602952+0000", 2024-04-19T01:05:05.541 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0481e-05 2024-04-19T01:05:05.542 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.542 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.542 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:05.542 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.657858+0000", 2024-04-19T01:05:05.542 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.054905909000000003 2024-04-19T01:05:05.542 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:05.542 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:05.542 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:05.542 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.542 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.543 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:627 3.5s0 3:ab059e23:::smithi15528762-38:head [append 0~557056] snapc 1c=[1c,1b,19,18,17,14,12] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e81)", 2024-04-19T01:05:05.543 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:51.602874+0000", 2024-04-19T01:05:05.543 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 13.828554471, 2024-04-19T01:05:05.543 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.055263801000000001, 2024-04-19T01:05:05.543 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:05.543 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:05.543 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "waiting for ondisk", 2024-04-19T01:05:05.543 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:05.543 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:05.543 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:05.543 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 627 2024-04-19T01:05:05.543 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.543 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:05.543 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.544 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:05.544 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.602874+0000", 2024-04-19T01:05:05.544 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:05.544 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.544 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.544 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:05.544 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.602874+0000", 2024-04-19T01:05:05.544 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:05.544 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.544 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.544 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:05.544 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.602875+0000", 2024-04-19T01:05:05.544 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.4300000000000002e-07 2024-04-19T01:05:05.544 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.545 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.545 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:05.545 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603321+0000", 2024-04-19T01:05:05.545 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000446468 2024-04-19T01:05:05.545 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.545 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.545 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:05.545 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603332+0000", 2024-04-19T01:05:05.545 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1079e-05 2024-04-19T01:05:05.545 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.545 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.545 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:05.545 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603337+0000", 2024-04-19T01:05:05.545 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0760000000000002e-06 2024-04-19T01:05:05.546 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.546 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.546 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:05.546 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603389+0000", 2024-04-19T01:05:05.546 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.1990000000000002e-05 2024-04-19T01:05:05.546 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.546 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.546 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for ondisk", 2024-04-19T01:05:05.546 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603501+0000", 2024-04-19T01:05:05.546 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000111539 2024-04-19T01:05:05.546 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.546 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.546 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:05.546 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.658138+0000", 2024-04-19T01:05:05.547 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.054636905999999999 2024-04-19T01:05:05.547 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:05.547 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:05.547 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:05.547 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.547 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.547 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:628 3.5s0 3:ab059e23:::smithi15528762-38:head [append 0~606208] snapc 1c=[1c,1b,19,18,17,14,12] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e81)", 2024-04-19T01:05:05.547 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:51.603350+0000", 2024-04-19T01:05:05.547 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 13.828078379000001, 2024-04-19T01:05:05.547 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.055106055000000001, 2024-04-19T01:05:05.547 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:05.547 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:05.547 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:05.547 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:05.547 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:05.548 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:05.548 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 628 2024-04-19T01:05:05.548 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.548 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:05.548 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.548 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:05.548 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603350+0000", 2024-04-19T01:05:05.548 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:05.548 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.548 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.548 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:05.548 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603350+0000", 2024-04-19T01:05:05.548 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:05.548 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.549 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.549 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:05.549 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603351+0000", 2024-04-19T01:05:05.549 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3499999999999996e-07 2024-04-19T01:05:05.549 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.549 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.549 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:05.549 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603847+0000", 2024-04-19T01:05:05.549 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00049669000000000004 2024-04-19T01:05:05.549 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.549 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.549 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:05.549 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603858+0000", 2024-04-19T01:05:05.549 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0729e-05 2024-04-19T01:05:05.549 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.550 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.550 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:05.550 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603863+0000", 2024-04-19T01:05:05.550 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0869999999999997e-06 2024-04-19T01:05:05.550 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.550 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.550 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:05.550 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603919+0000", 2024-04-19T01:05:05.550 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.5618000000000002e-05 2024-04-19T01:05:05.550 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.550 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.550 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:05.550 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.604028+0000", 2024-04-19T01:05:05.550 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000109187 2024-04-19T01:05:05.550 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.551 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.551 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:05.551 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.604655+0000", 2024-04-19T01:05:05.551 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00062722400000000001 2024-04-19T01:05:05.551 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.551 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.551 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:05.551 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.609598+0000", 2024-04-19T01:05:05.551 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.004943037 2024-04-19T01:05:05.551 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.551 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.551 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:05.551 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.658378+0000", 2024-04-19T01:05:05.551 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.048779623000000001 2024-04-19T01:05:05.551 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.552 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.552 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:05.552 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.658456+0000", 2024-04-19T01:05:05.552 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.8225000000000004e-05 2024-04-19T01:05:05.552 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:05.552 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:05.552 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:05.552 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.552 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.552 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:629 3.5s0 3:ab059e23:::smithi15528762-38:head [setxattr _header (57) in=7b] snapc 1c=[1c,1b,19,18,17,14,12] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e81)", 2024-04-19T01:05:05.552 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:51.603874+0000", 2024-04-19T01:05:05.552 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 13.827554265, 2024-04-19T01:05:05.552 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.054746045, 2024-04-19T01:05:05.552 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:05.553 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:05.553 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:05.553 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:05.553 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:05.553 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:05.553 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 629 2024-04-19T01:05:05.553 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.553 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:05.553 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.553 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:05.553 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603874+0000", 2024-04-19T01:05:05.553 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:05.553 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.553 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.554 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:05.554 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603874+0000", 2024-04-19T01:05:05.554 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:05.554 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.554 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.554 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:05.554 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603875+0000", 2024-04-19T01:05:05.554 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8800000000000002e-07 2024-04-19T01:05:05.554 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.554 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.554 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:05.554 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603878+0000", 2024-04-19T01:05:05.554 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.6399999999999999e-06 2024-04-19T01:05:05.555 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.555 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.555 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:05.555 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603885+0000", 2024-04-19T01:05:05.555 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.1640000000000004e-06 2024-04-19T01:05:05.555 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.555 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.555 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:05.555 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603892+0000", 2024-04-19T01:05:05.556 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1469999999999998e-06 2024-04-19T01:05:05.556 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.556 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.556 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:05.556 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.608675+0000", 2024-04-19T01:05:05.556 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0047830379999999999 2024-04-19T01:05:05.556 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.556 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.556 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:05.556 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.608741+0000", 2024-04-19T01:05:05.556 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.6212999999999998e-05 2024-04-19T01:05:05.556 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.556 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.557 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:05.557 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.609027+0000", 2024-04-19T01:05:05.557 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00028588600000000001 2024-04-19T01:05:05.557 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.557 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.557 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:05.557 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.609773+0000", 2024-04-19T01:05:05.557 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000746563 2024-04-19T01:05:05.557 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.557 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.557 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:05.557 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.658557+0000", 2024-04-19T01:05:05.557 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.048783343999999999 2024-04-19T01:05:05.557 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.558 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.558 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:05.558 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.658620+0000", 2024-04-19T01:05:05.558 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3461999999999996e-05 2024-04-19T01:05:05.558 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:05.558 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:05.558 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:05.558 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.558 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.558 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:630 3.5s0 3:ab059e23:::smithi15528762-38:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e81)", 2024-04-19T01:05:05.558 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:51.603912+0000", 2024-04-19T01:05:05.558 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 13.827516816999999, 2024-04-19T01:05:05.558 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.056857458, 2024-04-19T01:05:05.559 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:05.559 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:05.559 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:05:05.559 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:05.559 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:05.559 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:05.559 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 630 2024-04-19T01:05:05.559 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.559 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:05.559 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.559 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:05.559 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603912+0000", 2024-04-19T01:05:05.559 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:05.559 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.560 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.560 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:05.560 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603912+0000", 2024-04-19T01:05:05.560 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:05.560 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.560 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.560 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:05.560 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603912+0000", 2024-04-19T01:05:05.560 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6899999999999997e-07 2024-04-19T01:05:05.560 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.560 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.560 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:05.560 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603915+0000", 2024-04-19T01:05:05.560 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9730000000000002e-06 2024-04-19T01:05:05.560 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.561 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.561 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:05.561 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603923+0000", 2024-04-19T01:05:05.561 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1100000000000003e-06 2024-04-19T01:05:05.561 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.561 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.561 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:05.561 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603929+0000", 2024-04-19T01:05:05.561 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0170000000000002e-06 2024-04-19T01:05:05.561 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.561 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.561 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:05.561 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.609249+0000", 2024-04-19T01:05:05.561 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0053197380000000001 2024-04-19T01:05:05.561 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.562 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.562 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:05:05.562 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.609358+0000", 2024-04-19T01:05:05.562 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010952599999999999 2024-04-19T01:05:05.562 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.562 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.562 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:05.562 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.658676+0000", 2024-04-19T01:05:05.562 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.049317986000000001 2024-04-19T01:05:05.563 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.563 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.563 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:05.563 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.658789+0000", 2024-04-19T01:05:05.563 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011216400000000001 2024-04-19T01:05:05.564 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.564 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.564 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:05.564 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.660769+0000", 2024-04-19T01:05:05.564 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0019803749999999999 2024-04-19T01:05:05.564 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:05.564 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:05.564 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:05.564 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.564 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.564 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:631 3.5s0 3:abe8a6c4:::smithi15528762-18:head [copy-from ver 8] snapc 1c=[1c,1b,19,18,17,14,12] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e81)", 2024-04-19T01:05:05.564 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:51.603938+0000", 2024-04-19T01:05:05.564 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 13.827490343000001, 2024-04-19T01:05:05.565 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.055423066, 2024-04-19T01:05:05.565 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:05.565 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:05.565 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:05.565 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:05.565 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:05.565 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:05.565 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 631 2024-04-19T01:05:05.566 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.566 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:05.566 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.566 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:05.566 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603938+0000", 2024-04-19T01:05:05.566 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:05.566 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.566 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.566 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:05.566 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603938+0000", 2024-04-19T01:05:05.566 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:05.567 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.567 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.567 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:05.567 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603938+0000", 2024-04-19T01:05:05.567 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7199999999999999e-07 2024-04-19T01:05:05.567 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.567 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.567 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:05.567 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603942+0000", 2024-04-19T01:05:05.567 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0809999999999998e-06 2024-04-19T01:05:05.567 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.567 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.567 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:05.567 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603953+0000", 2024-04-19T01:05:05.568 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1477e-05 2024-04-19T01:05:05.568 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.568 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.568 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:05.568 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603962+0000", 2024-04-19T01:05:05.568 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.5010000000000004e-06 2024-04-19T01:05:05.568 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.568 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.568 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:05.568 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.609402+0000", 2024-04-19T01:05:05.568 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0054401069999999996 2024-04-19T01:05:05.568 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.568 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.568 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:05.568 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.609457+0000", 2024-04-19T01:05:05.569 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.5016000000000003e-05 2024-04-19T01:05:05.569 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.569 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.569 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:05.569 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.617371+0000", 2024-04-19T01:05:05.569 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0079141149999999993 2024-04-19T01:05:05.569 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.569 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.569 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:05.569 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.630663+0000", 2024-04-19T01:05:05.569 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013291956000000001 2024-04-19T01:05:05.569 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.569 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.569 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:05.569 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.659260+0000", 2024-04-19T01:05:05.570 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.028597651000000002 2024-04-19T01:05:05.570 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:05.570 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:05.570 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:05.570 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.659361+0000", 2024-04-19T01:05:05.570 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010069 2024-04-19T01:05:05.570 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:05.570 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:05.570 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:05.570 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:05.570 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:05.570 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:05:05.572 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-04-19T01:05:05.572 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2024-04-19T01:05:05.573 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-19T01:05:05.573 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:05:05.572+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:05:05.594 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2024-04-19T01:05:05.594 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-19T01:05:05.616 INFO:teuthology.orchestra.run.smithi033.stderr:marked out osd.0. 2024-04-19T01:05:05.616 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2024-04-19T01:05:05.623 INFO:tasks.rados.rados.0.smithi155.stdout:298: done (0 left) 2024-04-19T01:05:05.623 INFO:tasks.rados.rados.0.smithi155.stdout:299: snap_create 2024-04-19T01:05:05.702 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:05:05.701+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:05:05.733 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:05:05.733 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:05:05.733 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2024-04-19T01:05:05.733 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:05:05.747 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-04-19T01:05:05.773 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:05:05.770+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:05:05.874 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:05:05.872+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:05:05.918 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:05:05.919 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:05:05.919 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2024-04-19T01:05:05.919 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2024-04-19T01:05:05.919 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:05:05.934 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-04-19T01:05:05.976 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:05:05.974+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:05:06.067 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:05:06.067 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2024-04-19T01:05:06.067 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2024-04-19T01:05:06.067 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-04-19T01:05:06.067 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.067 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:253 3.ds0 3:b9317ffa:::smithi15528762-1:head [append 0~704512] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:05:06.067 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.983022+0000", 2024-04-19T01:05:06.067 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 52.080419650000003, 2024-04-19T01:05:06.069 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.034825673000000001, 2024-04-19T01:05:06.069 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:06.069 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:06.069 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:06.069 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:06.069 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:06.070 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:06.070 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 253 2024-04-19T01:05:06.070 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.070 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:06.071 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.071 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:06.071 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.983022+0000", 2024-04-19T01:05:06.071 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:06.071 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.071 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.071 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:06.071 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.983022+0000", 2024-04-19T01:05:06.071 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:06.071 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.071 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.072 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:06.072 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.983023+0000", 2024-04-19T01:05:06.072 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.6500000000000008e-07 2024-04-19T01:05:06.072 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.072 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.072 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:06.072 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.983925+0000", 2024-04-19T01:05:06.074 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000902439 2024-04-19T01:05:06.074 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.074 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.074 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:06.074 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.983943+0000", 2024-04-19T01:05:06.074 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7975e-05 2024-04-19T01:05:06.074 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.074 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.074 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:06.075 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.983952+0000", 2024-04-19T01:05:06.075 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.4270000000000008e-06 2024-04-19T01:05:06.075 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.075 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.076 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:06.076 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.984029+0000", 2024-04-19T01:05:06.076 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7631000000000003e-05 2024-04-19T01:05:06.076 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.076 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.984150+0000", 2024-04-19T01:05:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000120548 2024-04-19T01:05:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.986166+0000", 2024-04-19T01:05:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0020163410000000001 2024-04-19T01:05:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.001084+0000", 2024-04-19T01:05:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014917872 2024-04-19T01:05:06.077 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.017779+0000", 2024-04-19T01:05:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.016694817000000001 2024-04-19T01:05:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.017848+0000", 2024-04-19T01:05:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.8658000000000006e-05 2024-04-19T01:05:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:06.078 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.079 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.079 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:254 3.ds0 3:b9317ffa:::smithi15528762-1:head [append 0~376832] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:05:06.079 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.985096+0000", 2024-04-19T01:05:06.079 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 52.078346035000003, 2024-04-19T01:05:06.079 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.043912868000000001, 2024-04-19T01:05:06.079 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:06.079 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:06.079 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:06.079 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:06.079 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:06.080 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:06.080 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 254 2024-04-19T01:05:06.080 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.080 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:06.080 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.080 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:06.080 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985096+0000", 2024-04-19T01:05:06.080 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:06.080 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.080 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.080 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:06.080 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985096+0000", 2024-04-19T01:05:06.080 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:06.081 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.081 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.081 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:06.081 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985098+0000", 2024-04-19T01:05:06.081 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0540000000000002e-06 2024-04-19T01:05:06.081 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.081 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.081 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:06.081 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985632+0000", 2024-04-19T01:05:06.081 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00053471200000000001 2024-04-19T01:05:06.081 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.081 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.081 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:06.081 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985650+0000", 2024-04-19T01:05:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7217000000000001e-05 2024-04-19T01:05:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985658+0000", 2024-04-19T01:05:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.8689999999999993e-06 2024-04-19T01:05:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.995027+0000", 2024-04-19T01:05:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0093689159999999997 2024-04-19T01:05:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:06.082 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.995141+0000", 2024-04-19T01:05:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011387000000000001 2024-04-19T01:05:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.996005+0000", 2024-04-19T01:05:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00086397999999999998 2024-04-19T01:05:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.002138+0000", 2024-04-19T01:05:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.006133182 2024-04-19T01:05:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.083 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.084 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:06.084 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.028913+0000", 2024-04-19T01:05:06.084 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.026774686999999998 2024-04-19T01:05:06.084 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.084 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.084 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:06.084 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.029008+0000", 2024-04-19T01:05:06.084 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.5381e-05 2024-04-19T01:05:06.084 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:06.084 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:06.084 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:06.084 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.084 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.084 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:255 3.ds0 3:b9317ffa:::smithi15528762-1:head [setxattr _header (55) in=7b] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:05:06.085 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.985699+0000", 2024-04-19T01:05:06.085 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 52.077742497000003, 2024-04-19T01:05:06.085 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.043535886000000003, 2024-04-19T01:05:06.085 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:06.085 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:06.085 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:06.085 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:06.085 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:06.085 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:06.085 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 255 2024-04-19T01:05:06.086 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.086 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:06.086 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.086 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:06.086 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985699+0000", 2024-04-19T01:05:06.087 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:06.087 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.087 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.087 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:06.087 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985699+0000", 2024-04-19T01:05:06.087 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:06.087 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.087 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.087 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:06.087 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985700+0000", 2024-04-19T01:05:06.087 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4199999999999995e-07 2024-04-19T01:05:06.087 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.087 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.088 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:06.088 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985705+0000", 2024-04-19T01:05:06.088 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7960000000000003e-06 2024-04-19T01:05:06.088 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.088 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.088 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:06.088 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985715+0000", 2024-04-19T01:05:06.088 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0285999999999999e-05 2024-04-19T01:05:06.088 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.088 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.088 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:06.088 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985735+0000", 2024-04-19T01:05:06.088 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9666000000000001e-05 2024-04-19T01:05:06.088 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.001214+0000", 2024-04-19T01:05:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015479694 2024-04-19T01:05:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.001417+0000", 2024-04-19T01:05:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00020274099999999999 2024-04-19T01:05:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.001931+0000", 2024-04-19T01:05:06.089 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000513599 2024-04-19T01:05:06.090 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.090 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.090 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:06.090 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.002446+0000", 2024-04-19T01:05:06.090 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00051521100000000001 2024-04-19T01:05:06.091 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.091 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.091 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:06.091 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.029136+0000", 2024-04-19T01:05:06.091 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.026690523000000001 2024-04-19T01:05:06.091 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.091 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.091 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:06.091 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.029235+0000", 2024-04-19T01:05:06.091 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.8727999999999994e-05 2024-04-19T01:05:06.091 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:06.091 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:06.092 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:06.092 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.092 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.092 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:256 3.ds0 3:b9317ffa:::smithi15528762-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:05:06.092 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.985744+0000", 2024-04-19T01:05:06.092 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 52.077697710999999, 2024-04-19T01:05:06.092 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.046574693, 2024-04-19T01:05:06.092 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:06.092 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:06.092 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:05:06.092 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:06.092 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:06.092 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:06.092 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 256 2024-04-19T01:05:06.093 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.093 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:06.093 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.093 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:06.093 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985744+0000", 2024-04-19T01:05:06.093 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:06.093 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.093 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.093 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:06.093 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985744+0000", 2024-04-19T01:05:06.093 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:06.093 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.094 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.094 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:06.094 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985744+0000", 2024-04-19T01:05:06.094 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1699999999999999e-07 2024-04-19T01:05:06.094 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.094 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.094 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:06.094 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985747+0000", 2024-04-19T01:05:06.094 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3429999999999998e-06 2024-04-19T01:05:06.094 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.094 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.095 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:06.095 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985753+0000", 2024-04-19T01:05:06.095 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8619999999999999e-06 2024-04-19T01:05:06.095 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.095 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.095 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:06.095 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985756+0000", 2024-04-19T01:05:06.095 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8820000000000003e-06 2024-04-19T01:05:06.095 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.095 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.095 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:06.095 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.002179+0000", 2024-04-19T01:05:06.095 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.01642302 2024-04-19T01:05:06.096 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.096 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.096 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:05:06.096 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.002278+0000", 2024-04-19T01:05:06.096 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.8548000000000006e-05 2024-04-19T01:05:06.096 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.097 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.097 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:06.097 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.029285+0000", 2024-04-19T01:05:06.097 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.027006935999999999 2024-04-19T01:05:06.097 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.097 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.097 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:06.097 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.029400+0000", 2024-04-19T01:05:06.097 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000115438 2024-04-19T01:05:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.032319+0000", 2024-04-19T01:05:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0029182470000000001 2024-04-19T01:05:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:330 3.ds0 3:bf91dcb5:::smithi15528762-2:head [append 0~507904] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:05:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.674394+0000", 2024-04-19T01:05:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 48.389047560000002, 2024-04-19T01:05:06.098 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.036711424999999999, 2024-04-19T01:05:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 330 2024-04-19T01:05:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.674394+0000", 2024-04-19T01:05:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.099 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.100 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:06.100 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.674394+0000", 2024-04-19T01:05:06.100 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:06.100 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.100 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.100 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:06.100 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.674396+0000", 2024-04-19T01:05:06.100 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3870000000000002e-06 2024-04-19T01:05:06.100 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.100 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.675539+0000", 2024-04-19T01:05:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001142321 2024-04-19T01:05:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.675564+0000", 2024-04-19T01:05:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5126e-05 2024-04-19T01:05:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.675575+0000", 2024-04-19T01:05:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0951999999999999e-05 2024-04-19T01:05:06.101 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.675641+0000", 2024-04-19T01:05:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5914999999999995e-05 2024-04-19T01:05:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.675713+0000", 2024-04-19T01:05:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2169000000000006e-05 2024-04-19T01:05:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.676428+0000", 2024-04-19T01:05:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00071535000000000001 2024-04-19T01:05:06.102 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.103 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.103 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:06.103 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680290+0000", 2024-04-19T01:05:06.103 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003861532 2024-04-19T01:05:06.103 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.103 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.103 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:06.103 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.711050+0000", 2024-04-19T01:05:06.103 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.030759916000000002 2024-04-19T01:05:06.103 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.103 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.103 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:06.103 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.711106+0000", 2024-04-19T01:05:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.5757000000000002e-05 2024-04-19T01:05:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:331 3.ds0 3:bf91dcb5:::smithi15528762-2:head [append 0~507904] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:05:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.680101+0000", 2024-04-19T01:05:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 48.383340781000001, 2024-04-19T01:05:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.031204012, 2024-04-19T01:05:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:06.104 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 331 2024-04-19T01:05:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680101+0000", 2024-04-19T01:05:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680101+0000", 2024-04-19T01:05:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:06.105 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.106 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.106 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:06.106 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680103+0000", 2024-04-19T01:05:06.106 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.956e-06 2024-04-19T01:05:06.106 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.106 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.106 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:06.107 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680444+0000", 2024-04-19T01:05:06.107 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00034149199999999999 2024-04-19T01:05:06.107 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.107 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.108 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:06.108 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680460+0000", 2024-04-19T01:05:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5554e-05 2024-04-19T01:05:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680471+0000", 2024-04-19T01:05:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0852e-05 2024-04-19T01:05:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680539+0000", 2024-04-19T01:05:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.8619000000000001e-05 2024-04-19T01:05:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.109 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.680652+0000", 2024-04-19T01:05:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011258499999999999 2024-04-19T01:05:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.681392+0000", 2024-04-19T01:05:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00074039899999999998 2024-04-19T01:05:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.691230+0000", 2024-04-19T01:05:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0098373260000000004 2024-04-19T01:05:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.110 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.111 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:06.111 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.711264+0000", 2024-04-19T01:05:06.111 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.020034501999999999 2024-04-19T01:05:06.111 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.111 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.111 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:06.111 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.711305+0000", 2024-04-19T01:05:06.111 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0726999999999998e-05 2024-04-19T01:05:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:332 3.ds0 3:bf91dcb5:::smithi15528762-2:head [append 0~262144] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:05:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.682638+0000", 2024-04-19T01:05:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 48.380803677000003, 2024-04-19T01:05:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.029333649999999999, 2024-04-19T01:05:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:06.112 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 332 2024-04-19T01:05:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682638+0000", 2024-04-19T01:05:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682638+0000", 2024-04-19T01:05:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682640+0000", 2024-04-19T01:05:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5990000000000001e-06 2024-04-19T01:05:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682770+0000", 2024-04-19T01:05:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013076200000000001 2024-04-19T01:05:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682784+0000", 2024-04-19T01:05:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3786e-05 2024-04-19T01:05:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.114 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.115 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:06.115 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682792+0000", 2024-04-19T01:05:06.115 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.3000000000000002e-06 2024-04-19T01:05:06.115 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.115 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.115 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:06.115 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.686722+0000", 2024-04-19T01:05:06.115 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0039299749999999996 2024-04-19T01:05:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.686804+0000", 2024-04-19T01:05:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1588000000000006e-05 2024-04-19T01:05:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.687521+0000", 2024-04-19T01:05:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00071744300000000001 2024-04-19T01:05:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.116 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.117 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:06.117 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.691494+0000", 2024-04-19T01:05:06.117 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0039721239999999996 2024-04-19T01:05:06.117 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.117 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.117 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:06.118 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.711934+0000", 2024-04-19T01:05:06.118 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.020440649000000002 2024-04-19T01:05:06.118 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.118 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.118 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:06.118 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.711972+0000", 2024-04-19T01:05:06.118 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7423999999999999e-05 2024-04-19T01:05:06.118 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:06.118 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:06.118 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:06.118 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.119 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.119 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:333 3.ds0 3:bf91dcb5:::smithi15528762-2:head [setxattr _header (55) in=7b] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:05:06.119 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.682805+0000", 2024-04-19T01:05:06.120 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 48.380636275999997, 2024-04-19T01:05:06.120 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.029376039, 2024-04-19T01:05:06.120 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:06.120 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:06.120 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:06.120 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:06.120 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 333 2024-04-19T01:05:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682805+0000", 2024-04-19T01:05:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682805+0000", 2024-04-19T01:05:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:06.121 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682806+0000", 2024-04-19T01:05:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4099999999999999e-07 2024-04-19T01:05:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682815+0000", 2024-04-19T01:05:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.516e-06 2024-04-19T01:05:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682823+0000", 2024-04-19T01:05:06.122 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6459999999999993e-06 2024-04-19T01:05:06.123 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.123 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.123 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:06.123 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682832+0000", 2024-04-19T01:05:06.123 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.6260000000000001e-06 2024-04-19T01:05:06.123 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.123 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.123 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:06.123 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.690350+0000", 2024-04-19T01:05:06.123 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0075183730000000001 2024-04-19T01:05:06.123 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.123 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.124 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:06.124 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.690456+0000", 2024-04-19T01:05:06.124 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010617599999999999 2024-04-19T01:05:06.124 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.124 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.124 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:06.124 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.690951+0000", 2024-04-19T01:05:06.124 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00049519699999999995 2024-04-19T01:05:06.124 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.125 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.125 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:06.125 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.693060+0000", 2024-04-19T01:05:06.125 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0021084239999999998 2024-04-19T01:05:06.125 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.125 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.125 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:06.125 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.712088+0000", 2024-04-19T01:05:06.126 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.019028024000000001 2024-04-19T01:05:06.126 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.126 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.126 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:06.126 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.712181+0000", 2024-04-19T01:05:06.126 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.3516000000000004e-05 2024-04-19T01:05:06.126 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:06.126 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:06.126 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:06.126 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.126 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.126 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:334 3.ds0 3:bf91dcb5:::smithi15528762-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:05:06.126 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.682840+0000", 2024-04-19T01:05:06.127 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 48.380602127000003, 2024-04-19T01:05:06.127 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.031162179000000002, 2024-04-19T01:05:06.127 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:06.127 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:06.127 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:05:06.127 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:06.127 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:06.127 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:06.127 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 334 2024-04-19T01:05:06.127 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.127 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:06.127 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.127 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:06.127 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682840+0000", 2024-04-19T01:05:06.128 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:06.128 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.128 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.128 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:06.128 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682840+0000", 2024-04-19T01:05:06.128 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:06.128 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.128 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.128 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:06.128 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682840+0000", 2024-04-19T01:05:06.128 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4899999999999995e-07 2024-04-19T01:05:06.128 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.128 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.128 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:06.128 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682848+0000", 2024-04-19T01:05:06.129 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5270000000000001e-06 2024-04-19T01:05:06.129 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.129 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.129 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:06.129 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682855+0000", 2024-04-19T01:05:06.129 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.4889999999999998e-06 2024-04-19T01:05:06.129 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.129 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.129 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:06.129 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.682860+0000", 2024-04-19T01:05:06.129 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.164e-06 2024-04-19T01:05:06.129 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.129 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.129 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:06.130 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.691525+0000", 2024-04-19T01:05:06.130 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0086643239999999993 2024-04-19T01:05:06.130 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.130 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.130 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:05:06.130 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.691618+0000", 2024-04-19T01:05:06.130 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2983000000000001e-05 2024-04-19T01:05:06.130 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.130 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.131 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:06.131 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.712661+0000", 2024-04-19T01:05:06.131 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021043777 2024-04-19T01:05:06.131 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.131 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.131 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:06.131 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.712733+0000", 2024-04-19T01:05:06.131 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2063000000000004e-05 2024-04-19T01:05:06.131 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.131 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.131 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:06.131 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.714002+0000", 2024-04-19T01:05:06.131 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0012683029999999999 2024-04-19T01:05:06.132 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:06.132 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:06.132 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:06.132 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.132 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.132 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:354 3.as0 3:56c32ee7:::smithi15528762-47:head [copy-from ver 35] snapc 9=[8,7] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:05:06.132 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.544030+0000", 2024-04-19T01:05:06.132 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 47.519412099999997, 2024-04-19T01:05:06.132 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.049063250000000003, 2024-04-19T01:05:06.132 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:06.132 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:06.132 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:06.132 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:06.132 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 354 2024-04-19T01:05:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544030+0000", 2024-04-19T01:05:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544030+0000", 2024-04-19T01:05:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:06.133 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.134 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.134 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:06.134 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544033+0000", 2024-04-19T01:05:06.134 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.506e-06 2024-04-19T01:05:06.134 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.134 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.134 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:06.134 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544046+0000", 2024-04-19T01:05:06.134 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3208e-05 2024-04-19T01:05:06.135 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.135 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.135 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:06.135 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544082+0000", 2024-04-19T01:05:06.135 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5509999999999997e-05 2024-04-19T01:05:06.135 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.135 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.135 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:06.135 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544101+0000", 2024-04-19T01:05:06.135 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9500999999999998e-05 2024-04-19T01:05:06.135 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.135 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.135 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:06.136 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544230+0000", 2024-04-19T01:05:06.136 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000129114 2024-04-19T01:05:06.136 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.136 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.136 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:06.136 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544428+0000", 2024-04-19T01:05:06.136 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00019776799999999999 2024-04-19T01:05:06.136 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.136 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.136 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:06.136 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.556395+0000", 2024-04-19T01:05:06.136 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011967261 2024-04-19T01:05:06.136 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.136 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.137 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:06.137 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.574011+0000", 2024-04-19T01:05:06.137 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017615654000000001 2024-04-19T01:05:06.137 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.137 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.137 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:06.137 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.592920+0000", 2024-04-19T01:05:06.137 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018908812000000001 2024-04-19T01:05:06.137 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.137 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.137 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:06.137 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.593093+0000", 2024-04-19T01:05:06.137 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00017291600000000001 2024-04-19T01:05:06.137 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:06.137 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:06.138 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:06.138 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.138 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.138 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:355 3.as0 3:56c32ee7:::smithi15528762-47:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:05:06.138 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.544150+0000", 2024-04-19T01:05:06.138 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 47.519291328000001, 2024-04-19T01:05:06.138 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.029722035000000001, 2024-04-19T01:05:06.138 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:06.138 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:06.138 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:05:06.138 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:06.138 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:06.138 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:06.139 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 355 2024-04-19T01:05:06.139 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.139 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:06.139 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.139 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:06.139 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544150+0000", 2024-04-19T01:05:06.139 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:06.139 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.139 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.139 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:06.139 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544150+0000", 2024-04-19T01:05:06.139 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:06.139 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.139 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.140 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:06.140 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544152+0000", 2024-04-19T01:05:06.140 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.28e-06 2024-04-19T01:05:06.140 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.140 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.140 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:06.140 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544158+0000", 2024-04-19T01:05:06.140 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4160000000000001e-06 2024-04-19T01:05:06.140 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.140 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.140 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:06.140 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544178+0000", 2024-04-19T01:05:06.140 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0054000000000001e-05 2024-04-19T01:05:06.140 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.141 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.141 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:06.141 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544195+0000", 2024-04-19T01:05:06.141 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6835999999999999e-05 2024-04-19T01:05:06.141 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.141 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.141 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:06.142 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544717+0000", 2024-04-19T01:05:06.142 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00052258800000000005 2024-04-19T01:05:06.142 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.142 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.142 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for blocked object", 2024-04-19T01:05:06.142 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544835+0000", 2024-04-19T01:05:06.142 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000117448 2024-04-19T01:05:06.142 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.142 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.142 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:06.142 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.573609+0000", 2024-04-19T01:05:06.142 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.028774094 2024-04-19T01:05:06.142 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.143 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.143 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:06.143 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.573721+0000", 2024-04-19T01:05:06.143 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011205900000000001 2024-04-19T01:05:06.143 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.143 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.143 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:06.143 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.573872+0000", 2024-04-19T01:05:06.143 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015126000000000001 2024-04-19T01:05:06.143 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:06.143 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:06.143 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:06.143 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.143 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.144 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:478 3.as0 3:5a488dbb:::smithi15528762-13:head [append 0~770048] snapc 12=[12,11,10] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-19T01:05:06.144 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:35.631704+0000", 2024-04-19T01:05:06.144 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 30.4317381, 2024-04-19T01:05:06.144 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.031533347000000003, 2024-04-19T01:05:06.144 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:06.144 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:06.144 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:06.144 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:06.144 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:06.144 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:06.144 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 478 2024-04-19T01:05:06.144 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.144 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:06.144 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.145 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:06.145 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.631704+0000", 2024-04-19T01:05:06.145 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:06.145 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.145 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.145 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:06.145 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.631704+0000", 2024-04-19T01:05:06.145 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:06.146 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.146 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.146 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:06.146 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.631707+0000", 2024-04-19T01:05:06.146 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0800000000000002e-06 2024-04-19T01:05:06.146 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.146 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.146 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:06.146 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.632909+0000", 2024-04-19T01:05:06.146 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0012019070000000001 2024-04-19T01:05:06.146 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.146 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.146 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:06.147 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.632934+0000", 2024-04-19T01:05:06.147 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5289000000000001e-05 2024-04-19T01:05:06.147 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.147 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.147 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:06.147 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.632945+0000", 2024-04-19T01:05:06.147 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1046999999999999e-05 2024-04-19T01:05:06.147 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.147 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.147 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:06.147 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.633050+0000", 2024-04-19T01:05:06.147 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010516 2024-04-19T01:05:06.147 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.147 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.147 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op must wait for map", 2024-04-19T01:05:06.148 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.633065+0000", 2024-04-19T01:05:06.148 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4765e-05 2024-04-19T01:05:06.148 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.148 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.148 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:06.148 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.636005+0000", 2024-04-19T01:05:06.148 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0029403530000000002 2024-04-19T01:05:06.148 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.148 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.148 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:06.148 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.636188+0000", 2024-04-19T01:05:06.148 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00018290099999999999 2024-04-19T01:05:06.148 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.148 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.149 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:06.149 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.638553+0000", 2024-04-19T01:05:06.149 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0023651800000000001 2024-04-19T01:05:06.149 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.149 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.149 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:06.149 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.662256+0000", 2024-04-19T01:05:06.149 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.023702524999999999 2024-04-19T01:05:06.149 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.149 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.149 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:06.149 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.662993+0000", 2024-04-19T01:05:06.149 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00073735000000000001 2024-04-19T01:05:06.149 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.150 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.150 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:06.150 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.663237+0000", 2024-04-19T01:05:06.150 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00024379 2024-04-19T01:05:06.150 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:06.150 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:06.150 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:06.150 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.150 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.150 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:479 3.as0 3:5a488dbb:::smithi15528762-13:head [append 0~475136] snapc 12=[12,11,10] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-19T01:05:06.150 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:35.633850+0000", 2024-04-19T01:05:06.150 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 30.429591842000001, 2024-04-19T01:05:06.150 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.042497337000000003, 2024-04-19T01:05:06.150 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:06.151 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:06.151 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:06.151 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:06.151 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:06.151 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:06.151 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 479 2024-04-19T01:05:06.151 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.151 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:06.152 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.152 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:06.152 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.633850+0000", 2024-04-19T01:05:06.152 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:06.152 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.152 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.152 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:06.152 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.633850+0000", 2024-04-19T01:05:06.152 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:06.152 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.152 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.152 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:06.152 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.633852+0000", 2024-04-19T01:05:06.153 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.26e-06 2024-04-19T01:05:06.153 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.153 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.153 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:06.153 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.634421+0000", 2024-04-19T01:05:06.153 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000569058 2024-04-19T01:05:06.153 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.153 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.153 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:06.153 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.634441+0000", 2024-04-19T01:05:06.153 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0169e-05 2024-04-19T01:05:06.153 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.153 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.153 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:06.153 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.634452+0000", 2024-04-19T01:05:06.154 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0963e-05 2024-04-19T01:05:06.154 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.154 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.154 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:06.154 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.634540+0000", 2024-04-19T01:05:06.154 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.7563000000000002e-05 2024-04-19T01:05:06.154 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.154 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.154 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:05:06.154 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.634550+0000", 2024-04-19T01:05:06.154 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0491e-05 2024-04-19T01:05:06.154 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.154 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.155 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:06.155 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.653619+0000", 2024-04-19T01:05:06.155 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.019068284000000001 2024-04-19T01:05:06.155 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.155 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.155 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:06.155 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.653716+0000", 2024-04-19T01:05:06.155 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7546000000000004e-05 2024-04-19T01:05:06.155 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.156 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.156 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:06.156 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.654525+0000", 2024-04-19T01:05:06.156 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00080861300000000002 2024-04-19T01:05:06.156 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.156 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.156 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:06.156 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.674163+0000", 2024-04-19T01:05:06.156 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.019638666999999999 2024-04-19T01:05:06.156 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.156 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.156 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:06.156 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.676228+0000", 2024-04-19T01:05:06.157 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0020650379999999999 2024-04-19T01:05:06.157 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.157 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.157 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:06.157 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.676347+0000", 2024-04-19T01:05:06.157 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011868500000000001 2024-04-19T01:05:06.157 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:06.157 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:06.157 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:06.157 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.157 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.157 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:480 3.as0 3:5a488dbb:::smithi15528762-13:head [append 0~294912] snapc 12=[12,11,10] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-19T01:05:06.157 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:35.639539+0000", 2024-04-19T01:05:06.157 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 30.423902208000001, 2024-04-19T01:05:06.157 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.038046241000000001, 2024-04-19T01:05:06.158 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:06.158 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:06.158 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:06.158 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:06.158 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:06.158 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:06.158 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 480 2024-04-19T01:05:06.158 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.158 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:06.158 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.158 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:06.158 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639539+0000", 2024-04-19T01:05:06.158 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:06.158 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.159 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.159 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:06.159 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639539+0000", 2024-04-19T01:05:06.159 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:06.159 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.159 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.159 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:06.159 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639541+0000", 2024-04-19T01:05:06.159 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.922e-06 2024-04-19T01:05:06.159 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.159 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.159 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:06.159 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639847+0000", 2024-04-19T01:05:06.159 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00030593599999999998 2024-04-19T01:05:06.160 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.160 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.160 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:06.160 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639886+0000", 2024-04-19T01:05:06.160 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9171000000000002e-05 2024-04-19T01:05:06.160 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.160 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.160 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:06.160 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639898+0000", 2024-04-19T01:05:06.160 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1103e-05 2024-04-19T01:05:06.160 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.160 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.160 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:06.160 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.663566+0000", 2024-04-19T01:05:06.160 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.023668392 2024-04-19T01:05:06.161 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.161 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.161 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:06.161 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.663853+0000", 2024-04-19T01:05:06.161 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00028731200000000002 2024-04-19T01:05:06.161 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.161 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.161 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:06.161 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.665713+0000", 2024-04-19T01:05:06.161 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001860109 2024-04-19T01:05:06.161 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.161 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.162 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:06.162 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.675595+0000", 2024-04-19T01:05:06.162 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0098813939999999999 2024-04-19T01:05:06.162 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.162 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.162 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:06.162 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.677466+0000", 2024-04-19T01:05:06.162 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0018710199999999999 2024-04-19T01:05:06.162 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.163 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.163 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:06.163 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.677586+0000", 2024-04-19T01:05:06.163 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000119882 2024-04-19T01:05:06.163 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:06.163 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:06.163 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:06.163 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.163 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.163 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:481 3.as0 3:5a488dbb:::smithi15528762-13:head [setxattr _header (57) in=7b] snapc 12=[12,11,10] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-19T01:05:06.163 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:35.639915+0000", 2024-04-19T01:05:06.163 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 30.423526986999999, 2024-04-19T01:05:06.163 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.038026319000000003, 2024-04-19T01:05:06.163 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:06.164 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:06.164 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:06.164 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:06.164 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:06.164 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:06.164 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 481 2024-04-19T01:05:06.164 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.164 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:06.164 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.164 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:06.164 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639915+0000", 2024-04-19T01:05:06.164 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:06.164 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.164 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.164 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:06.165 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639915+0000", 2024-04-19T01:05:06.165 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:06.165 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.165 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.165 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:06.165 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639915+0000", 2024-04-19T01:05:06.165 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.0800000000000004e-07 2024-04-19T01:05:06.165 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.165 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.165 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:06.165 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639919+0000", 2024-04-19T01:05:06.166 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0589999999999996e-06 2024-04-19T01:05:06.166 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.166 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.166 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:06.166 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639929+0000", 2024-04-19T01:05:06.166 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.9590000000000007e-06 2024-04-19T01:05:06.166 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.166 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.166 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:06.166 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639936+0000", 2024-04-19T01:05:06.166 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2480000000000001e-06 2024-04-19T01:05:06.167 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.167 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.167 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:06.167 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.674290+0000", 2024-04-19T01:05:06.167 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.034354279000000001 2024-04-19T01:05:06.167 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.167 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.167 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:06.167 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.674404+0000", 2024-04-19T01:05:06.167 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000114189 2024-04-19T01:05:06.167 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.167 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.167 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:06.167 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.675012+0000", 2024-04-19T01:05:06.168 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00060776199999999997 2024-04-19T01:05:06.168 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.168 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.168 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:06.168 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.676365+0000", 2024-04-19T01:05:06.168 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0013529849999999999 2024-04-19T01:05:06.168 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.168 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.168 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:06.168 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.677805+0000", 2024-04-19T01:05:06.168 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014399160000000001 2024-04-19T01:05:06.168 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.168 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.168 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:06.169 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.677941+0000", 2024-04-19T01:05:06.169 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013621400000000001 2024-04-19T01:05:06.169 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:06.169 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:06.169 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:06.169 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.169 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.169 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:482 3.as0 3:5a488dbb:::smithi15528762-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-04-19T01:05:06.169 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:35.639943+0000", 2024-04-19T01:05:06.169 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 30.423498789, 2024-04-19T01:05:06.169 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.040997591999999999, 2024-04-19T01:05:06.169 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:06.169 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:06.170 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:05:06.170 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:06.170 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:06.170 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:06.170 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 482 2024-04-19T01:05:06.170 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.170 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:06.170 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.170 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:06.170 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639943+0000", 2024-04-19T01:05:06.170 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:06.170 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.170 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.171 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:06.171 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639943+0000", 2024-04-19T01:05:06.171 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:06.171 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.171 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.171 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:06.171 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639943+0000", 2024-04-19T01:05:06.171 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.89e-07 2024-04-19T01:05:06.171 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.171 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.171 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:06.171 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639945+0000", 2024-04-19T01:05:06.171 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9870000000000002e-06 2024-04-19T01:05:06.171 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.172 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.172 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:06.172 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639951+0000", 2024-04-19T01:05:06.172 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.9479999999999996e-06 2024-04-19T01:05:06.172 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.172 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.172 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:06.172 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639956+0000", 2024-04-19T01:05:06.173 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8040000000000001e-06 2024-04-19T01:05:06.173 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.173 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.173 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:06.173 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.676400+0000", 2024-04-19T01:05:06.173 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.036444184999999997 2024-04-19T01:05:06.173 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.173 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.173 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:05:06.173 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.676473+0000", 2024-04-19T01:05:06.173 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2704000000000001e-05 2024-04-19T01:05:06.173 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.173 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.174 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:06.174 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.678615+0000", 2024-04-19T01:05:06.174 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0021422030000000001 2024-04-19T01:05:06.174 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.174 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.174 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:06.174 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.678811+0000", 2024-04-19T01:05:06.174 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000196026 2024-04-19T01:05:06.174 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.174 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.174 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:06.174 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.680940+0000", 2024-04-19T01:05:06.174 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0021293459999999998 2024-04-19T01:05:06.174 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:06.175 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:06.175 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:06.175 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.175 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.175 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:520 3.as0 3:58da707e:::smithi15528762-12:head [copy-from ver 38] snapc 16=[16,15,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-04-19T01:05:06.175 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:41.512306+0000", 2024-04-19T01:05:06.175 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 24.551135865999999, 2024-04-19T01:05:06.175 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.089648318000000005, 2024-04-19T01:05:06.175 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:06.175 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:06.175 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:06.175 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:06.175 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:06.175 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:06.175 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 520 2024-04-19T01:05:06.176 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.176 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:06.176 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.176 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:06.176 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512306+0000", 2024-04-19T01:05:06.176 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:06.176 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.177 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.177 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:06.177 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512306+0000", 2024-04-19T01:05:06.177 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:06.177 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.177 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.177 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:06.177 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512309+0000", 2024-04-19T01:05:06.177 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2509999999999999e-06 2024-04-19T01:05:06.177 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.177 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.177 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:06.177 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512326+0000", 2024-04-19T01:05:06.177 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6881999999999999e-05 2024-04-19T01:05:06.178 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.178 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.178 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:06.178 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512356+0000", 2024-04-19T01:05:06.178 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9920000000000002e-05 2024-04-19T01:05:06.178 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.178 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.178 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:06.178 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512371+0000", 2024-04-19T01:05:06.178 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5325999999999999e-05 2024-04-19T01:05:06.179 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.179 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.179 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:06.179 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512489+0000", 2024-04-19T01:05:06.179 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011835100000000001 2024-04-19T01:05:06.179 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.179 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.179 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op must wait for map", 2024-04-19T01:05:06.179 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512500+0000", 2024-04-19T01:05:06.179 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0437e-05 2024-04-19T01:05:06.179 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.179 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.179 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:06.179 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.514800+0000", 2024-04-19T01:05:06.180 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0023000719999999998 2024-04-19T01:05:06.180 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.180 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.180 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:06.180 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.514923+0000", 2024-04-19T01:05:06.180 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012253599999999999 2024-04-19T01:05:06.180 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.180 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.180 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:06.180 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.541802+0000", 2024-04-19T01:05:06.180 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.026879473000000001 2024-04-19T01:05:06.180 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.180 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.180 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:06.181 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.564261+0000", 2024-04-19T01:05:06.181 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.022458887 2024-04-19T01:05:06.181 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.181 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.181 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:06.181 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.601886+0000", 2024-04-19T01:05:06.181 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.037625204000000002 2024-04-19T01:05:06.181 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.181 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.181 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:06.181 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.601954+0000", 2024-04-19T01:05:06.181 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.7978999999999994e-05 2024-04-19T01:05:06.181 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:06.181 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:06.182 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:06.182 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.182 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.182 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:521 3.as0 3:58da707e:::smithi15528762-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e71)", 2024-04-19T01:05:06.182 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:41.512407+0000", 2024-04-19T01:05:06.182 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 24.551034443999999, 2024-04-19T01:05:06.182 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.051806598000000002, 2024-04-19T01:05:06.182 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:06.183 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:06.183 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:05:06.183 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:06.183 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:06.183 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:06.183 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 521 2024-04-19T01:05:06.183 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.183 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:06.183 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.183 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:06.183 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512407+0000", 2024-04-19T01:05:06.183 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:06.184 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.184 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.184 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:06.184 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512407+0000", 2024-04-19T01:05:06.184 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:06.184 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.184 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.184 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:06.184 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512408+0000", 2024-04-19T01:05:06.184 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.4799999999999997e-07 2024-04-19T01:05:06.184 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.184 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.184 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:06.184 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512411+0000", 2024-04-19T01:05:06.185 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5109999999999999e-06 2024-04-19T01:05:06.185 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.185 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.185 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:06.185 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512423+0000", 2024-04-19T01:05:06.185 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1188e-05 2024-04-19T01:05:06.185 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.185 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.185 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:06.185 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512430+0000", 2024-04-19T01:05:06.185 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.3540000000000002e-06 2024-04-19T01:05:06.185 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.185 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.185 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:06.186 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512598+0000", 2024-04-19T01:05:06.186 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00016804 2024-04-19T01:05:06.186 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.186 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.186 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:05:06.186 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512605+0000", 2024-04-19T01:05:06.186 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.9290000000000001e-06 2024-04-19T01:05:06.186 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.186 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.187 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:06.187 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.515130+0000", 2024-04-19T01:05:06.187 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0025246299999999999 2024-04-19T01:05:06.187 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.187 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.187 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for blocked object", 2024-04-19T01:05:06.187 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.515181+0000", 2024-04-19T01:05:06.187 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.1100000000000002e-05 2024-04-19T01:05:06.187 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.187 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.187 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:06.187 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.563980+0000", 2024-04-19T01:05:06.187 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.048799373 2024-04-19T01:05:06.188 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.188 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.188 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:06.188 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.564087+0000", 2024-04-19T01:05:06.188 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000106827 2024-04-19T01:05:06.188 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.188 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.188 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:06.188 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.564214+0000", 2024-04-19T01:05:06.188 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012689799999999999 2024-04-19T01:05:06.188 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:06.188 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:06.188 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:06.188 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.188 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.189 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:522 3.as0 3:5a488dbb:::smithi15528762-13:head [copy-from ver 2] snapc 16=[16,15,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-04-19T01:05:06.189 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:41.512460+0000", 2024-04-19T01:05:06.189 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 24.550981895, 2024-04-19T01:05:06.189 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.022697987999999999, 2024-04-19T01:05:06.189 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:06.189 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:06.189 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:06.189 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:06.189 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:06.189 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:06.189 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 522 2024-04-19T01:05:06.189 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.189 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:06.189 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.190 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:06.190 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512460+0000", 2024-04-19T01:05:06.190 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:06.190 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.190 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.190 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:06.190 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512460+0000", 2024-04-19T01:05:06.190 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:06.190 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.190 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.190 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:06.190 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512460+0000", 2024-04-19T01:05:06.190 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4899999999999995e-07 2024-04-19T01:05:06.190 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.191 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.191 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:06.191 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512465+0000", 2024-04-19T01:05:06.191 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5560000000000001e-06 2024-04-19T01:05:06.191 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.191 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.191 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:06.191 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512474+0000", 2024-04-19T01:05:06.191 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9050000000000003e-06 2024-04-19T01:05:06.191 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.191 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.191 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:06.191 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512482+0000", 2024-04-19T01:05:06.191 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7819999999999994e-06 2024-04-19T01:05:06.192 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.192 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.192 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:06.192 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512683+0000", 2024-04-19T01:05:06.192 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00020168000000000001 2024-04-19T01:05:06.192 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.192 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.192 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:05:06.192 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512691+0000", 2024-04-19T01:05:06.192 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.9710000000000004e-06 2024-04-19T01:05:06.192 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.193 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.193 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:06.194 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.515212+0000", 2024-04-19T01:05:06.194 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.002520567 2024-04-19T01:05:06.194 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.194 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.194 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:06.194 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.515359+0000", 2024-04-19T01:05:06.194 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014727300000000001 2024-04-19T01:05:06.194 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.194 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.194 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:06.194 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.524092+0000", 2024-04-19T01:05:06.194 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0087328500000000003 2024-04-19T01:05:06.194 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.194 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.195 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:06.195 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.528284+0000", 2024-04-19T01:05:06.195 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0041917730000000002 2024-04-19T01:05:06.195 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.195 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.195 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:06.195 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.535055+0000", 2024-04-19T01:05:06.195 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0067715939999999997 2024-04-19T01:05:06.195 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.195 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.195 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:06.195 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.535158+0000", 2024-04-19T01:05:06.195 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000102388 2024-04-19T01:05:06.195 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:06.196 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:06.196 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:06.196 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.196 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.196 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:523 3.as0 3:5a488dbb:::smithi15528762-13:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e71)", 2024-04-19T01:05:06.196 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:41.512492+0000", 2024-04-19T01:05:06.196 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 24.550949938999999, 2024-04-19T01:05:06.196 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015724650999999999, 2024-04-19T01:05:06.196 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:06.196 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:06.196 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:05:06.196 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:06.196 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:06.196 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:06.196 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 523 2024-04-19T01:05:06.197 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.197 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:06.197 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.197 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:06.197 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512492+0000", 2024-04-19T01:05:06.197 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:06.197 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.197 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.197 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:06.198 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512492+0000", 2024-04-19T01:05:06.198 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:06.198 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.198 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.198 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:06.198 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512492+0000", 2024-04-19T01:05:06.198 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2500000000000001e-07 2024-04-19T01:05:06.198 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.198 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.198 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:06.198 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512495+0000", 2024-04-19T01:05:06.198 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7669999999999999e-06 2024-04-19T01:05:06.198 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.199 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.199 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:06.199 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512503+0000", 2024-04-19T01:05:06.199 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.3599999999999996e-06 2024-04-19T01:05:06.199 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.199 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.199 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:06.199 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512509+0000", 2024-04-19T01:05:06.199 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.772e-06 2024-04-19T01:05:06.199 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.199 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.199 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:06.199 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512981+0000", 2024-04-19T01:05:06.199 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00047226000000000001 2024-04-19T01:05:06.199 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.200 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.200 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:05:06.200 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512988+0000", 2024-04-19T01:05:06.200 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.9430000000000003e-06 2024-04-19T01:05:06.200 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.200 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.200 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:06.200 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.515588+0000", 2024-04-19T01:05:06.200 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0025993380000000001 2024-04-19T01:05:06.200 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.200 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.200 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for blocked object", 2024-04-19T01:05:06.200 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.515624+0000", 2024-04-19T01:05:06.200 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.6023999999999999e-05 2024-04-19T01:05:06.201 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.201 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.201 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:06.201 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.527975+0000", 2024-04-19T01:05:06.201 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012351735000000001 2024-04-19T01:05:06.201 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.201 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.201 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:06.201 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.528083+0000", 2024-04-19T01:05:06.201 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000107822 2024-04-19T01:05:06.201 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:06.201 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:06.201 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:06.201 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.528216+0000", 2024-04-19T01:05:06.202 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013320500000000001 2024-04-19T01:05:06.202 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:06.202 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:06.202 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:06.202 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:06.202 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:06.202 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:05:06.204 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-19T01:05:06.204 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:05:06.204+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:05:06.304 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:05:06.301+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:05:06.334 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:05:06.335 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [], 2024-04-19T01:05:06.335 INFO:teuthology.orchestra.run.smithi110.stdout: "num_ops": 0 2024-04-19T01:05:06.335 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:05:06.350 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-19T01:05:06.405 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:05:06.404+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:05:06.471 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:05:06.471 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [], 2024-04-19T01:05:06.471 INFO:teuthology.orchestra.run.smithi110.stdout: "complaint_time": 30, 2024-04-19T01:05:06.471 INFO:teuthology.orchestra.run.smithi110.stdout: "num_blocked_ops": 0 2024-04-19T01:05:06.471 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:05:06.485 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-19T01:05:06.507 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:05:06.506+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:05:06.523 INFO:tasks.rados.rados.0.smithi155.stdout:299: done (0 left) 2024-04-19T01:05:06.523 INFO:tasks.rados.rados.0.smithi155.stdout:300: read oid 34 snap 29 2024-04-19T01:05:06.523 INFO:tasks.rados.rados.0.smithi155.stdout:300: expect (ObjNum 66 snap 6 seq_num 66) 2024-04-19T01:05:06.523 INFO:tasks.rados.rados.0.smithi155.stdout:301: snap_create 2024-04-19T01:05:06.584 INFO:tasks.rados.rados.0.smithi155.stdout:300: expect (ObjNum 66 snap 6 seq_num 66) 2024-04-19T01:05:06.608 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:05:06.605+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:05:06.651 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:05:06.651 INFO:teuthology.orchestra.run.smithi110.stdout: "size": 20, 2024-04-19T01:05:06.651 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 600, 2024-04-19T01:05:06.651 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [ 2024-04-19T01:05:06.651 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.651 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs5 43/31 ECSubWrite(tid=47, reqid=client.4604.0:263, at_version=43'8, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:05:06.651 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.993731+0000", 2024-04-19T01:05:06.651 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 52.624459223999999, 2024-04-19T01:05:06.651 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.025167086000000002, 2024-04-19T01:05:06.651 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:06.651 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:06.652 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:06.652 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:06.652 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.652 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:06.652 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.993731+0000", 2024-04-19T01:05:06.652 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:06.652 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.652 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.652 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:06.652 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.993731+0000", 2024-04-19T01:05:06.652 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:06.652 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.652 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.653 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:06.653 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.993733+0000", 2024-04-19T01:05:06.653 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.114e-06 2024-04-19T01:05:06.653 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.653 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.653 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:06.653 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.994282+0000", 2024-04-19T01:05:06.653 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00054962800000000003 2024-04-19T01:05:06.653 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.653 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.653 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:06.653 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.994320+0000", 2024-04-19T01:05:06.653 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.7824999999999998e-05 2024-04-19T01:05:06.654 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.654 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.654 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:06.654 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.994345+0000", 2024-04-19T01:05:06.654 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4295e-05 2024-04-19T01:05:06.654 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.654 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.654 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:06.654 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.994505+0000", 2024-04-19T01:05:06.654 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00016080700000000001 2024-04-19T01:05:06.655 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.655 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.655 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:06.655 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.994525+0000", 2024-04-19T01:05:06.655 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0049000000000001e-05 2024-04-19T01:05:06.655 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.655 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.655 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:06.655 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.018713+0000", 2024-04-19T01:05:06.655 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.024187810000000001 2024-04-19T01:05:06.655 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.655 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.655 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:06.655 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.018898+0000", 2024-04-19T01:05:06.656 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000184558 2024-04-19T01:05:06.656 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:06.656 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:06.656 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:06.656 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.656 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.657 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs5 43/31 ECSubWrite(tid=49, reqid=client.4604.0:265, at_version=43'10, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:05:06.657 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.006353+0000", 2024-04-19T01:05:06.666 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 52.611836566000001, 2024-04-19T01:05:06.667 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.024130236999999999, 2024-04-19T01:05:06.667 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:06.667 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:06.667 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:06.667 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:06.670 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.670 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:06.670 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006353+0000", 2024-04-19T01:05:06.670 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:06.670 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.670 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.671 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:06.671 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006353+0000", 2024-04-19T01:05:06.671 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:06.671 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.672 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.672 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:06.672 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006354+0000", 2024-04-19T01:05:06.672 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.0899999999999999e-06 2024-04-19T01:05:06.672 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.672 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.672 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:06.673 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006393+0000", 2024-04-19T01:05:06.673 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.8897999999999999e-05 2024-04-19T01:05:06.673 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.673 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.673 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:06.673 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006417+0000", 2024-04-19T01:05:06.673 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3881999999999999e-05 2024-04-19T01:05:06.673 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.673 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.673 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:06.673 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006436+0000", 2024-04-19T01:05:06.673 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.8488999999999999e-05 2024-04-19T01:05:06.673 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.673 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.674 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:06.674 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.029248+0000", 2024-04-19T01:05:06.674 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.022812136 2024-04-19T01:05:06.674 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.674 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.674 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:06.674 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.029278+0000", 2024-04-19T01:05:06.674 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.9702000000000001e-05 2024-04-19T01:05:06.674 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.674 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.674 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:06.676 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.030387+0000", 2024-04-19T01:05:06.677 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001109814 2024-04-19T01:05:06.677 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.677 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.677 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:06.677 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.030484+0000", 2024-04-19T01:05:06.677 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.6225999999999997e-05 2024-04-19T01:05:06.677 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:06.677 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:06.677 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:06.677 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.677 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.677 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:310 3.0s0 3:0abdaea3:::smithi15528762-5:head [setxattr _header (55) in=7b] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:05:06.677 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:17.564685+0000", 2024-04-19T01:05:06.677 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 49.053504797999999, 2024-04-19T01:05:06.678 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.022864789999999999, 2024-04-19T01:05:06.678 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:06.678 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:06.678 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:06.678 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:06.682 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:06.682 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:06.682 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 310 2024-04-19T01:05:06.682 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.682 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:06.682 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.682 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:06.682 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564685+0000", 2024-04-19T01:05:06.682 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:06.682 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.682 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.683 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:06.683 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564685+0000", 2024-04-19T01:05:06.683 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:06.683 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.683 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.683 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:06.683 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564686+0000", 2024-04-19T01:05:06.683 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.0999999999999997e-07 2024-04-19T01:05:06.683 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.683 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.683 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:06.683 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564690+0000", 2024-04-19T01:05:06.683 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.3100000000000002e-06 2024-04-19T01:05:06.684 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.684 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.684 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:06.684 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564699+0000", 2024-04-19T01:05:06.684 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.6640000000000004e-06 2024-04-19T01:05:06.684 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.684 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.684 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:06.684 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564705+0000", 2024-04-19T01:05:06.684 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.4910000000000002e-06 2024-04-19T01:05:06.684 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.684 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.684 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:06.684 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.575044+0000", 2024-04-19T01:05:06.685 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.010338293 2024-04-19T01:05:06.685 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.685 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.690 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:06.691 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.575112+0000", 2024-04-19T01:05:06.691 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.7898999999999995e-05 2024-04-19T01:05:06.691 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.691 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.691 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:06.691 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.575501+0000", 2024-04-19T01:05:06.691 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000389491 2024-04-19T01:05:06.691 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.691 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.691 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:06.691 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.576333+0000", 2024-04-19T01:05:06.691 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00083234599999999998 2024-04-19T01:05:06.691 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.692 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.692 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:06.692 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.587286+0000", 2024-04-19T01:05:06.692 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.010952437000000001 2024-04-19T01:05:06.692 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.692 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.692 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:06.692 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.587550+0000", 2024-04-19T01:05:06.692 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00026404899999999998 2024-04-19T01:05:06.692 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:06.692 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:06.692 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:06.692 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.693 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.693 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:311 3.0s0 3:0abdaea3:::smithi15528762-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:05:06.693 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:17.564725+0000", 2024-04-19T01:05:06.693 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 49.053464898000001, 2024-04-19T01:05:06.693 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.026756121000000001, 2024-04-19T01:05:06.693 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:06.693 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:06.693 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:05:06.693 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:06.693 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:06.693 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:06.693 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 311 2024-04-19T01:05:06.693 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.693 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:06.694 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.694 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:06.694 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564725+0000", 2024-04-19T01:05:06.694 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:06.694 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.694 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.694 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:06.694 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564725+0000", 2024-04-19T01:05:06.694 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:06.694 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.694 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.694 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:06.694 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564726+0000", 2024-04-19T01:05:06.694 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.5700000000000002e-07 2024-04-19T01:05:06.695 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.695 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.695 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:06.695 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564730+0000", 2024-04-19T01:05:06.695 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.476e-06 2024-04-19T01:05:06.695 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.695 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.695 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:06.695 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564740+0000", 2024-04-19T01:05:06.695 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.2750000000000001e-06 2024-04-19T01:05:06.695 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.695 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.695 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:06.695 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.564749+0000", 2024-04-19T01:05:06.696 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.1640000000000007e-06 2024-04-19T01:05:06.696 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.696 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.696 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:06.696 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.576057+0000", 2024-04-19T01:05:06.696 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.011308573000000001 2024-04-19T01:05:06.696 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.696 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.696 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting for rw locks", 2024-04-19T01:05:06.696 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.576308+0000", 2024-04-19T01:05:06.696 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00025028300000000002 2024-04-19T01:05:06.696 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.696 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.696 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:06.697 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.588445+0000", 2024-04-19T01:05:06.697 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.012137129 2024-04-19T01:05:06.697 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.697 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.697 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:06.697 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.588680+0000", 2024-04-19T01:05:06.697 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00023515400000000001 2024-04-19T01:05:06.697 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.697 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.697 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:06.697 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:17.591481+0000", 2024-04-19T01:05:06.697 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00280131 2024-04-19T01:05:06.697 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:06.697 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:06.698 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:06.698 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.698 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.698 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs5 48/31 ECSubWrite(tid=76, reqid=client.4604.0:345, at_version=48'22, trim_to=0'0, roll_forward_to=47'19))", 2024-04-19T01:05:06.698 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.538139+0000", 2024-04-19T01:05:06.698 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 48.080050870000001, 2024-04-19T01:05:06.698 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.030818128, 2024-04-19T01:05:06.698 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:06.698 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:06.698 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:06.698 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:06.698 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.698 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:06.698 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538139+0000", 2024-04-19T01:05:06.699 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:06.699 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.699 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.699 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:06.699 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538139+0000", 2024-04-19T01:05:06.699 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:06.699 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.699 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.699 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:06.699 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538141+0000", 2024-04-19T01:05:06.699 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.796e-06 2024-04-19T01:05:06.699 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.699 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.699 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:06.701 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538876+0000", 2024-04-19T01:05:06.701 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00073526699999999995 2024-04-19T01:05:06.701 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.701 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.701 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:06.701 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538919+0000", 2024-04-19T01:05:06.701 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.3065000000000002e-05 2024-04-19T01:05:06.701 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.701 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.702 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:06.702 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538943+0000", 2024-04-19T01:05:06.702 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3578999999999999e-05 2024-04-19T01:05:06.702 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.702 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.702 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:06.702 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.539114+0000", 2024-04-19T01:05:06.702 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000171383 2024-04-19T01:05:06.702 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.702 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.702 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:06.702 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.539140+0000", 2024-04-19T01:05:06.702 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5709000000000001e-05 2024-04-19T01:05:06.703 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.703 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.703 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:06.703 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.568880+0000", 2024-04-19T01:05:06.703 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.029740128000000001 2024-04-19T01:05:06.703 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.703 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.703 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:06.703 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.568957+0000", 2024-04-19T01:05:06.703 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.7200999999999995e-05 2024-04-19T01:05:06.703 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:06.703 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:06.703 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:06.704 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.704 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.704 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs5 48/31 ECSubWrite(tid=78, reqid=client.4604.0:346, at_version=48'23, trim_to=0'0, roll_forward_to=48'21))", 2024-04-19T01:05:06.704 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.548746+0000", 2024-04-19T01:05:06.704 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 48.069444195000003, 2024-04-19T01:05:06.704 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.023711396999999999, 2024-04-19T01:05:06.704 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:06.704 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:06.704 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:06.704 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:06.704 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.704 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:06.704 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.548746+0000", 2024-04-19T01:05:06.704 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:06.705 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.705 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.705 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:06.705 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.548746+0000", 2024-04-19T01:05:06.705 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:06.705 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.705 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.705 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:06.705 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.548747+0000", 2024-04-19T01:05:06.705 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5799999999999999e-06 2024-04-19T01:05:06.705 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.705 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.705 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:06.705 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.549144+0000", 2024-04-19T01:05:06.705 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00039714900000000002 2024-04-19T01:05:06.706 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.706 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.706 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:06.706 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.549174+0000", 2024-04-19T01:05:06.706 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.9652999999999999e-05 2024-04-19T01:05:06.706 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.706 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.706 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:06.706 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.549195+0000", 2024-04-19T01:05:06.706 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0585e-05 2024-04-19T01:05:06.706 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.706 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.706 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:06.706 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.554787+0000", 2024-04-19T01:05:06.707 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0055927160000000002 2024-04-19T01:05:06.707 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.707 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.707 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:06.707 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.554813+0000", 2024-04-19T01:05:06.707 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5669000000000001e-05 2024-04-19T01:05:06.707 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.707 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.707 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:06.707 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.572370+0000", 2024-04-19T01:05:06.707 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.017556852000000001 2024-04-19T01:05:06.707 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.707 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.708 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:06.708 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.572457+0000", 2024-04-19T01:05:06.708 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.7193000000000004e-05 2024-04-19T01:05:06.708 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:06.708 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:06.708 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:06.708 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.708 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.708 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.as5 48/31 ECSubWrite(tid=71, reqid=client.4604.0:354, at_version=48'10, trim_to=0'0, roll_forward_to=34'8))", 2024-04-19T01:05:06.708 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.557068+0000", 2024-04-19T01:05:06.708 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 48.061122171999997, 2024-04-19T01:05:06.708 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.029586061, 2024-04-19T01:05:06.708 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:06.708 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:06.709 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:06.709 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:06.709 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.709 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:06.709 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557068+0000", 2024-04-19T01:05:06.709 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:06.709 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.709 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.709 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:06.709 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557068+0000", 2024-04-19T01:05:06.709 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:06.709 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.709 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.709 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:06.710 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557069+0000", 2024-04-19T01:05:06.710 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.9500000000000001e-07 2024-04-19T01:05:06.710 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.710 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.710 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:06.710 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.558172+0000", 2024-04-19T01:05:06.710 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001103637 2024-04-19T01:05:06.712 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.712 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.712 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:06.712 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.558206+0000", 2024-04-19T01:05:06.712 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.4069000000000002e-05 2024-04-19T01:05:06.712 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.712 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.712 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:06.712 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.558229+0000", 2024-04-19T01:05:06.712 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2495999999999999e-05 2024-04-19T01:05:06.712 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.712 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.712 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:06.712 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.558450+0000", 2024-04-19T01:05:06.713 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000221747 2024-04-19T01:05:06.713 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.713 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.713 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:06.713 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.558476+0000", 2024-04-19T01:05:06.713 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5755999999999999e-05 2024-04-19T01:05:06.713 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.713 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.713 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:06.713 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.586611+0000", 2024-04-19T01:05:06.713 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.028134731999999999 2024-04-19T01:05:06.713 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.713 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.713 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:06.714 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.586654+0000", 2024-04-19T01:05:06.714 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.2828999999999999e-05 2024-04-19T01:05:06.714 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:06.714 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:06.714 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:06.714 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.714 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.714 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.8s2 49/31 ECSubWrite(tid=203, reqid=client.4604.0:360, at_version=49'25, trim_to=0'0, roll_forward_to=49'24))", 2024-04-19T01:05:06.714 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:19.538361+0000", 2024-04-19T01:05:06.714 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 47.079828736000003, 2024-04-19T01:05:06.714 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.028842518000000001, 2024-04-19T01:05:06.714 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:06.714 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:06.715 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:06.715 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:06.715 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.715 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:06.715 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.538361+0000", 2024-04-19T01:05:06.715 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:06.715 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.715 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.715 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:06.715 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.538361+0000", 2024-04-19T01:05:06.715 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:06.715 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.715 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.715 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:06.715 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.538363+0000", 2024-04-19T01:05:06.716 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7600000000000001e-06 2024-04-19T01:05:06.716 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.716 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.716 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:06.716 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.540481+0000", 2024-04-19T01:05:06.716 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.002117833 2024-04-19T01:05:06.716 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.716 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.716 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:06.716 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.540516+0000", 2024-04-19T01:05:06.716 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.5435000000000002e-05 2024-04-19T01:05:06.716 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.716 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.716 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:06.717 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.540539+0000", 2024-04-19T01:05:06.717 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3136e-05 2024-04-19T01:05:06.717 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.717 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.717 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:06.717 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.540906+0000", 2024-04-19T01:05:06.717 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00036633199999999997 2024-04-19T01:05:06.717 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.717 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.717 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:06.717 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.540932+0000", 2024-04-19T01:05:06.717 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5972000000000001e-05 2024-04-19T01:05:06.717 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.717 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.718 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:06.718 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.567132+0000", 2024-04-19T01:05:06.718 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.026200655999999999 2024-04-19T01:05:06.718 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.718 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.718 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:06.718 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.567204+0000", 2024-04-19T01:05:06.718 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.1394000000000001e-05 2024-04-19T01:05:06.718 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:06.718 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:06.718 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:06.718 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.718 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.718 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:365 3.0s0 3:0e44b6f6:::smithi15528762-26:head [read 0~1605632,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-04-19T01:05:06.719 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:19.549248+0000", 2024-04-19T01:05:06.719 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 47.068942147999998, 2024-04-19T01:05:06.719 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.026478386999999999, 2024-04-19T01:05:06.719 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:06.719 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:06.719 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:05:06.719 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:06.719 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:06.719 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:06.719 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 365 2024-04-19T01:05:06.719 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.719 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:06.719 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.719 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:06.720 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549248+0000", 2024-04-19T01:05:06.720 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:06.720 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.720 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.720 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:06.720 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549248+0000", 2024-04-19T01:05:06.720 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:06.720 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.720 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.720 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:06.720 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549251+0000", 2024-04-19T01:05:06.720 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.8169999999999999e-06 2024-04-19T01:05:06.720 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.720 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.720 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:06.721 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549262+0000", 2024-04-19T01:05:06.722 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.1449e-05 2024-04-19T01:05:06.722 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.722 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.722 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:06.722 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549288+0000", 2024-04-19T01:05:06.722 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6169e-05 2024-04-19T01:05:06.723 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.723 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.723 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:06.723 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549312+0000", 2024-04-19T01:05:06.723 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3833e-05 2024-04-19T01:05:06.723 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.723 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.723 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:06.723 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.566277+0000", 2024-04-19T01:05:06.723 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.016964888000000001 2024-04-19T01:05:06.723 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.723 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.723 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:06.723 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.566503+0000", 2024-04-19T01:05:06.724 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00022616800000000001 2024-04-19T01:05:06.724 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.724 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.724 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:06.724 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.575726+0000", 2024-04-19T01:05:06.724 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0092230630000000001 2024-04-19T01:05:06.724 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:06.724 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:06.724 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:06.724 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.724 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.724 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:366 3.0s0 3:0e44b6f6:::smithi15528762-26:head [read 0~1605632,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-04-19T01:05:06.724 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:19.549572+0000", 2024-04-19T01:05:06.724 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 47.068617723999999, 2024-04-19T01:05:06.725 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.028385865999999999, 2024-04-19T01:05:06.725 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:06.725 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:06.725 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:05:06.725 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:06.725 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:06.725 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:06.725 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 366 2024-04-19T01:05:06.725 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.725 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:06.725 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.725 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:06.725 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549572+0000", 2024-04-19T01:05:06.725 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:06.726 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.726 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.726 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:06.726 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549572+0000", 2024-04-19T01:05:06.726 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:06.726 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.726 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.726 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:06.726 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549574+0000", 2024-04-19T01:05:06.726 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0260000000000002e-06 2024-04-19T01:05:06.726 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.726 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.726 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:06.726 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549588+0000", 2024-04-19T01:05:06.727 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.3737000000000001e-05 2024-04-19T01:05:06.727 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.727 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.727 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:06.727 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549616+0000", 2024-04-19T01:05:06.727 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.7619000000000001e-05 2024-04-19T01:05:06.727 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.727 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.727 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:06.727 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549633+0000", 2024-04-19T01:05:06.727 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7025999999999999e-05 2024-04-19T01:05:06.727 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.727 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.727 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:06.728 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.567197+0000", 2024-04-19T01:05:06.728 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.017564377999999999 2024-04-19T01:05:06.728 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.728 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.728 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:06.728 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.567461+0000", 2024-04-19T01:05:06.728 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00026393899999999998 2024-04-19T01:05:06.728 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.728 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.728 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:06.728 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.577958+0000", 2024-04-19T01:05:06.728 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.010497141 2024-04-19T01:05:06.728 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:06.728 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:06.729 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:06.729 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.729 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.729 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:367 3.0s0 3:0e44b6f6:::smithi15528762-26:head [read 0~1605632,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-04-19T01:05:06.729 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:19.549659+0000", 2024-04-19T01:05:06.729 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 47.068530916999997, 2024-04-19T01:05:06.729 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.030292887000000001, 2024-04-19T01:05:06.729 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:06.729 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:06.729 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:05:06.729 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:06.729 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:06.729 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:06.729 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 367 2024-04-19T01:05:06.729 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.730 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:06.730 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.730 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:06.730 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549659+0000", 2024-04-19T01:05:06.730 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:06.730 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.730 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.730 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:06.730 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549659+0000", 2024-04-19T01:05:06.730 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:06.730 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.730 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.730 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:06.730 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549661+0000", 2024-04-19T01:05:06.731 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2500000000000001e-06 2024-04-19T01:05:06.731 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.731 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.731 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:06.731 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549669+0000", 2024-04-19T01:05:06.731 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.7060000000000004e-06 2024-04-19T01:05:06.731 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.732 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.732 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:06.733 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549688+0000", 2024-04-19T01:05:06.733 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9309000000000001e-05 2024-04-19T01:05:06.733 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.733 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.733 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:06.733 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.549704+0000", 2024-04-19T01:05:06.733 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5398999999999999e-05 2024-04-19T01:05:06.733 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.733 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.733 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:06.733 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.570745+0000", 2024-04-19T01:05:06.733 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.021041432999999998 2024-04-19T01:05:06.733 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.734 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.734 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:06.734 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.570983+0000", 2024-04-19T01:05:06.734 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00023819299999999999 2024-04-19T01:05:06.734 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.734 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.734 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:06.734 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.579952+0000", 2024-04-19T01:05:06.734 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.008968597 2024-04-19T01:05:06.734 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:06.734 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:06.734 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:06.734 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.734 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.735 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.8s2 60/31 ECSubWrite(tid=225, reqid=client.4604.0:433, at_version=60'43, trim_to=0'0, roll_forward_to=59'41))", 2024-04-19T01:05:06.735 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:30.570124+0000", 2024-04-19T01:05:06.735 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 36.048065631999997, 2024-04-19T01:05:06.735 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.027307438, 2024-04-19T01:05:06.735 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:06.735 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:06.735 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:06.735 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:06.735 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.735 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:06.735 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.570124+0000", 2024-04-19T01:05:06.735 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:06.735 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.735 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.736 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:06.736 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.570124+0000", 2024-04-19T01:05:06.736 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:06.736 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.736 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.736 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:06.736 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.570125+0000", 2024-04-19T01:05:06.736 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.5400000000000001e-07 2024-04-19T01:05:06.736 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.736 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.736 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:06.736 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.570710+0000", 2024-04-19T01:05:06.736 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000585405 2024-04-19T01:05:06.736 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.736 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.737 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:06.737 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.570736+0000", 2024-04-19T01:05:06.737 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5511999999999999e-05 2024-04-19T01:05:06.738 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.738 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.738 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:06.738 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.570751+0000", 2024-04-19T01:05:06.739 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.4817000000000001e-05 2024-04-19T01:05:06.739 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.739 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.739 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:06.739 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.570881+0000", 2024-04-19T01:05:06.739 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000130586 2024-04-19T01:05:06.739 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.739 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.739 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:06.739 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.570900+0000", 2024-04-19T01:05:06.739 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.8868999999999998e-05 2024-04-19T01:05:06.739 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.739 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.739 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:06.740 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.597353+0000", 2024-04-19T01:05:06.740 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.026452750000000001 2024-04-19T01:05:06.740 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.740 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.740 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:06.740 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.597432+0000", 2024-04-19T01:05:06.740 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.8844999999999995e-05 2024-04-19T01:05:06.740 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:06.740 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:06.740 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:06.740 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.740 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.740 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs5 86/31 ECSubWrite(tid=131, reqid=client.4604.0:660, at_version=86'57, trim_to=0'0, roll_forward_to=86'55))", 2024-04-19T01:05:06.740 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:56.634941+0000", 2024-04-19T01:05:06.740 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 9.9832488250000004, 2024-04-19T01:05:06.741 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.025960699, 2024-04-19T01:05:06.741 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:06.741 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:06.741 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:06.741 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:06.741 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.741 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:06.741 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.634941+0000", 2024-04-19T01:05:06.741 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:06.741 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.741 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.741 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:06.741 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.634941+0000", 2024-04-19T01:05:06.742 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:06.742 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.743 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.744 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:06.744 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.634943+0000", 2024-04-19T01:05:06.744 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0959999999999999e-06 2024-04-19T01:05:06.744 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.744 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.744 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:06.744 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.635741+0000", 2024-04-19T01:05:06.744 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00079743800000000003 2024-04-19T01:05:06.744 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.744 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.745 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:06.745 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.635819+0000", 2024-04-19T01:05:06.745 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.8125000000000002e-05 2024-04-19T01:05:06.745 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.745 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.745 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:06.745 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.635857+0000", 2024-04-19T01:05:06.745 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.8025000000000003e-05 2024-04-19T01:05:06.745 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.745 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.745 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:06.745 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.635995+0000", 2024-04-19T01:05:06.745 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00013824200000000001 2024-04-19T01:05:06.745 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.746 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.746 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:06.746 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.636008+0000", 2024-04-19T01:05:06.746 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.3273e-05 2024-04-19T01:05:06.746 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.746 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.746 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:06.746 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.660827+0000", 2024-04-19T01:05:06.746 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.024818564000000001 2024-04-19T01:05:06.746 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.746 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.746 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:06.746 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.660902+0000", 2024-04-19T01:05:06.746 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.4936000000000002e-05 2024-04-19T01:05:06.747 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:06.747 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:06.747 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:06.747 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.747 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.747 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:683 3.0s0 3:07ce2446:::smithi15528762-30:head [append 0~458752] snapc 1f=[1f,1e,1c] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-19T01:05:06.747 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:01.581733+0000", 2024-04-19T01:05:06.747 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 5.0364564520000004, 2024-04-19T01:05:06.747 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.046044901999999999, 2024-04-19T01:05:06.747 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:06.747 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:06.747 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:06.747 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:06.747 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:06.748 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:06.748 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 683 2024-04-19T01:05:06.748 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.748 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:06.748 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.748 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:06.748 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.581733+0000", 2024-04-19T01:05:06.748 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:06.748 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.748 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.748 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:06.748 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.581733+0000", 2024-04-19T01:05:06.748 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:06.748 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.749 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.749 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:06.749 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.581737+0000", 2024-04-19T01:05:06.749 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.3699999999999999e-06 2024-04-19T01:05:06.749 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.749 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.749 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:06.749 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.582569+0000", 2024-04-19T01:05:06.749 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00083236400000000004 2024-04-19T01:05:06.749 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.749 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.749 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:06.749 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.582606+0000", 2024-04-19T01:05:06.749 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.6992000000000001e-05 2024-04-19T01:05:06.749 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.750 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.750 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:06.750 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.582623+0000", 2024-04-19T01:05:06.750 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7088999999999999e-05 2024-04-19T01:05:06.750 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.750 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.750 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:06.750 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.582774+0000", 2024-04-19T01:05:06.750 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015108299999999999 2024-04-19T01:05:06.750 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.750 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.750 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "op must wait for map", 2024-04-19T01:05:06.750 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.582792+0000", 2024-04-19T01:05:06.750 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7336000000000001e-05 2024-04-19T01:05:06.751 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.751 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.751 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:06.751 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.592365+0000", 2024-04-19T01:05:06.751 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0095734059999999996 2024-04-19T01:05:06.751 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.751 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.751 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:06.751 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.592530+0000", 2024-04-19T01:05:06.751 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000165169 2024-04-19T01:05:06.751 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.751 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.751 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:06.751 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.594498+0000", 2024-04-19T01:05:06.752 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0019679279999999999 2024-04-19T01:05:06.752 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.752 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.752 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:06.752 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.626973+0000", 2024-04-19T01:05:06.752 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.032474513000000003 2024-04-19T01:05:06.752 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.752 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.753 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:06.753 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.627604+0000", 2024-04-19T01:05:06.753 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00063180400000000002 2024-04-19T01:05:06.753 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.754 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.754 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:06.754 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.627778+0000", 2024-04-19T01:05:06.754 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00017384799999999999 2024-04-19T01:05:06.754 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:06.754 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:06.754 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:06.754 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.754 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.754 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:684 3.0s0 3:07ce2446:::smithi15528762-30:head [append 0~507904] snapc 1f=[1f,1e,1c] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-19T01:05:06.755 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:01.587330+0000", 2024-04-19T01:05:06.755 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 5.0308597730000004, 2024-04-19T01:05:06.755 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.041423123999999999, 2024-04-19T01:05:06.755 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:06.755 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:06.755 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:06.755 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:06.755 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:06.755 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:06.755 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 684 2024-04-19T01:05:06.755 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.755 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:06.755 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.755 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:06.756 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.587330+0000", 2024-04-19T01:05:06.756 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:06.756 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.756 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.756 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:06.756 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.587330+0000", 2024-04-19T01:05:06.756 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:06.756 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.756 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.756 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:06.756 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.587333+0000", 2024-04-19T01:05:06.757 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.1030000000000002e-06 2024-04-19T01:05:06.757 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.757 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.757 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:06.757 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.588112+0000", 2024-04-19T01:05:06.757 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00077844999999999998 2024-04-19T01:05:06.757 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.757 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.757 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:06.757 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.588139+0000", 2024-04-19T01:05:06.757 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.7616000000000001e-05 2024-04-19T01:05:06.757 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.757 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.757 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:06.757 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.588155+0000", 2024-04-19T01:05:06.758 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5432000000000001e-05 2024-04-19T01:05:06.758 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.758 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.758 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:06.758 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.588288+0000", 2024-04-19T01:05:06.758 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00013369400000000001 2024-04-19T01:05:06.758 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.758 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.758 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:05:06.758 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.588300+0000", 2024-04-19T01:05:06.758 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.2053e-05 2024-04-19T01:05:06.758 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.758 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.759 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:06.759 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.604734+0000", 2024-04-19T01:05:06.759 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.016433758999999999 2024-04-19T01:05:06.759 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.759 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.759 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:06.759 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.604928+0000", 2024-04-19T01:05:06.759 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00019333100000000001 2024-04-19T01:05:06.759 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.759 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.759 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:06.759 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.606556+0000", 2024-04-19T01:05:06.759 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0016286670000000001 2024-04-19T01:05:06.759 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.760 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.760 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:06.760 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.626976+0000", 2024-04-19T01:05:06.760 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.020420062999999999 2024-04-19T01:05:06.760 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.760 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.760 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:06.760 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.628638+0000", 2024-04-19T01:05:06.760 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001661495 2024-04-19T01:05:06.760 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.760 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.760 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:06.760 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.628753+0000", 2024-04-19T01:05:06.760 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000115461 2024-04-19T01:05:06.760 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:06.761 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:06.761 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:06.761 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.761 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.761 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:685 3.0s0 3:07ce2446:::smithi15528762-30:head [append 0~507904] snapc 1f=[1f,1e,1c] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-19T01:05:06.761 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:01.592396+0000", 2024-04-19T01:05:06.761 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 5.0257942629999999, 2024-04-19T01:05:06.761 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.050032994999999997, 2024-04-19T01:05:06.761 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:06.761 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:06.761 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:06.761 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:06.761 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:06.761 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:06.762 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 685 2024-04-19T01:05:06.762 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.762 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:06.762 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.762 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:06.762 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.592396+0000", 2024-04-19T01:05:06.762 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:06.762 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.762 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.762 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:06.762 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.592396+0000", 2024-04-19T01:05:06.762 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:06.763 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.763 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.763 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:06.763 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.592398+0000", 2024-04-19T01:05:06.763 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2359999999999999e-06 2024-04-19T01:05:06.763 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.763 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.764 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:06.764 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.593152+0000", 2024-04-19T01:05:06.765 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00075385199999999995 2024-04-19T01:05:06.765 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.765 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.765 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:06.765 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.593176+0000", 2024-04-19T01:05:06.765 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4771999999999999e-05 2024-04-19T01:05:06.765 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.765 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.765 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:06.765 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.593192+0000", 2024-04-19T01:05:06.765 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5509e-05 2024-04-19T01:05:06.765 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.765 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.765 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:06.766 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.616030+0000", 2024-04-19T01:05:06.766 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.022837803 2024-04-19T01:05:06.766 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.766 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.766 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:06.766 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.616273+0000", 2024-04-19T01:05:06.766 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00024351900000000001 2024-04-19T01:05:06.766 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.766 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.766 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:06.766 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.617970+0000", 2024-04-19T01:05:06.766 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001696674 2024-04-19T01:05:06.766 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.766 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.767 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:06.767 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.628051+0000", 2024-04-19T01:05:06.767 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.010080505 2024-04-19T01:05:06.767 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.767 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.767 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:06.767 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.642308+0000", 2024-04-19T01:05:06.767 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.014257801000000001 2024-04-19T01:05:06.767 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.767 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.767 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:06.768 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.642429+0000", 2024-04-19T01:05:06.768 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000120324 2024-04-19T01:05:06.768 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:06.768 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:06.768 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:06.768 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.768 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.768 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:686 3.0s0 3:07ce2446:::smithi15528762-30:head [append 0~458752] snapc 1f=[1f,1e,1c] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-19T01:05:06.768 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:01.597293+0000", 2024-04-19T01:05:06.768 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 5.0208964690000002, 2024-04-19T01:05:06.768 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.046212640999999999, 2024-04-19T01:05:06.768 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:06.768 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:06.768 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:06.769 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:06.769 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:06.769 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:06.769 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 686 2024-04-19T01:05:06.769 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.769 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:06.769 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.769 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:06.769 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.597293+0000", 2024-04-19T01:05:06.769 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:06.769 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.770 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.770 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:06.770 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.597293+0000", 2024-04-19T01:05:06.770 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:06.770 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.770 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.770 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:06.770 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.597296+0000", 2024-04-19T01:05:06.770 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.7499999999999999e-06 2024-04-19T01:05:06.770 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.770 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.770 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:06.770 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598220+0000", 2024-04-19T01:05:06.771 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00092368400000000003 2024-04-19T01:05:06.771 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.771 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.771 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:06.771 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598258+0000", 2024-04-19T01:05:06.771 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.7694000000000002e-05 2024-04-19T01:05:06.771 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.771 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.771 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:06.771 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598280+0000", 2024-04-19T01:05:06.771 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2626e-05 2024-04-19T01:05:06.771 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.771 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.771 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:06.772 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.629327+0000", 2024-04-19T01:05:06.772 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.03104641 2024-04-19T01:05:06.772 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.772 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.772 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:06.772 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.629495+0000", 2024-04-19T01:05:06.772 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00016847499999999999 2024-04-19T01:05:06.772 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.772 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.772 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:06.772 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.630802+0000", 2024-04-19T01:05:06.772 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0013073620000000001 2024-04-19T01:05:06.772 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.772 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.773 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:06.773 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.642744+0000", 2024-04-19T01:05:06.773 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.011941396 2024-04-19T01:05:06.773 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.773 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.773 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:06.773 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.643393+0000", 2024-04-19T01:05:06.774 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00064892900000000002 2024-04-19T01:05:06.774 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.774 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.774 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:06.774 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.643506+0000", 2024-04-19T01:05:06.774 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000113315 2024-04-19T01:05:06.775 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:06.775 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:06.775 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:06.775 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.775 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.775 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:687 3.0s0 3:07ce2446:::smithi15528762-30:head [append 0~81920] snapc 1f=[1f,1e,1c] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-19T01:05:06.776 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:01.598323+0000", 2024-04-19T01:05:06.776 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 5.0198669069999999, 2024-04-19T01:05:06.776 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.048342318000000002, 2024-04-19T01:05:06.776 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:06.776 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:06.776 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:06.776 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:06.776 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:06.776 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:06.776 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 687 2024-04-19T01:05:06.776 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.776 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:06.776 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.776 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:06.777 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598323+0000", 2024-04-19T01:05:06.777 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:06.777 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.777 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.777 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:06.777 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598323+0000", 2024-04-19T01:05:06.777 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:06.777 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.777 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.777 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:06.777 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598325+0000", 2024-04-19T01:05:06.777 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.897e-06 2024-04-19T01:05:06.777 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.777 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.778 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:06.778 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598490+0000", 2024-04-19T01:05:06.778 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000164928 2024-04-19T01:05:06.778 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.778 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.778 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:06.778 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598525+0000", 2024-04-19T01:05:06.778 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.5342999999999997e-05 2024-04-19T01:05:06.778 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.778 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.778 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:06.778 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598545+0000", 2024-04-19T01:05:06.778 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0125000000000001e-05 2024-04-19T01:05:06.778 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.779 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.779 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:06.779 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.638867+0000", 2024-04-19T01:05:06.779 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.040321362999999999 2024-04-19T01:05:06.779 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.779 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.779 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:06.779 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.639029+0000", 2024-04-19T01:05:06.779 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00016188899999999999 2024-04-19T01:05:06.779 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.779 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.779 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:06.779 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.640093+0000", 2024-04-19T01:05:06.779 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001064307 2024-04-19T01:05:06.780 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.780 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.780 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:06.780 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.643531+0000", 2024-04-19T01:05:06.780 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0034386059999999999 2024-04-19T01:05:06.780 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.780 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.780 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:06.780 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.646439+0000", 2024-04-19T01:05:06.780 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0029075770000000002 2024-04-19T01:05:06.780 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.780 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.780 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:06.780 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.646665+0000", 2024-04-19T01:05:06.781 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000226283 2024-04-19T01:05:06.781 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:06.781 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:06.781 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:06.781 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.781 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.781 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:688 3.0s0 3:07ce2446:::smithi15528762-30:head [setxattr _header (57) in=7b] snapc 1f=[1f,1e,1c] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-19T01:05:06.781 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:01.598570+0000", 2024-04-19T01:05:06.781 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 5.0196197079999996, 2024-04-19T01:05:06.781 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.049224946999999998, 2024-04-19T01:05:06.781 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:06.782 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:06.782 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:06.782 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:06.782 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:06.782 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:06.782 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 688 2024-04-19T01:05:06.793 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.793 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:06.793 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.793 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:06.793 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598570+0000", 2024-04-19T01:05:06.794 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:06.794 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.794 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.796 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:06.796 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598570+0000", 2024-04-19T01:05:06.796 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:06.796 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.796 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.796 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:06.796 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598572+0000", 2024-04-19T01:05:06.796 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6929999999999999e-06 2024-04-19T01:05:06.796 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.796 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.796 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:06.796 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598580+0000", 2024-04-19T01:05:06.796 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.4190000000000002e-06 2024-04-19T01:05:06.797 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.798 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.798 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:06.798 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598602+0000", 2024-04-19T01:05:06.798 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2133e-05 2024-04-19T01:05:06.798 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.798 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.799 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:06.799 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598617+0000", 2024-04-19T01:05:06.799 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.4576000000000001e-05 2024-04-19T01:05:06.799 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.799 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.799 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:06.799 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.644017+0000", 2024-04-19T01:05:06.799 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.045400082000000001 2024-04-19T01:05:06.799 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.799 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.800 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:06.800 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.644184+0000", 2024-04-19T01:05:06.800 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00016675500000000001 2024-04-19T01:05:06.800 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.800 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.800 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:06.800 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.644907+0000", 2024-04-19T01:05:06.800 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00072305000000000004 2024-04-19T01:05:06.801 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.801 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.801 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:06.802 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.647032+0000", 2024-04-19T01:05:06.802 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0021255940000000002 2024-04-19T01:05:06.802 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.802 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.802 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:06.802 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.647630+0000", 2024-04-19T01:05:06.802 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00059702599999999996 2024-04-19T01:05:06.802 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.802 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.802 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:06.802 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.647795+0000", 2024-04-19T01:05:06.802 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00016561899999999999 2024-04-19T01:05:06.802 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:06.803 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:06.803 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:06.803 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.803 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.803 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:689 3.0s0 3:07ce2446:::smithi15528762-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e91)", 2024-04-19T01:05:06.803 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:01.598635+0000", 2024-04-19T01:05:06.803 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 5.0195552010000002, 2024-04-19T01:05:06.803 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.052904010000000001, 2024-04-19T01:05:06.804 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:06.804 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:06.804 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:05:06.804 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:06.804 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:06.804 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:06.804 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 689 2024-04-19T01:05:06.804 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.804 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:06.805 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.805 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:06.805 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598635+0000", 2024-04-19T01:05:06.805 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:06.805 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.805 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.805 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:06.805 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598635+0000", 2024-04-19T01:05:06.805 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:06.805 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.805 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.805 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:06.805 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598636+0000", 2024-04-19T01:05:06.806 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.516e-06 2024-04-19T01:05:06.806 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.806 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.806 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:06.806 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598642+0000", 2024-04-19T01:05:06.806 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.8100000000000003e-06 2024-04-19T01:05:06.806 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.806 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.806 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:06.806 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598664+0000", 2024-04-19T01:05:06.806 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1702e-05 2024-04-19T01:05:06.806 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.807 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.807 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:06.807 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598679+0000", 2024-04-19T01:05:06.807 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5705999999999998e-05 2024-04-19T01:05:06.807 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.807 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.807 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:06.807 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.645587+0000", 2024-04-19T01:05:06.807 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.046907881999999998 2024-04-19T01:05:06.807 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.807 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.807 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting for rw locks", 2024-04-19T01:05:06.807 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.645846+0000", 2024-04-19T01:05:06.808 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00025909400000000002 2024-04-19T01:05:06.808 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.808 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.808 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:06.808 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.648563+0000", 2024-04-19T01:05:06.808 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0027163840000000001 2024-04-19T01:05:06.808 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.808 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.808 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:06.808 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.648857+0000", 2024-04-19T01:05:06.808 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00029466900000000001 2024-04-19T01:05:06.808 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:06.808 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:06.808 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:06.808 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.651539+0000", 2024-04-19T01:05:06.809 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0026812469999999999 2024-04-19T01:05:06.809 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:06.809 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:06.809 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:06.809 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:06.809 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:06.809 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:05:06.810 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-19T01:05:06.812 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:05:06.809+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:05:06.912 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:05:06.909+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:05:06.943 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:05:06.943 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [], 2024-04-19T01:05:06.943 INFO:teuthology.orchestra.run.smithi110.stdout: "num_ops": 0 2024-04-19T01:05:06.943 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:05:06.958 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-19T01:05:07.018 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:05:07.016+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:05:07.079 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:05:07.079 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [], 2024-04-19T01:05:07.079 INFO:teuthology.orchestra.run.smithi110.stdout: "complaint_time": 30, 2024-04-19T01:05:07.080 INFO:teuthology.orchestra.run.smithi110.stdout: "num_blocked_ops": 0 2024-04-19T01:05:07.080 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:05:07.093 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-19T01:05:07.119 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:05:07.117+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:05:07.213 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:05:07.213 INFO:teuthology.orchestra.run.smithi110.stdout: "size": 20, 2024-04-19T01:05:07.213 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 600, 2024-04-19T01:05:07.214 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [ 2024-04-19T01:05:07.214 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.214 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs2 43/31 ECSubWrite(tid=47, reqid=client.4604.0:263, at_version=43'8, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:05:07.214 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.995028+0000", 2024-04-19T01:05:07.214 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 53.217185465, 2024-04-19T01:05:07.214 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.038142937000000002, 2024-04-19T01:05:07.214 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:07.214 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:07.214 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:07.214 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:07.214 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.214 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:07.214 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995028+0000", 2024-04-19T01:05:07.214 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.215 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.215 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.215 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:07.215 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995028+0000", 2024-04-19T01:05:07.215 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.215 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.215 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.215 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:07.215 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995029+0000", 2024-04-19T01:05:07.215 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.47e-07 2024-04-19T01:05:07.215 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.215 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.215 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:07.216 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995397+0000", 2024-04-19T01:05:07.216 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00036862599999999999 2024-04-19T01:05:07.216 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.216 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.216 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:07.216 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995437+0000", 2024-04-19T01:05:07.216 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.9564999999999998e-05 2024-04-19T01:05:07.216 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.216 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.216 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:07.216 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995461+0000", 2024-04-19T01:05:07.216 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4261999999999999e-05 2024-04-19T01:05:07.216 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.216 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.216 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:07.217 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995661+0000", 2024-04-19T01:05:07.217 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00019937500000000001 2024-04-19T01:05:07.217 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.217 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.217 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:07.217 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995689+0000", 2024-04-19T01:05:07.217 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.8282000000000001e-05 2024-04-19T01:05:07.217 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.217 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.217 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:07.217 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.033059+0000", 2024-04-19T01:05:07.222 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.037370288000000002 2024-04-19T01:05:07.223 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.223 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.223 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:07.223 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.033171+0000", 2024-04-19T01:05:07.223 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000111692 2024-04-19T01:05:07.224 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.226 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.226 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.226 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.226 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.227 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs2 43/31 ECSubWrite(tid=48, reqid=client.4604.0:264, at_version=43'9, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:05:07.227 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.005925+0000", 2024-04-19T01:05:07.227 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 53.206288809, 2024-04-19T01:05:07.227 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.028007243000000001, 2024-04-19T01:05:07.227 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:07.227 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:07.227 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:07.227 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:07.227 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.227 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:07.227 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.005925+0000", 2024-04-19T01:05:07.227 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.227 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.228 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.228 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:07.228 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.005925+0000", 2024-04-19T01:05:07.228 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.228 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.228 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.228 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:07.228 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.005926+0000", 2024-04-19T01:05:07.228 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9700000000000002e-06 2024-04-19T01:05:07.228 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.228 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.229 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:07.229 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006164+0000", 2024-04-19T01:05:07.229 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00023784199999999999 2024-04-19T01:05:07.229 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.229 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.229 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:07.229 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006203+0000", 2024-04-19T01:05:07.229 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.9171999999999997e-05 2024-04-19T01:05:07.229 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.229 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.229 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:07.229 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006235+0000", 2024-04-19T01:05:07.229 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.1937999999999998e-05 2024-04-19T01:05:07.230 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.230 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.230 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:07.230 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.015013+0000", 2024-04-19T01:05:07.230 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0087775909999999995 2024-04-19T01:05:07.230 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.230 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.230 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:07.230 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.015043+0000", 2024-04-19T01:05:07.230 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.9739000000000001e-05 2024-04-19T01:05:07.230 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.230 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.230 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:07.230 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.033841+0000", 2024-04-19T01:05:07.231 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.018797861999999999 2024-04-19T01:05:07.231 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.231 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.231 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:07.231 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.033932+0000", 2024-04-19T01:05:07.231 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.1129000000000001e-05 2024-04-19T01:05:07.231 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.231 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.231 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.231 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.231 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.231 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs2 43/31 ECSubWrite(tid=49, reqid=client.4604.0:265, at_version=43'10, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:05:07.231 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.006295+0000", 2024-04-19T01:05:07.231 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 53.205918038999997, 2024-04-19T01:05:07.232 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.02899026, 2024-04-19T01:05:07.232 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:07.232 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:07.232 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:07.232 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:07.232 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.232 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:07.232 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006295+0000", 2024-04-19T01:05:07.232 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.232 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.232 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.232 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:07.232 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006295+0000", 2024-04-19T01:05:07.232 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.232 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.233 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.233 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:07.233 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006296+0000", 2024-04-19T01:05:07.233 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.1230000000000001e-06 2024-04-19T01:05:07.233 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.233 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.233 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:07.233 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006329+0000", 2024-04-19T01:05:07.233 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.2465000000000002e-05 2024-04-19T01:05:07.233 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.233 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.233 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:07.233 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006362+0000", 2024-04-19T01:05:07.233 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.3118999999999999e-05 2024-04-19T01:05:07.234 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.234 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.234 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:07.234 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006401+0000", 2024-04-19T01:05:07.234 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.9347999999999997e-05 2024-04-19T01:05:07.234 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.234 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.234 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:07.234 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.033284+0000", 2024-04-19T01:05:07.234 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.026882553999999999 2024-04-19T01:05:07.234 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.234 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.234 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:07.234 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.033308+0000", 2024-04-19T01:05:07.234 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4295999999999999e-05 2024-04-19T01:05:07.235 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.235 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.235 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:07.235 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.035213+0000", 2024-04-19T01:05:07.235 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0019049749999999999 2024-04-19T01:05:07.235 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.235 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.235 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:07.235 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.035286+0000", 2024-04-19T01:05:07.235 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.2379999999999995e-05 2024-04-19T01:05:07.235 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.235 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.235 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.235 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.236 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.236 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs2 48/31 ECSubWrite(tid=76, reqid=client.4604.0:345, at_version=48'22, trim_to=0'0, roll_forward_to=47'19))", 2024-04-19T01:05:07.236 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.538229+0000", 2024-04-19T01:05:07.236 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 48.673983831999998, 2024-04-19T01:05:07.236 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.041621538999999999, 2024-04-19T01:05:07.236 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:07.236 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:07.236 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:07.236 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:07.236 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.236 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:07.236 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538229+0000", 2024-04-19T01:05:07.236 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.236 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.237 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.237 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:07.237 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538229+0000", 2024-04-19T01:05:07.237 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.237 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.237 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.237 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:07.237 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538232+0000", 2024-04-19T01:05:07.237 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1730000000000002e-06 2024-04-19T01:05:07.237 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.237 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.237 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:07.237 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538714+0000", 2024-04-19T01:05:07.237 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00048223299999999999 2024-04-19T01:05:07.238 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.238 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.238 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:07.238 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538755+0000", 2024-04-19T01:05:07.239 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.1044999999999998e-05 2024-04-19T01:05:07.239 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.239 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.239 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:07.239 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538779+0000", 2024-04-19T01:05:07.239 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3828999999999998e-05 2024-04-19T01:05:07.239 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.239 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.239 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:07.239 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.539007+0000", 2024-04-19T01:05:07.239 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00022818400000000001 2024-04-19T01:05:07.240 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.240 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.240 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:07.240 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.539033+0000", 2024-04-19T01:05:07.240 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6455999999999999e-05 2024-04-19T01:05:07.240 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.240 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.240 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:07.241 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.579777+0000", 2024-04-19T01:05:07.241 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.040743133000000001 2024-04-19T01:05:07.241 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.241 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.241 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:07.241 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.579851+0000", 2024-04-19T01:05:07.241 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.4486000000000005e-05 2024-04-19T01:05:07.241 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.241 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.241 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.241 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.241 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.241 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs2 48/31 ECSubWrite(tid=78, reqid=client.4604.0:346, at_version=48'23, trim_to=0'0, roll_forward_to=48'21))", 2024-04-19T01:05:07.242 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.549034+0000", 2024-04-19T01:05:07.242 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 48.663179788999997, 2024-04-19T01:05:07.242 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.030907911999999999, 2024-04-19T01:05:07.243 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:07.244 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:07.244 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:07.244 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:07.244 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.244 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:07.244 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.549034+0000", 2024-04-19T01:05:07.244 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.244 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.245 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.245 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:07.245 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.549034+0000", 2024-04-19T01:05:07.245 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.245 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.245 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.245 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:07.245 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.549035+0000", 2024-04-19T01:05:07.245 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9429999999999999e-06 2024-04-19T01:05:07.245 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.245 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.245 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:07.245 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.549509+0000", 2024-04-19T01:05:07.245 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00047399999999999997 2024-04-19T01:05:07.246 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.246 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.246 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:07.246 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.549541+0000", 2024-04-19T01:05:07.246 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.1843e-05 2024-04-19T01:05:07.246 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.246 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.246 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:07.246 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.549563+0000", 2024-04-19T01:05:07.247 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2045e-05 2024-04-19T01:05:07.247 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.247 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.247 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:07.247 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.558487+0000", 2024-04-19T01:05:07.247 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0089238470000000004 2024-04-19T01:05:07.247 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.247 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.247 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:07.247 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.558508+0000", 2024-04-19T01:05:07.247 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1236000000000001e-05 2024-04-19T01:05:07.247 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.247 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.247 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:07.247 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.579870+0000", 2024-04-19T01:05:07.248 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.021361601000000001 2024-04-19T01:05:07.248 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.248 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.248 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:07.248 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.579941+0000", 2024-04-19T01:05:07.248 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.1396999999999994e-05 2024-04-19T01:05:07.248 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.248 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.248 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.248 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.248 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.248 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs2 48/31 ECSubWrite(tid=79, reqid=client.4604.0:347, at_version=48'24, trim_to=0'0, roll_forward_to=48'21))", 2024-04-19T01:05:07.248 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.555772+0000", 2024-04-19T01:05:07.248 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 48.656440963000001, 2024-04-19T01:05:07.249 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.024949988999999999, 2024-04-19T01:05:07.249 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:07.249 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:07.249 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:07.249 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:07.249 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.249 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:07.249 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.555772+0000", 2024-04-19T01:05:07.249 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.249 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.249 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.249 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:07.249 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.555772+0000", 2024-04-19T01:05:07.249 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.250 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.250 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.250 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:07.250 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.555774+0000", 2024-04-19T01:05:07.250 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.734e-06 2024-04-19T01:05:07.250 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.250 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.250 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:07.250 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.555841+0000", 2024-04-19T01:05:07.250 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.6434999999999997e-05 2024-04-19T01:05:07.250 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.250 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.250 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:07.250 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.555877+0000", 2024-04-19T01:05:07.251 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.6544000000000001e-05 2024-04-19T01:05:07.251 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.251 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.251 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:07.251 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.555895+0000", 2024-04-19T01:05:07.251 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7558999999999999e-05 2024-04-19T01:05:07.251 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.251 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.251 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:07.251 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.576874+0000", 2024-04-19T01:05:07.251 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.020979564999999999 2024-04-19T01:05:07.251 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.251 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.251 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:07.251 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.576898+0000", 2024-04-19T01:05:07.252 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3787000000000001e-05 2024-04-19T01:05:07.252 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.252 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.252 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:07.252 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.580616+0000", 2024-04-19T01:05:07.252 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0037177289999999999 2024-04-19T01:05:07.252 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.252 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.252 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:07.253 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.580722+0000", 2024-04-19T01:05:07.253 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000106636 2024-04-19T01:05:07.253 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.253 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.253 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.254 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.254 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.254 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.8s6 48/31 ECSubWrite(tid=199, reqid=client.4604.0:353, at_version=0'0, trim_to=0'0, roll_forward_to=48'20))", 2024-04-19T01:05:07.254 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.557241+0000", 2024-04-19T01:05:07.254 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 48.654972628000003, 2024-04-19T01:05:07.255 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.020337572000000002, 2024-04-19T01:05:07.255 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:07.255 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:07.255 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:07.255 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:07.255 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.255 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:07.255 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557241+0000", 2024-04-19T01:05:07.256 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.256 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.256 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.256 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:07.256 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557241+0000", 2024-04-19T01:05:07.256 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.256 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.256 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.256 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:07.256 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557242+0000", 2024-04-19T01:05:07.256 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.4449999999999999e-06 2024-04-19T01:05:07.256 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.256 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.257 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:07.257 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557262+0000", 2024-04-19T01:05:07.257 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0103999999999999e-05 2024-04-19T01:05:07.257 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.257 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.257 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:07.257 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557287+0000", 2024-04-19T01:05:07.257 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4556e-05 2024-04-19T01:05:07.257 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.258 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.258 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:07.259 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557304+0000", 2024-04-19T01:05:07.259 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7592e-05 2024-04-19T01:05:07.259 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.259 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.260 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:07.260 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557935+0000", 2024-04-19T01:05:07.260 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00063064499999999999 2024-04-19T01:05:07.260 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.260 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.260 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:07.260 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557959+0000", 2024-04-19T01:05:07.260 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3736000000000001e-05 2024-04-19T01:05:07.260 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.260 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.260 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:07.260 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.577484+0000", 2024-04-19T01:05:07.260 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.019525415000000001 2024-04-19T01:05:07.261 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.261 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.261 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:07.261 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.577578+0000", 2024-04-19T01:05:07.261 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.4079000000000005e-05 2024-04-19T01:05:07.261 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.262 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.262 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.262 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.262 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.262 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.es2 48/31 ECSubWrite(tid=118, reqid=client.4604.0:357, at_version=48'25, trim_to=0'0, roll_forward_to=48'23))", 2024-04-19T01:05:07.262 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.557362+0000", 2024-04-19T01:05:07.262 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 48.654851540999999, 2024-04-19T01:05:07.262 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.020223954999999998, 2024-04-19T01:05:07.262 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:07.262 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:07.262 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:07.263 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:07.263 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.263 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:07.263 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557362+0000", 2024-04-19T01:05:07.263 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.263 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.263 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.263 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:07.263 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557362+0000", 2024-04-19T01:05:07.263 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.263 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.263 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.263 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:07.264 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557363+0000", 2024-04-19T01:05:07.264 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.7499999999999999e-07 2024-04-19T01:05:07.264 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.264 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.264 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:07.264 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557396+0000", 2024-04-19T01:05:07.264 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.3878999999999999e-05 2024-04-19T01:05:07.264 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.264 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.264 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:07.264 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557421+0000", 2024-04-19T01:05:07.264 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4627999999999999e-05 2024-04-19T01:05:07.264 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.264 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.265 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:07.265 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.557435+0000", 2024-04-19T01:05:07.265 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.4185000000000001e-05 2024-04-19T01:05:07.265 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.265 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.265 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:07.265 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.558839+0000", 2024-04-19T01:05:07.265 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0014035860000000001 2024-04-19T01:05:07.265 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.265 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.265 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:07.265 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.558863+0000", 2024-04-19T01:05:07.265 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4434e-05 2024-04-19T01:05:07.265 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.265 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.266 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:07.266 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.577527+0000", 2024-04-19T01:05:07.266 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.018663800000000001 2024-04-19T01:05:07.266 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.266 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.266 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:07.266 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.577586+0000", 2024-04-19T01:05:07.266 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.8668000000000001e-05 2024-04-19T01:05:07.266 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.266 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.266 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.266 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.266 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.266 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs2 48/31 ECSubWrite(tid=80, reqid=client.4604.0:348, at_version=48'25, trim_to=0'0, roll_forward_to=48'21))", 2024-04-19T01:05:07.267 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.561184+0000", 2024-04-19T01:05:07.267 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 48.651029590999997, 2024-04-19T01:05:07.267 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.021208286, 2024-04-19T01:05:07.267 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:07.267 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:07.267 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:07.267 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:07.267 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.267 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:07.267 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.561184+0000", 2024-04-19T01:05:07.267 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.267 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.267 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.267 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:07.268 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.561184+0000", 2024-04-19T01:05:07.268 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.268 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.268 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.268 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:07.268 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.561185+0000", 2024-04-19T01:05:07.268 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.685e-06 2024-04-19T01:05:07.268 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.268 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.268 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:07.268 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.561230+0000", 2024-04-19T01:05:07.268 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.4867000000000003e-05 2024-04-19T01:05:07.268 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.268 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.268 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:07.269 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.561282+0000", 2024-04-19T01:05:07.269 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.1712999999999998e-05 2024-04-19T01:05:07.269 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.269 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.269 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:07.269 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.561304+0000", 2024-04-19T01:05:07.269 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2209999999999999e-05 2024-04-19T01:05:07.269 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.269 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.269 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:07.269 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.580080+0000", 2024-04-19T01:05:07.269 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.018775475999999999 2024-04-19T01:05:07.269 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.269 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.270 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:07.270 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.580193+0000", 2024-04-19T01:05:07.270 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000113244 2024-04-19T01:05:07.270 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.270 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.270 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:07.270 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.582324+0000", 2024-04-19T01:05:07.270 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0021313790000000001 2024-04-19T01:05:07.270 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.270 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.270 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:07.270 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.582392+0000", 2024-04-19T01:05:07.270 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.7712000000000005e-05 2024-04-19T01:05:07.270 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.270 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.271 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.271 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.271 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.271 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.8s6 49/31 ECSubWrite(tid=203, reqid=client.4604.0:360, at_version=49'25, trim_to=0'0, roll_forward_to=49'24))", 2024-04-19T01:05:07.271 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:19.538676+0000", 2024-04-19T01:05:07.271 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 47.673537572999997, 2024-04-19T01:05:07.271 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.019513513, 2024-04-19T01:05:07.271 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:07.271 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:07.271 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:07.271 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:07.271 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.271 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:07.271 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.538676+0000", 2024-04-19T01:05:07.272 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.272 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.272 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.272 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:07.272 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.538676+0000", 2024-04-19T01:05:07.272 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.272 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.272 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.272 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:07.272 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.538678+0000", 2024-04-19T01:05:07.272 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0260000000000002e-06 2024-04-19T01:05:07.272 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.272 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.272 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:07.273 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.538968+0000", 2024-04-19T01:05:07.273 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000290172 2024-04-19T01:05:07.273 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.273 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.273 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:07.273 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.539364+0000", 2024-04-19T01:05:07.273 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00039640399999999999 2024-04-19T01:05:07.273 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.274 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.274 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:07.274 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.539393+0000", 2024-04-19T01:05:07.274 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.8912e-05 2024-04-19T01:05:07.274 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.274 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.275 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:07.275 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.539604+0000", 2024-04-19T01:05:07.275 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00021122900000000001 2024-04-19T01:05:07.275 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.275 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.277 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:07.278 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.539630+0000", 2024-04-19T01:05:07.278 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5151999999999999e-05 2024-04-19T01:05:07.278 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.278 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.278 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:07.278 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.558098+0000", 2024-04-19T01:05:07.278 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.018468232000000001 2024-04-19T01:05:07.278 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.278 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.278 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:07.278 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:19.558189+0000", 2024-04-19T01:05:07.278 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.1385999999999995e-05 2024-04-19T01:05:07.278 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.279 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.279 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.279 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.279 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.279 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.5s6 59/31 ECSubWrite(tid=66, reqid=client.4604.0:423, at_version=59'51, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:05:07.279 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:29.540851+0000", 2024-04-19T01:05:07.279 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 37.671362297999998, 2024-04-19T01:05:07.279 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.030615177, 2024-04-19T01:05:07.279 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:07.279 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:07.279 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:07.279 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:07.279 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.279 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:07.280 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.540851+0000", 2024-04-19T01:05:07.280 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.280 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.280 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.280 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:07.280 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.540851+0000", 2024-04-19T01:05:07.280 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.280 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.280 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.280 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:07.280 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.540852+0000", 2024-04-19T01:05:07.280 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.6400000000000002e-07 2024-04-19T01:05:07.280 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.280 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.281 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:07.281 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.541398+0000", 2024-04-19T01:05:07.281 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00054620600000000004 2024-04-19T01:05:07.281 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.281 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.281 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:07.281 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.541433+0000", 2024-04-19T01:05:07.281 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.4853e-05 2024-04-19T01:05:07.281 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.281 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.281 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:07.281 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.541457+0000", 2024-04-19T01:05:07.281 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4641999999999999e-05 2024-04-19T01:05:07.281 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.281 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.282 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:07.282 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.541639+0000", 2024-04-19T01:05:07.282 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00018139100000000001 2024-04-19T01:05:07.282 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.282 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.282 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:07.282 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.541667+0000", 2024-04-19T01:05:07.282 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.7943e-05 2024-04-19T01:05:07.282 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.282 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.282 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:07.282 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.571413+0000", 2024-04-19T01:05:07.282 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0297462 2024-04-19T01:05:07.283 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.283 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.283 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:07.283 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.571466+0000", 2024-04-19T01:05:07.283 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.3378000000000003e-05 2024-04-19T01:05:07.283 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.283 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.283 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.283 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.283 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.283 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.5s6 59/31 ECSubWrite(tid=67, reqid=client.4604.0:424, at_version=59'52, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:05:07.283 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:29.548369+0000", 2024-04-19T01:05:07.284 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 37.663844554000001, 2024-04-19T01:05:07.284 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.023140674999999999, 2024-04-19T01:05:07.284 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:07.284 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:07.284 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:07.284 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:07.284 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.284 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:07.284 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.548369+0000", 2024-04-19T01:05:07.284 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.284 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.284 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.284 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:07.284 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.548369+0000", 2024-04-19T01:05:07.284 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.285 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.285 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.285 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:07.285 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.548370+0000", 2024-04-19T01:05:07.285 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.066e-06 2024-04-19T01:05:07.285 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.285 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.285 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:07.285 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.548631+0000", 2024-04-19T01:05:07.285 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00026157799999999999 2024-04-19T01:05:07.285 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.285 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.285 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:07.285 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.548648+0000", 2024-04-19T01:05:07.286 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7079e-05 2024-04-19T01:05:07.286 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.286 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.286 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:07.286 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.548657+0000", 2024-04-19T01:05:07.286 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.9500000000000007e-06 2024-04-19T01:05:07.286 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.286 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.286 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:07.286 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.555857+0000", 2024-04-19T01:05:07.286 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0071992569999999997 2024-04-19T01:05:07.286 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.286 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.286 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:07.287 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.555883+0000", 2024-04-19T01:05:07.287 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6522000000000001e-05 2024-04-19T01:05:07.287 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.287 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.287 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:07.287 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.571475+0000", 2024-04-19T01:05:07.287 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.015591727999999999 2024-04-19T01:05:07.287 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.288 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.288 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:07.288 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.571509+0000", 2024-04-19T01:05:07.288 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.4495000000000001e-05 2024-04-19T01:05:07.288 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.288 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.288 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.288 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.288 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.288 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.8s6 60/31 ECSubWrite(tid=225, reqid=client.4604.0:433, at_version=60'43, trim_to=0'0, roll_forward_to=59'41))", 2024-04-19T01:05:07.288 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:30.569927+0000", 2024-04-19T01:05:07.288 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 36.642286257999999, 2024-04-19T01:05:07.288 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.027044315999999999, 2024-04-19T01:05:07.288 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:07.289 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:07.289 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:07.289 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:07.289 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.289 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:07.289 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.569927+0000", 2024-04-19T01:05:07.289 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.289 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.289 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.289 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:07.289 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.569927+0000", 2024-04-19T01:05:07.289 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.289 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.289 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.289 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:07.290 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.569927+0000", 2024-04-19T01:05:07.290 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.34e-07 2024-04-19T01:05:07.290 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.290 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.290 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:07.290 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.570430+0000", 2024-04-19T01:05:07.290 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00050269400000000004 2024-04-19T01:05:07.290 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.290 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.290 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:07.290 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.570465+0000", 2024-04-19T01:05:07.290 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.4898999999999999e-05 2024-04-19T01:05:07.290 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.290 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.291 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:07.291 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.570490+0000", 2024-04-19T01:05:07.291 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4805999999999999e-05 2024-04-19T01:05:07.291 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.291 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.291 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:07.291 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.570639+0000", 2024-04-19T01:05:07.291 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00014890899999999999 2024-04-19T01:05:07.291 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.291 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.291 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:07.291 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.570665+0000", 2024-04-19T01:05:07.291 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5825000000000001e-05 2024-04-19T01:05:07.291 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.292 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.292 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:07.292 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.596871+0000", 2024-04-19T01:05:07.292 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.026206291999999999 2024-04-19T01:05:07.292 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.292 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.292 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:07.292 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.596971+0000", 2024-04-19T01:05:07.292 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000100457 2024-04-19T01:05:07.292 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.292 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.292 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.292 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.292 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.292 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.5s6 65/31 ECSubWrite(tid=75, reqid=client.4604.0:486, at_version=65'56, trim_to=0'0, roll_forward_to=59'54))", 2024-04-19T01:05:07.293 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:35.660833+0000", 2024-04-19T01:05:07.293 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 31.551380104, 2024-04-19T01:05:07.293 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.026740791999999999, 2024-04-19T01:05:07.293 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:07.293 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:07.293 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:07.293 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:07.293 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.293 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:07.293 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.660833+0000", 2024-04-19T01:05:07.293 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.293 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.293 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.293 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:07.294 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.660833+0000", 2024-04-19T01:05:07.294 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.294 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.294 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.294 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:07.294 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.660834+0000", 2024-04-19T01:05:07.294 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.7299999999999995e-07 2024-04-19T01:05:07.294 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.294 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.294 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:07.294 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.661764+0000", 2024-04-19T01:05:07.294 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00092978500000000003 2024-04-19T01:05:07.294 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.294 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.295 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:07.295 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.661794+0000", 2024-04-19T01:05:07.295 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.021e-05 2024-04-19T01:05:07.295 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.295 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.295 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:07.295 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.661810+0000", 2024-04-19T01:05:07.295 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6082e-05 2024-04-19T01:05:07.295 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.295 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.295 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:07.295 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.662014+0000", 2024-04-19T01:05:07.295 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000203625 2024-04-19T01:05:07.295 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.295 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.296 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:07.296 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.662030+0000", 2024-04-19T01:05:07.296 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6293000000000001e-05 2024-04-19T01:05:07.296 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.296 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.296 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:07.296 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.687509+0000", 2024-04-19T01:05:07.296 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.025479485999999999 2024-04-19T01:05:07.296 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.296 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.296 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:07.296 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.687574+0000", 2024-04-19T01:05:07.296 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.4638e-05 2024-04-19T01:05:07.296 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.296 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.297 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.297 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.297 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.297 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.5s6 74/31 ECSubWrite(tid=88, reqid=osd.8.0:88, at_version=74'69, trim_to=0'0, roll_forward_to=72'65))", 2024-04-19T01:05:07.297 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:44.559798+0000", 2024-04-19T01:05:07.297 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 22.652415273999999, 2024-04-19T01:05:07.297 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.024864188999999998, 2024-04-19T01:05:07.297 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:07.298 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:07.298 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:07.301 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:07.301 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.301 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:07.301 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.559798+0000", 2024-04-19T01:05:07.301 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.302 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.302 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.302 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:07.302 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.559798+0000", 2024-04-19T01:05:07.302 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.302 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.302 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.302 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:07.302 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.559800+0000", 2024-04-19T01:05:07.302 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.88e-06 2024-04-19T01:05:07.302 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.302 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.302 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:07.303 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.559851+0000", 2024-04-19T01:05:07.303 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.0739000000000003e-05 2024-04-19T01:05:07.303 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.303 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.303 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:07.303 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.559882+0000", 2024-04-19T01:05:07.303 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.1038999999999999e-05 2024-04-19T01:05:07.303 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.303 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.303 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:07.303 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.559908+0000", 2024-04-19T01:05:07.303 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6460000000000001e-05 2024-04-19T01:05:07.303 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.303 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.303 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:07.304 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.564678+0000", 2024-04-19T01:05:07.304 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0047699079999999998 2024-04-19T01:05:07.304 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.304 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.304 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:07.304 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.564694+0000", 2024-04-19T01:05:07.304 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6056e-05 2024-04-19T01:05:07.304 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.304 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.304 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:07.304 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.584573+0000", 2024-04-19T01:05:07.304 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.019878939000000002 2024-04-19T01:05:07.304 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.304 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.305 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:07.305 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.584662+0000", 2024-04-19T01:05:07.305 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.9167999999999997e-05 2024-04-19T01:05:07.305 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.305 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.305 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.305 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.305 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.305 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.5s6 74/31 ECSubWrite(tid=89, reqid=client.4604.0:544, at_version=74'70, trim_to=0'0, roll_forward_to=72'65))", 2024-04-19T01:05:07.305 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:44.566121+0000", 2024-04-19T01:05:07.305 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 22.646092767999999, 2024-04-19T01:05:07.305 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.026764973000000001, 2024-04-19T01:05:07.305 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:07.305 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:07.306 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:07.306 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:07.306 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.306 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:07.306 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.566121+0000", 2024-04-19T01:05:07.306 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.306 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.306 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.306 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:07.306 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.566121+0000", 2024-04-19T01:05:07.306 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.306 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.306 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.306 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:07.306 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.566123+0000", 2024-04-19T01:05:07.307 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0770000000000002e-06 2024-04-19T01:05:07.307 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.307 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.307 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:07.307 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.566618+0000", 2024-04-19T01:05:07.307 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000495296 2024-04-19T01:05:07.307 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.307 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.307 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:07.307 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.566651+0000", 2024-04-19T01:05:07.307 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.2795e-05 2024-04-19T01:05:07.308 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.308 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.308 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:07.308 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.566673+0000", 2024-04-19T01:05:07.309 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1926999999999999e-05 2024-04-19T01:05:07.309 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.309 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.309 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:07.309 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.566823+0000", 2024-04-19T01:05:07.309 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015024999999999999 2024-04-19T01:05:07.309 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.309 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.309 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:07.309 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.566852+0000", 2024-04-19T01:05:07.309 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.9493e-05 2024-04-19T01:05:07.309 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.310 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.310 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:07.310 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.592770+0000", 2024-04-19T01:05:07.310 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.025917557000000001 2024-04-19T01:05:07.310 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.310 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.310 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:07.310 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.592886+0000", 2024-04-19T01:05:07.310 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000115578 2024-04-19T01:05:07.310 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.310 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.310 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.310 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.310 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.310 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.4s6 81/31 ECSubWrite(tid=117, reqid=client.4604.0:618, at_version=81'41, trim_to=0'0, roll_forward_to=79'39))", 2024-04-19T01:05:07.311 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:51.536457+0000", 2024-04-19T01:05:07.311 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 15.675756758, 2024-04-19T01:05:07.311 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.01987593, 2024-04-19T01:05:07.311 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:07.311 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:07.311 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:07.311 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:07.311 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.311 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:07.311 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:51.536457+0000", 2024-04-19T01:05:07.311 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.311 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.311 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.311 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:07.312 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:51.536457+0000", 2024-04-19T01:05:07.312 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.312 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.312 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.312 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:07.312 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:51.536457+0000", 2024-04-19T01:05:07.312 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.9399999999999995e-07 2024-04-19T01:05:07.312 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.312 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.312 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:07.312 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:51.537268+0000", 2024-04-19T01:05:07.312 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00081112000000000003 2024-04-19T01:05:07.312 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.312 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.313 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:07.313 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:51.537301+0000", 2024-04-19T01:05:07.313 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.3188000000000001e-05 2024-04-19T01:05:07.313 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.313 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.313 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:07.313 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:51.537329+0000", 2024-04-19T01:05:07.313 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.7701000000000001e-05 2024-04-19T01:05:07.313 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.313 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.313 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:07.313 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:51.537426+0000", 2024-04-19T01:05:07.313 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.6477999999999993e-05 2024-04-19T01:05:07.313 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.313 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.314 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:07.314 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:51.537435+0000", 2024-04-19T01:05:07.314 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.302e-06 2024-04-19T01:05:07.314 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.314 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.314 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:07.314 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:51.556199+0000", 2024-04-19T01:05:07.314 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.018764062000000001 2024-04-19T01:05:07.314 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.314 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.314 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:07.314 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:51.556332+0000", 2024-04-19T01:05:07.314 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000133585 2024-04-19T01:05:07.314 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.315 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.315 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.315 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.315 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.315 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.es2 86/31 ECSubWrite(tid=195, reqid=client.4604.0:653, at_version=86'39, trim_to=0'0, roll_forward_to=84'38))", 2024-04-19T01:05:07.315 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:56.572591+0000", 2024-04-19T01:05:07.315 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 10.639622624999999, 2024-04-19T01:05:07.315 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.030973436999999999, 2024-04-19T01:05:07.315 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:07.315 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:07.315 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:07.315 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:07.315 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.315 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:07.316 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.572591+0000", 2024-04-19T01:05:07.316 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.316 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.316 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.316 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:07.316 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.572591+0000", 2024-04-19T01:05:07.316 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.316 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.316 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.316 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:07.316 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.572592+0000", 2024-04-19T01:05:07.316 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.004e-06 2024-04-19T01:05:07.316 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.316 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.316 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:07.317 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.573633+0000", 2024-04-19T01:05:07.317 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0010417340000000001 2024-04-19T01:05:07.317 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.317 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.317 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:07.317 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.573675+0000", 2024-04-19T01:05:07.317 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.1347999999999998e-05 2024-04-19T01:05:07.317 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.317 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.317 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:07.317 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.573704+0000", 2024-04-19T01:05:07.317 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.8781000000000001e-05 2024-04-19T01:05:07.317 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.317 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.317 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:07.318 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.573872+0000", 2024-04-19T01:05:07.318 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00016797900000000001 2024-04-19T01:05:07.318 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.318 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.318 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:07.318 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.573888+0000", 2024-04-19T01:05:07.318 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6164e-05 2024-04-19T01:05:07.319 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.319 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.319 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:07.319 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.603491+0000", 2024-04-19T01:05:07.319 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.029603401000000001 2024-04-19T01:05:07.319 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.319 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.319 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:07.319 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.603564+0000", 2024-04-19T01:05:07.319 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.3026000000000002e-05 2024-04-19T01:05:07.319 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.319 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.319 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.319 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.319 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.320 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.2s1 86/31 ECSubWrite(tid=265, reqid=client.4604.0:655, at_version=86'35, trim_to=0'0, roll_forward_to=86'33))", 2024-04-19T01:05:07.320 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:56.580658+0000", 2024-04-19T01:05:07.320 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 10.631555328999999, 2024-04-19T01:05:07.320 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.030122023000000001, 2024-04-19T01:05:07.320 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:07.320 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:07.320 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:07.320 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:07.320 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.320 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:07.320 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.580658+0000", 2024-04-19T01:05:07.320 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.320 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.320 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.321 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:07.321 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.580658+0000", 2024-04-19T01:05:07.321 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.321 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.321 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.321 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:07.321 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.580660+0000", 2024-04-19T01:05:07.321 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.8169999999999999e-06 2024-04-19T01:05:07.321 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.322 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.322 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:07.322 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.581247+0000", 2024-04-19T01:05:07.322 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00058683699999999999 2024-04-19T01:05:07.322 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.322 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.322 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:07.322 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.581276+0000", 2024-04-19T01:05:07.322 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.9349999999999999e-05 2024-04-19T01:05:07.322 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.322 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.322 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:07.322 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.581297+0000", 2024-04-19T01:05:07.323 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1027e-05 2024-04-19T01:05:07.323 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.323 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.323 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:07.323 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.581466+0000", 2024-04-19T01:05:07.323 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00016856799999999999 2024-04-19T01:05:07.323 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.323 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.323 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:07.324 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.581495+0000", 2024-04-19T01:05:07.324 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.959e-05 2024-04-19T01:05:07.324 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.324 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.324 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:07.325 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.610709+0000", 2024-04-19T01:05:07.325 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.029214107 2024-04-19T01:05:07.325 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.325 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.325 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:07.325 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.610780+0000", 2024-04-19T01:05:07.325 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.0727000000000002e-05 2024-04-19T01:05:07.325 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.325 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.325 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.325 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.325 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.325 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs2 86/31 ECSubWrite(tid=132, reqid=client.4604.0:661, at_version=86'58, trim_to=0'0, roll_forward_to=86'55))", 2024-04-19T01:05:07.325 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:56.651076+0000", 2024-04-19T01:05:07.326 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 10.561137771, 2024-04-19T01:05:07.326 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.019993653, 2024-04-19T01:05:07.326 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:07.326 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:07.326 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:07.326 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:07.326 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.326 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:07.326 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.651076+0000", 2024-04-19T01:05:07.326 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.326 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.326 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.326 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:07.326 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.651076+0000", 2024-04-19T01:05:07.327 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.327 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.327 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.327 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:07.327 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.651078+0000", 2024-04-19T01:05:07.327 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1339999999999998e-06 2024-04-19T01:05:07.327 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.327 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.327 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:07.327 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.651399+0000", 2024-04-19T01:05:07.327 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00032083200000000001 2024-04-19T01:05:07.327 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.327 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.327 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:07.327 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.651439+0000", 2024-04-19T01:05:07.328 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.0744000000000002e-05 2024-04-19T01:05:07.328 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.328 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.328 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:07.328 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.651464+0000", 2024-04-19T01:05:07.328 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4462999999999999e-05 2024-04-19T01:05:07.328 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.328 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.328 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:07.328 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.651637+0000", 2024-04-19T01:05:07.329 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000173124 2024-04-19T01:05:07.329 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.329 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.329 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:07.329 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.651663+0000", 2024-04-19T01:05:07.329 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6251e-05 2024-04-19T01:05:07.329 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.329 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.329 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:07.329 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.670916+0000", 2024-04-19T01:05:07.329 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.019252849999999998 2024-04-19T01:05:07.329 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.330 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.330 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:07.330 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.671069+0000", 2024-04-19T01:05:07.330 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015325500000000001 2024-04-19T01:05:07.330 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.330 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.330 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.330 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.330 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.330 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:05:07.332 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-19T01:05:07.332 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:05:07.332+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:05:07.433 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:05:07.430+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:05:07.458 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:05:07.458 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [], 2024-04-19T01:05:07.458 INFO:teuthology.orchestra.run.smithi110.stdout: "num_ops": 0 2024-04-19T01:05:07.458 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:05:07.472 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-19T01:05:07.523 INFO:tasks.rados.rados.0.smithi155.stdout:300: done (1 left) 2024-04-19T01:05:07.523 INFO:tasks.rados.rados.0.smithi155.stdout:301: done (0 left) 2024-04-19T01:05:07.523 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 46 current snap is 34 2024-04-19T01:05:07.523 INFO:tasks.rados.rados.0.smithi155.stdout:302: seq_num 123 ranges {2342912=524288,2867200=573440,3440640=688128,4128768=294912} 2024-04-19T01:05:07.536 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:05:07.532+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:05:07.575 INFO:tasks.rados.rados.0.smithi155.stdout:302: writing smithi15528762-46 from 2342912 to 2867200 tid 1 2024-04-19T01:05:07.584 INFO:tasks.rados.rados.0.smithi155.stdout:302: writing smithi15528762-46 from 2867200 to 3440640 tid 2 2024-04-19T01:05:07.592 INFO:tasks.rados.rados.0.smithi155.stdout:302: writing smithi15528762-46 from 3440640 to 4128768 tid 3 2024-04-19T01:05:07.595 INFO:tasks.rados.rados.0.smithi155.stdout:302: writing smithi15528762-46 from 4128768 to 4423680 tid 4 2024-04-19T01:05:07.595 INFO:tasks.rados.rados.0.smithi155.stdout:303: snap_create 2024-04-19T01:05:07.608 INFO:tasks.rados.rados.0.smithi155.stdout:302: finishing write tid 1 to smithi15528762-46 2024-04-19T01:05:07.620 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:05:07.621 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [], 2024-04-19T01:05:07.621 INFO:teuthology.orchestra.run.smithi110.stdout: "complaint_time": 30, 2024-04-19T01:05:07.621 INFO:teuthology.orchestra.run.smithi110.stdout: "num_blocked_ops": 0 2024-04-19T01:05:07.621 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:05:07.625 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-19T01:05:07.635 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:05:07.632+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:05:07.637 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-19T01:05:07.637 INFO:tasks.rados.rados.0.smithi155.stdout:302: finishing write tid 2 to smithi15528762-46 2024-04-19T01:05:07.638 INFO:tasks.rados.rados.0.smithi155.stdout:302: finishing write tid 3 to smithi15528762-46 2024-04-19T01:05:07.638 INFO:tasks.rados.rados.0.smithi155.stdout:302: finishing write tid 4 to smithi15528762-46 2024-04-19T01:05:07.642 INFO:tasks.rados.rados.0.smithi155.stdout:302: finishing write tid 5 to smithi15528762-46 2024-04-19T01:05:07.647 INFO:tasks.rados.rados.0.smithi155.stdout:302: finishing write tid 6 to smithi15528762-46 2024-04-19T01:05:07.647 INFO:tasks.rados.rados.0.smithi155.stdout:302: oid 46 updating version 0 to 92 2024-04-19T01:05:07.647 INFO:tasks.rados.rados.0.smithi155.stdout:302: oid 46 version 92 is already newer than 91 2024-04-19T01:05:07.647 INFO:tasks.rados.rados.0.smithi155.stdout:302: oid 46 version 92 is already newer than 90 2024-04-19T01:05:07.647 INFO:tasks.rados.rados.0.smithi155.stdout:302: oid 46 version 92 is already newer than 88 2024-04-19T01:05:07.647 INFO:tasks.rados.rados.0.smithi155.stdout:302: oid 46 version 92 is already newer than 89 2024-04-19T01:05:07.647 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 46 v 92 (ObjNum 123 snap 34 seq_num 123) dirty exists 2024-04-19T01:05:07.647 INFO:tasks.rados.rados.0.smithi155.stdout:302: left oid 46 (ObjNum 123 snap 34 seq_num 123) 2024-04-19T01:05:07.649 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-19T01:05:07.737 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:05:07.733+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:05:07.758 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:05:07.758 INFO:teuthology.orchestra.run.smithi110.stdout: "size": 20, 2024-04-19T01:05:07.759 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 600, 2024-04-19T01:05:07.759 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [ 2024-04-19T01:05:07.759 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.759 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 39/31 ECSubWrite(tid=32, reqid=client.4604.0:181, at_version=39'16, trim_to=0'0, roll_forward_to=38'14))", 2024-04-19T01:05:07.759 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:09.820805+0000", 2024-04-19T01:05:07.759 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 57.937657645000002, 2024-04-19T01:05:07.759 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.013829808000000001, 2024-04-19T01:05:07.759 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:07.759 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:07.759 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:07.759 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:07.759 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.760 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:07.760 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.820805+0000", 2024-04-19T01:05:07.760 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.760 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.760 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.760 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:07.760 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.820805+0000", 2024-04-19T01:05:07.760 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.760 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.760 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.761 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:07.761 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.820806+0000", 2024-04-19T01:05:07.761 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.2699999999999999e-07 2024-04-19T01:05:07.761 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.761 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.761 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:07.761 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.821061+0000", 2024-04-19T01:05:07.761 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000255133 2024-04-19T01:05:07.761 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.761 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.762 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:07.762 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.821256+0000", 2024-04-19T01:05:07.762 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00019562400000000001 2024-04-19T01:05:07.762 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.762 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.762 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:07.762 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.821283+0000", 2024-04-19T01:05:07.763 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6712000000000001e-05 2024-04-19T01:05:07.763 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.763 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.763 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:07.763 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.821504+0000", 2024-04-19T01:05:07.763 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00022120199999999999 2024-04-19T01:05:07.763 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.763 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.763 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:07.763 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.821557+0000", 2024-04-19T01:05:07.763 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.2534000000000002e-05 2024-04-19T01:05:07.763 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.764 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.764 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:07.764 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.834585+0000", 2024-04-19T01:05:07.764 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.013027617999999999 2024-04-19T01:05:07.764 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.764 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.764 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:07.764 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:09.834635+0000", 2024-04-19T01:05:07.764 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.0257999999999999e-05 2024-04-19T01:05:07.764 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.764 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.765 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.765 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.765 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.765 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 43/31 ECSubWrite(tid=70, reqid=client.4604.0:235, at_version=43'43, trim_to=0'0, roll_forward_to=41'37))", 2024-04-19T01:05:07.765 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.870038+0000", 2024-04-19T01:05:07.765 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 53.888424282999999, 2024-04-19T01:05:07.765 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.020219719000000001, 2024-04-19T01:05:07.765 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:07.765 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:07.765 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:07.765 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:07.765 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.765 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:07.766 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.870038+0000", 2024-04-19T01:05:07.766 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.766 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.766 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.766 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:07.766 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.870038+0000", 2024-04-19T01:05:07.766 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.766 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.766 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.766 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:07.766 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.870040+0000", 2024-04-19T01:05:07.767 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.81e-06 2024-04-19T01:05:07.767 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.767 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.767 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:07.767 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.870453+0000", 2024-04-19T01:05:07.767 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000412965 2024-04-19T01:05:07.767 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.767 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.767 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:07.767 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.870494+0000", 2024-04-19T01:05:07.767 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.0938e-05 2024-04-19T01:05:07.767 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.768 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.768 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:07.768 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.870510+0000", 2024-04-19T01:05:07.768 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5458999999999999e-05 2024-04-19T01:05:07.768 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.768 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.768 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:07.768 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.870799+0000", 2024-04-19T01:05:07.768 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00028974699999999998 2024-04-19T01:05:07.769 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.769 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.769 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:07.769 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.870832+0000", 2024-04-19T01:05:07.769 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.2434999999999997e-05 2024-04-19T01:05:07.769 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.769 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.769 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:07.769 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.890185+0000", 2024-04-19T01:05:07.769 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.019353467999999999 2024-04-19T01:05:07.769 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.770 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.770 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:07.770 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.890258+0000", 2024-04-19T01:05:07.770 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.2897000000000004e-05 2024-04-19T01:05:07.780 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.780 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.780 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.788 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.788 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.788 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.ds2 43/31 ECSubWrite(tid=38, reqid=client.4604.0:253, at_version=43'14, trim_to=0'0, roll_forward_to=43'12))", 2024-04-19T01:05:07.788 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.992478+0000", 2024-04-19T01:05:07.788 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 53.765985078, 2024-04-19T01:05:07.789 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.025708154, 2024-04-19T01:05:07.789 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:07.789 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:07.789 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:07.789 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:07.789 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.789 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:07.789 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.992478+0000", 2024-04-19T01:05:07.789 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.789 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.789 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.789 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:07.790 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.992478+0000", 2024-04-19T01:05:07.790 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.790 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.790 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.790 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:07.790 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.992480+0000", 2024-04-19T01:05:07.790 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9870000000000002e-06 2024-04-19T01:05:07.790 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.790 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.790 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:07.790 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.992860+0000", 2024-04-19T01:05:07.791 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00038019600000000002 2024-04-19T01:05:07.791 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.791 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.791 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:07.791 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.992891+0000", 2024-04-19T01:05:07.791 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.0982000000000003e-05 2024-04-19T01:05:07.791 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.791 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.791 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:07.791 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.992906+0000", 2024-04-19T01:05:07.791 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5503e-05 2024-04-19T01:05:07.791 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.791 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.792 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:07.796 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.993064+0000", 2024-04-19T01:05:07.796 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015782900000000001 2024-04-19T01:05:07.796 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.796 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.796 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:07.796 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.993111+0000", 2024-04-19T01:05:07.796 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.7138000000000002e-05 2024-04-19T01:05:07.796 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.796 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.796 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:07.796 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.018092+0000", 2024-04-19T01:05:07.797 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.024980822999999999 2024-04-19T01:05:07.797 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.797 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.797 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:07.797 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.018186+0000", 2024-04-19T01:05:07.797 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.3696000000000006e-05 2024-04-19T01:05:07.797 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.797 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.797 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.797 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.797 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.797 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.ds2 43/31 ECSubWrite(tid=39, reqid=client.4604.0:254, at_version=43'15, trim_to=0'0, roll_forward_to=43'12))", 2024-04-19T01:05:07.797 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.996882+0000", 2024-04-19T01:05:07.797 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 53.761580959, 2024-04-19T01:05:07.797 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.032356653999999999, 2024-04-19T01:05:07.798 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:07.798 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:07.798 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:07.798 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:07.798 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.798 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:07.798 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.996882+0000", 2024-04-19T01:05:07.798 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.798 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.798 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.798 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:07.798 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.996882+0000", 2024-04-19T01:05:07.798 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.799 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.799 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.799 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:07.799 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.996884+0000", 2024-04-19T01:05:07.799 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2029999999999999e-06 2024-04-19T01:05:07.799 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.799 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.799 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:07.799 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.997205+0000", 2024-04-19T01:05:07.799 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00032076899999999999 2024-04-19T01:05:07.799 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.799 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.799 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:07.799 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.997249+0000", 2024-04-19T01:05:07.800 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.4654000000000003e-05 2024-04-19T01:05:07.800 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.800 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.800 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:07.800 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.997273+0000", 2024-04-19T01:05:07.800 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3589000000000001e-05 2024-04-19T01:05:07.800 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.800 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.800 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:07.800 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.007691+0000", 2024-04-19T01:05:07.800 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.010418495999999999 2024-04-19T01:05:07.802 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.802 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.802 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:07.803 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.007717+0000", 2024-04-19T01:05:07.803 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5876000000000001e-05 2024-04-19T01:05:07.803 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.803 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.803 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:07.803 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.029151+0000", 2024-04-19T01:05:07.803 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.021433651000000001 2024-04-19T01:05:07.803 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.803 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.804 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:07.804 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.029238+0000", 2024-04-19T01:05:07.804 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.7416000000000005e-05 2024-04-19T01:05:07.804 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.804 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.804 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.804 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.804 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.804 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.ds2 43/31 ECSubWrite(tid=40, reqid=client.4604.0:255, at_version=43'16, trim_to=0'0, roll_forward_to=43'12))", 2024-04-19T01:05:07.804 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.002736+0000", 2024-04-19T01:05:07.804 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 53.755726719999998, 2024-04-19T01:05:07.804 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.026570379000000002, 2024-04-19T01:05:07.804 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:07.804 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:07.805 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:07.805 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:07.805 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.805 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:07.805 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.002736+0000", 2024-04-19T01:05:07.805 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.805 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.805 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.805 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:07.805 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.002736+0000", 2024-04-19T01:05:07.805 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.805 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.805 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.805 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:07.806 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.002738+0000", 2024-04-19T01:05:07.806 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0899999999999999e-06 2024-04-19T01:05:07.806 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.806 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.806 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:07.806 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.002782+0000", 2024-04-19T01:05:07.806 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.3611e-05 2024-04-19T01:05:07.806 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.806 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.806 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:07.806 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.002823+0000", 2024-04-19T01:05:07.806 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.0954999999999997e-05 2024-04-19T01:05:07.806 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.806 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.806 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:07.807 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.002845+0000", 2024-04-19T01:05:07.807 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2775000000000001e-05 2024-04-19T01:05:07.807 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.807 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.807 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:07.807 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.019866+0000", 2024-04-19T01:05:07.807 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.017021162999999999 2024-04-19T01:05:07.807 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.807 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.807 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:07.807 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.019893+0000", 2024-04-19T01:05:07.807 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6176000000000001e-05 2024-04-19T01:05:07.807 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.807 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.808 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:07.808 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.029258+0000", 2024-04-19T01:05:07.808 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0093652990000000005 2024-04-19T01:05:07.808 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.808 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.808 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:07.808 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.029306+0000", 2024-04-19T01:05:07.808 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.8310000000000003e-05 2024-04-19T01:05:07.808 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.808 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.808 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.808 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.808 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.808 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 43/31 ECSubWrite(tid=76, reqid=client.4604.0:267, at_version=43'46, trim_to=0'0, roll_forward_to=43'45))", 2024-04-19T01:05:07.808 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.020527+0000", 2024-04-19T01:05:07.809 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 53.737935712000002, 2024-04-19T01:05:07.809 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.016212788999999998, 2024-04-19T01:05:07.809 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:07.809 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:07.809 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:07.809 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:07.809 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.809 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:07.809 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.020527+0000", 2024-04-19T01:05:07.809 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.809 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.809 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.809 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:07.809 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.020527+0000", 2024-04-19T01:05:07.809 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.810 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.810 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.810 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:07.810 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.020529+0000", 2024-04-19T01:05:07.810 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.046e-06 2024-04-19T01:05:07.810 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.810 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.810 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:07.810 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.020873+0000", 2024-04-19T01:05:07.810 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00034371800000000002 2024-04-19T01:05:07.810 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.810 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.810 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:07.810 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.020902+0000", 2024-04-19T01:05:07.811 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.8903000000000001e-05 2024-04-19T01:05:07.811 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.811 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.811 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:07.811 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.020924+0000", 2024-04-19T01:05:07.811 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2892e-05 2024-04-19T01:05:07.811 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.811 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.813 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:07.813 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.021071+0000", 2024-04-19T01:05:07.813 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00014612300000000001 2024-04-19T01:05:07.813 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.813 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.813 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:07.813 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.021113+0000", 2024-04-19T01:05:07.814 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.2573e-05 2024-04-19T01:05:07.814 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.814 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.814 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:07.814 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.036641+0000", 2024-04-19T01:05:07.814 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.015527501000000001 2024-04-19T01:05:07.814 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.814 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.814 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:07.814 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.036740+0000", 2024-04-19T01:05:07.814 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.9032999999999999e-05 2024-04-19T01:05:07.814 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.815 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.815 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.815 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.815 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.815 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.1s4 44/31 ECSubWrite(tid=15, reqid=client.4604.0:289, at_version=44'8, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:05:07.815 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.875457+0000", 2024-04-19T01:05:07.815 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 52.883005394999998, 2024-04-19T01:05:07.815 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.036567641999999997, 2024-04-19T01:05:07.815 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:07.815 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:07.815 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:07.815 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:07.815 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.815 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:07.815 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875457+0000", 2024-04-19T01:05:07.816 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.816 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.816 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.816 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:07.816 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875457+0000", 2024-04-19T01:05:07.816 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.816 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.816 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.816 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:07.816 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875458+0000", 2024-04-19T01:05:07.816 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.8400000000000004e-07 2024-04-19T01:05:07.816 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.816 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.817 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:07.817 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875519+0000", 2024-04-19T01:05:07.817 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.1377000000000005e-05 2024-04-19T01:05:07.817 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.817 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.817 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:07.817 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875588+0000", 2024-04-19T01:05:07.817 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.9099999999999999e-05 2024-04-19T01:05:07.817 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.817 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.817 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:07.817 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875615+0000", 2024-04-19T01:05:07.817 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6902000000000001e-05 2024-04-19T01:05:07.817 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.817 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.818 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:07.818 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875846+0000", 2024-04-19T01:05:07.818 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00023076699999999999 2024-04-19T01:05:07.818 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.818 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.818 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:07.818 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875874+0000", 2024-04-19T01:05:07.818 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.7725e-05 2024-04-19T01:05:07.818 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.818 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.818 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:07.818 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.911826+0000", 2024-04-19T01:05:07.818 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.035952348000000002 2024-04-19T01:05:07.818 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.819 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.819 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:07.819 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.912025+0000", 2024-04-19T01:05:07.819 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000198739 2024-04-19T01:05:07.819 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.819 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.819 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.819 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.819 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.819 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 44/31 ECSubWrite(tid=82, reqid=client.4604.0:280, at_version=44'50, trim_to=0'0, roll_forward_to=43'48))", 2024-04-19T01:05:07.819 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.880504+0000", 2024-04-19T01:05:07.819 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 52.877958301, 2024-04-19T01:05:07.819 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.041548157000000002, 2024-04-19T01:05:07.819 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:07.820 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:07.820 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:07.820 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:07.820 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.820 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:07.820 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.880504+0000", 2024-04-19T01:05:07.820 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.820 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.820 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.820 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:07.820 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.880504+0000", 2024-04-19T01:05:07.820 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.820 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.820 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.820 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:07.821 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.880506+0000", 2024-04-19T01:05:07.821 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9439999999999999e-06 2024-04-19T01:05:07.821 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.821 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.821 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:07.821 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.880844+0000", 2024-04-19T01:05:07.821 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00033782799999999998 2024-04-19T01:05:07.821 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.821 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.821 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:07.821 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.880877+0000", 2024-04-19T01:05:07.821 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.2524999999999998e-05 2024-04-19T01:05:07.821 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.821 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.821 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:07.823 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.880899+0000", 2024-04-19T01:05:07.823 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2656000000000001e-05 2024-04-19T01:05:07.823 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.824 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.824 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:07.824 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.881129+0000", 2024-04-19T01:05:07.824 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00022929600000000001 2024-04-19T01:05:07.824 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.824 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.824 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:07.824 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.881155+0000", 2024-04-19T01:05:07.825 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6729000000000001e-05 2024-04-19T01:05:07.825 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.825 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.825 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:07.825 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.921810+0000", 2024-04-19T01:05:07.825 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.040654807000000001 2024-04-19T01:05:07.825 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.825 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.825 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:07.825 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.922052+0000", 2024-04-19T01:05:07.825 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000242372 2024-04-19T01:05:07.825 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.825 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.825 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.825 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.826 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.826 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 44/31 ECSubWrite(tid=84, reqid=client.4604.0:281, at_version=44'51, trim_to=0'0, roll_forward_to=43'48))", 2024-04-19T01:05:07.826 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.906686+0000", 2024-04-19T01:05:07.826 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 52.851776903999998, 2024-04-19T01:05:07.826 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.018013254999999999, 2024-04-19T01:05:07.826 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:07.826 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:07.826 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:07.826 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:07.826 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.826 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:07.826 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.906686+0000", 2024-04-19T01:05:07.826 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.827 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.827 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.827 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:07.827 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.906686+0000", 2024-04-19T01:05:07.827 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.827 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.827 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.827 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:07.827 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.906688+0000", 2024-04-19T01:05:07.827 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1569999999999998e-06 2024-04-19T01:05:07.827 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.827 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.827 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:07.827 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.906789+0000", 2024-04-19T01:05:07.827 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000101488 2024-04-19T01:05:07.828 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.828 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.828 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:07.828 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.906831+0000", 2024-04-19T01:05:07.828 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.1488e-05 2024-04-19T01:05:07.828 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.828 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.828 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:07.828 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.906855+0000", 2024-04-19T01:05:07.828 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4269000000000001e-05 2024-04-19T01:05:07.828 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.828 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.828 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:07.828 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.913161+0000", 2024-04-19T01:05:07.828 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0063058070000000001 2024-04-19T01:05:07.829 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.829 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.829 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:07.829 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.913193+0000", 2024-04-19T01:05:07.829 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.2007e-05 2024-04-19T01:05:07.829 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.829 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.829 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:07.829 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.924624+0000", 2024-04-19T01:05:07.829 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.011430879999999999 2024-04-19T01:05:07.829 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.829 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.829 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:07.829 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.924699+0000", 2024-04-19T01:05:07.829 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.5159000000000003e-05 2024-04-19T01:05:07.830 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.830 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.830 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.830 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.830 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.830 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.0s1 44/31 ECSubWrite(tid=25, reqid=client.4604.0:284, at_version=44'22, trim_to=0'0, roll_forward_to=44'20))", 2024-04-19T01:05:07.830 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.922536+0000", 2024-04-19T01:05:07.830 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 52.835926626000003, 2024-04-19T01:05:07.830 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.029007871000000001, 2024-04-19T01:05:07.830 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:07.830 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:07.830 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:07.830 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:07.830 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.831 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:07.831 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.922536+0000", 2024-04-19T01:05:07.831 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.831 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.831 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.831 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:07.832 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.922536+0000", 2024-04-19T01:05:07.832 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.832 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.832 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.832 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:07.832 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.922538+0000", 2024-04-19T01:05:07.832 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0829999999999998e-06 2024-04-19T01:05:07.832 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.834 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.834 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:07.834 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.923282+0000", 2024-04-19T01:05:07.834 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000743554 2024-04-19T01:05:07.835 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.835 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.835 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:07.835 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.923311+0000", 2024-04-19T01:05:07.835 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.9147999999999999e-05 2024-04-19T01:05:07.835 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.835 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.835 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:07.835 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.923337+0000", 2024-04-19T01:05:07.835 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6648999999999998e-05 2024-04-19T01:05:07.835 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.835 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.835 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:07.835 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.923466+0000", 2024-04-19T01:05:07.836 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00012821100000000001 2024-04-19T01:05:07.836 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.836 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.836 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:07.836 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.923478+0000", 2024-04-19T01:05:07.836 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.2291e-05 2024-04-19T01:05:07.836 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.836 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.836 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:07.836 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.951479+0000", 2024-04-19T01:05:07.836 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.02800066 2024-04-19T01:05:07.836 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.836 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.836 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:07.836 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.951544+0000", 2024-04-19T01:05:07.837 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.5275000000000001e-05 2024-04-19T01:05:07.837 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.837 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.837 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.837 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.837 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.837 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 74/31 ECSubWrite(tid=174, reqid=client.4604.0:548, at_version=74'110, trim_to=0'0, roll_forward_to=70'108))", 2024-04-19T01:05:07.837 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:44.595181+0000", 2024-04-19T01:05:07.837 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 23.163281684000001, 2024-04-19T01:05:07.837 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.023202622999999999, 2024-04-19T01:05:07.837 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:07.837 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:07.837 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:07.837 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:07.838 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.838 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:07.838 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.595181+0000", 2024-04-19T01:05:07.838 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.838 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.838 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.838 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:07.838 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.595181+0000", 2024-04-19T01:05:07.838 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.838 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.838 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.838 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:07.838 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.595182+0000", 2024-04-19T01:05:07.838 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.13e-06 2024-04-19T01:05:07.839 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.839 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.839 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:07.839 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.595558+0000", 2024-04-19T01:05:07.839 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00037566500000000002 2024-04-19T01:05:07.839 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.839 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.839 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:07.839 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.595596+0000", 2024-04-19T01:05:07.839 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.7815999999999998e-05 2024-04-19T01:05:07.839 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.839 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.839 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:07.839 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.595624+0000", 2024-04-19T01:05:07.839 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.8906000000000001e-05 2024-04-19T01:05:07.840 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.840 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.840 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:07.840 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.595857+0000", 2024-04-19T01:05:07.840 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00023279099999999999 2024-04-19T01:05:07.840 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.840 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.840 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:07.840 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.595886+0000", 2024-04-19T01:05:07.840 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.8770000000000001e-05 2024-04-19T01:05:07.840 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.840 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.841 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:07.841 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.618262+0000", 2024-04-19T01:05:07.841 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.022375757999999999 2024-04-19T01:05:07.841 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.841 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.841 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:07.841 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.618384+0000", 2024-04-19T01:05:07.841 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000121787 2024-04-19T01:05:07.841 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.841 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.841 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.841 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.841 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.841 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.0s1 76/31 ECSubWrite(tid=66, reqid=client.4604.0:563, at_version=76'56, trim_to=0'0, roll_forward_to=67'54))", 2024-04-19T01:05:07.842 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:46.555265+0000", 2024-04-19T01:05:07.842 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 21.203197393, 2024-04-19T01:05:07.842 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.014802898, 2024-04-19T01:05:07.842 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:07.842 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:07.842 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:07.842 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:07.842 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.842 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:07.842 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:46.555265+0000", 2024-04-19T01:05:07.843 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.843 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.843 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.843 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:07.843 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:46.555265+0000", 2024-04-19T01:05:07.844 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.844 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.844 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.844 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:07.844 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:46.555266+0000", 2024-04-19T01:05:07.845 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.4700000000000001e-07 2024-04-19T01:05:07.845 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.845 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.845 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:07.845 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:46.555593+0000", 2024-04-19T01:05:07.845 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00032677999999999998 2024-04-19T01:05:07.845 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.845 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.845 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:07.845 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:46.555633+0000", 2024-04-19T01:05:07.846 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.0092000000000002e-05 2024-04-19T01:05:07.846 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.846 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.846 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:07.846 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:46.555658+0000", 2024-04-19T01:05:07.846 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5122000000000001e-05 2024-04-19T01:05:07.846 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.846 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.846 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:07.846 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:46.555833+0000", 2024-04-19T01:05:07.846 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00017493399999999999 2024-04-19T01:05:07.846 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.846 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.846 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:07.847 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:46.555860+0000", 2024-04-19T01:05:07.847 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.7589e-05 2024-04-19T01:05:07.847 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.847 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.847 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:07.847 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:46.569978+0000", 2024-04-19T01:05:07.847 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.014117299999999999 2024-04-19T01:05:07.847 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.847 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.847 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:07.847 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:46.570068+0000", 2024-04-19T01:05:07.847 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.0334000000000006e-05 2024-04-19T01:05:07.847 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.847 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.847 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.848 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.848 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.848 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.cs4 84/31 ECSubWrite(tid=119, reqid=osd.0.0:119, at_version=84'53, trim_to=0'0, roll_forward_to=82'51))", 2024-04-19T01:05:07.848 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:54.546893+0000", 2024-04-19T01:05:07.848 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 13.211569915, 2024-04-19T01:05:07.848 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.014222828999999999, 2024-04-19T01:05:07.848 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:07.848 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:07.848 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:07.848 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:07.848 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.848 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:07.848 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:54.546893+0000", 2024-04-19T01:05:07.848 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.849 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.849 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.849 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:07.849 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:54.546893+0000", 2024-04-19T01:05:07.849 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.849 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.849 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.849 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:07.849 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:54.546894+0000", 2024-04-19T01:05:07.849 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.0299999999999998e-07 2024-04-19T01:05:07.849 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.849 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.849 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:07.849 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:54.546984+0000", 2024-04-19T01:05:07.849 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.0341999999999997e-05 2024-04-19T01:05:07.850 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.850 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.850 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:07.850 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:54.547017+0000", 2024-04-19T01:05:07.850 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.2984999999999997e-05 2024-04-19T01:05:07.850 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.850 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.850 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:07.850 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:54.547040+0000", 2024-04-19T01:05:07.850 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3178999999999999e-05 2024-04-19T01:05:07.850 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.850 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.850 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:07.851 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:54.547794+0000", 2024-04-19T01:05:07.851 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00075413900000000002 2024-04-19T01:05:07.851 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.851 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.851 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:07.851 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:54.547820+0000", 2024-04-19T01:05:07.851 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6216000000000001e-05 2024-04-19T01:05:07.851 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.851 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.851 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:07.851 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:54.561024+0000", 2024-04-19T01:05:07.851 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.013203914000000001 2024-04-19T01:05:07.851 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.851 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.852 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:07.852 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:54.561116+0000", 2024-04-19T01:05:07.852 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.1250999999999998e-05 2024-04-19T01:05:07.852 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.852 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.852 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.852 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.852 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.852 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.1s4 84/31 ECSubWrite(tid=38, reqid=osd.6.0:38, at_version=84'30, trim_to=0'0, roll_forward_to=82'28))", 2024-04-19T01:05:07.852 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:54.551154+0000", 2024-04-19T01:05:07.852 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 13.207308340999999, 2024-04-19T01:05:07.852 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.017220893000000001, 2024-04-19T01:05:07.852 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:07.852 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:07.853 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:07.853 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:07.853 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.853 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:07.853 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:54.551154+0000", 2024-04-19T01:05:07.853 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.853 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.854 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.854 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:07.855 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:54.551154+0000", 2024-04-19T01:05:07.855 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.855 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.855 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.855 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:07.855 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:54.551155+0000", 2024-04-19T01:05:07.855 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.4300000000000003e-07 2024-04-19T01:05:07.855 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.856 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.856 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:07.856 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:54.551226+0000", 2024-04-19T01:05:07.856 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.1182999999999999e-05 2024-04-19T01:05:07.856 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.856 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.856 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:07.856 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:54.551250+0000", 2024-04-19T01:05:07.856 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4488999999999999e-05 2024-04-19T01:05:07.856 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.856 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.856 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:07.856 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:54.551269+0000", 2024-04-19T01:05:07.856 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.8352e-05 2024-04-19T01:05:07.856 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.857 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.857 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:07.857 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:54.552781+0000", 2024-04-19T01:05:07.857 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00151245 2024-04-19T01:05:07.857 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.857 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.857 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:07.857 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:54.552807+0000", 2024-04-19T01:05:07.857 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5908e-05 2024-04-19T01:05:07.857 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.857 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.857 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:07.857 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:54.568276+0000", 2024-04-19T01:05:07.857 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.01546904 2024-04-19T01:05:07.858 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.858 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.858 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:07.858 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:54.568375+0000", 2024-04-19T01:05:07.858 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.8927999999999999e-05 2024-04-19T01:05:07.858 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.858 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.858 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.858 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.858 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.858 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.es5 86/31 ECSubWrite(tid=195, reqid=client.4604.0:653, at_version=86'39, trim_to=0'0, roll_forward_to=84'38))", 2024-04-19T01:05:07.858 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:56.572682+0000", 2024-04-19T01:05:07.858 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 11.185780686999999, 2024-04-19T01:05:07.858 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.020998065999999999, 2024-04-19T01:05:07.858 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:07.859 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:07.859 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:07.859 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:07.859 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.859 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:07.859 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.572682+0000", 2024-04-19T01:05:07.859 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.859 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.859 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.859 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:07.859 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.572682+0000", 2024-04-19T01:05:07.859 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.859 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.859 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.860 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:07.860 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.572683+0000", 2024-04-19T01:05:07.860 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.1699999999999997e-07 2024-04-19T01:05:07.860 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.860 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.860 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:07.860 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.572975+0000", 2024-04-19T01:05:07.860 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000292596 2024-04-19T01:05:07.860 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.860 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.860 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:07.860 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.573007+0000", 2024-04-19T01:05:07.860 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.1785999999999997e-05 2024-04-19T01:05:07.860 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.860 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.861 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:07.861 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.573032+0000", 2024-04-19T01:05:07.861 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4896e-05 2024-04-19T01:05:07.861 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.861 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.861 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:07.861 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.573262+0000", 2024-04-19T01:05:07.861 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000230385 2024-04-19T01:05:07.861 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.861 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.861 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:07.861 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.573292+0000", 2024-04-19T01:05:07.861 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.9836e-05 2024-04-19T01:05:07.861 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.861 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.862 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:07.862 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.593615+0000", 2024-04-19T01:05:07.862 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.020323305999999999 2024-04-19T01:05:07.862 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.862 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.862 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:07.862 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.593680+0000", 2024-04-19T01:05:07.862 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.4543999999999997e-05 2024-04-19T01:05:07.862 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.862 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.862 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.862 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.862 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.862 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.0s1 91/31 ECSubWrite(tid=96, reqid=client.4604.0:683, at_version=91'89, trim_to=0'0, roll_forward_to=89'87))", 2024-04-19T01:05:07.862 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:01.594644+0000", 2024-04-19T01:05:07.863 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 6.1638186260000003, 2024-04-19T01:05:07.863 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.02175761, 2024-04-19T01:05:07.863 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:07.863 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:07.863 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:07.863 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:07.863 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.863 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:07.863 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.594644+0000", 2024-04-19T01:05:07.864 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.864 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.864 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.864 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:07.864 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.594644+0000", 2024-04-19T01:05:07.864 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.864 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.865 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.866 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:07.866 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.594645+0000", 2024-04-19T01:05:07.866 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.9599999999999999e-07 2024-04-19T01:05:07.866 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.866 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.866 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:07.866 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.594912+0000", 2024-04-19T01:05:07.866 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00026693099999999998 2024-04-19T01:05:07.866 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.866 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.866 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:07.866 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.594948+0000", 2024-04-19T01:05:07.866 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.6582000000000003e-05 2024-04-19T01:05:07.867 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.867 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.867 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:07.867 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.594975+0000", 2024-04-19T01:05:07.867 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6458999999999999e-05 2024-04-19T01:05:07.867 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.867 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.867 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:07.867 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.595154+0000", 2024-04-19T01:05:07.868 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000179884 2024-04-19T01:05:07.868 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.868 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.868 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:07.868 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.595187+0000", 2024-04-19T01:05:07.868 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.2919000000000001e-05 2024-04-19T01:05:07.868 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.868 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.868 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:07.868 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.616333+0000", 2024-04-19T01:05:07.868 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.021145586000000001 2024-04-19T01:05:07.868 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.868 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.869 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:07.869 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.616402+0000", 2024-04-19T01:05:07.869 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.8652999999999995e-05 2024-04-19T01:05:07.869 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.869 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.869 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.869 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.869 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.869 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 92/31 ECSubWrite(tid=203, reqid=client.4604.0:692, at_version=92'125, trim_to=0'0, roll_forward_to=89'122))", 2024-04-19T01:05:07.869 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:02.646026+0000", 2024-04-19T01:05:07.869 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 5.1124363730000004, 2024-04-19T01:05:07.869 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.015123141999999999, 2024-04-19T01:05:07.869 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:07.869 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:07.869 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:07.870 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:07.870 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.870 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:07.870 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.646026+0000", 2024-04-19T01:05:07.870 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.870 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.870 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.870 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:07.870 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.646026+0000", 2024-04-19T01:05:07.870 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.870 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.870 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.870 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:07.870 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.646028+0000", 2024-04-19T01:05:07.871 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9870000000000002e-06 2024-04-19T01:05:07.871 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.871 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.871 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:07.871 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.646347+0000", 2024-04-19T01:05:07.871 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00031919799999999998 2024-04-19T01:05:07.871 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.871 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.871 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:07.871 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.646382+0000", 2024-04-19T01:05:07.871 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.4264999999999998e-05 2024-04-19T01:05:07.871 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.871 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.871 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:07.872 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.646398+0000", 2024-04-19T01:05:07.872 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6203000000000001e-05 2024-04-19T01:05:07.872 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.872 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.872 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:07.872 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.646544+0000", 2024-04-19T01:05:07.872 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00014660699999999999 2024-04-19T01:05:07.872 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.872 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.872 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:07.872 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.646561+0000", 2024-04-19T01:05:07.872 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6844999999999999e-05 2024-04-19T01:05:07.872 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.872 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.873 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:07.873 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.661049+0000", 2024-04-19T01:05:07.873 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.014487567999999999 2024-04-19T01:05:07.873 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.873 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.873 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:07.873 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.661149+0000", 2024-04-19T01:05:07.873 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000100469 2024-04-19T01:05:07.874 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.874 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.874 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.874 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.874 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.874 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 92/31 ECSubWrite(tid=204, reqid=client.4604.0:693, at_version=92'126, trim_to=0'0, roll_forward_to=89'122))", 2024-04-19T01:05:07.874 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:02.665210+0000", 2024-04-19T01:05:07.874 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 5.0932526810000001, 2024-04-19T01:05:07.876 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.012848103, 2024-04-19T01:05:07.876 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:07.876 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:07.876 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:07.876 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:07.877 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.877 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:07.877 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.665210+0000", 2024-04-19T01:05:07.877 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.877 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.877 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.877 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:07.877 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.665210+0000", 2024-04-19T01:05:07.877 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.877 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.877 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.877 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:07.877 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.665212+0000", 2024-04-19T01:05:07.877 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0040000000000002e-06 2024-04-19T01:05:07.878 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.878 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.878 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:07.878 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.665489+0000", 2024-04-19T01:05:07.878 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000277337 2024-04-19T01:05:07.878 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.878 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.878 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:07.878 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.665519+0000", 2024-04-19T01:05:07.878 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.0105000000000001e-05 2024-04-19T01:05:07.878 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.878 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.878 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:07.878 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.665536+0000", 2024-04-19T01:05:07.879 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6166000000000001e-05 2024-04-19T01:05:07.879 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.879 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.879 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:07.879 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.665693+0000", 2024-04-19T01:05:07.879 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000157756 2024-04-19T01:05:07.879 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.879 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.879 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:07.879 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.665722+0000", 2024-04-19T01:05:07.879 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.8949e-05 2024-04-19T01:05:07.879 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.879 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.879 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:07.879 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.677993+0000", 2024-04-19T01:05:07.880 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.012270526 2024-04-19T01:05:07.880 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.880 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.880 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:07.880 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.678058+0000", 2024-04-19T01:05:07.880 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.5259999999999995e-05 2024-04-19T01:05:07.880 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.880 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.880 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.880 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.880 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.881 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.es5 92/31 ECSubWrite(tid=209, reqid=client.4604.0:701, at_version=92'41, trim_to=0'0, roll_forward_to=86'39))", 2024-04-19T01:05:07.881 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:02.701192+0000", 2024-04-19T01:05:07.881 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 5.0572711019999996, 2024-04-19T01:05:07.881 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.025340898000000001, 2024-04-19T01:05:07.881 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:07.881 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:07.881 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:07.881 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:07.881 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.881 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:07.881 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.701192+0000", 2024-04-19T01:05:07.881 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.881 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.881 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.881 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:07.882 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.701192+0000", 2024-04-19T01:05:07.882 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.882 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.882 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.882 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:07.882 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.701194+0000", 2024-04-19T01:05:07.882 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.114e-06 2024-04-19T01:05:07.882 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.896 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.896 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:07.896 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.701818+0000", 2024-04-19T01:05:07.896 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00062439099999999997 2024-04-19T01:05:07.896 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.896 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.896 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:07.896 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.701859+0000", 2024-04-19T01:05:07.896 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.1227999999999999e-05 2024-04-19T01:05:07.896 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.896 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.896 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:07.896 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.701887+0000", 2024-04-19T01:05:07.896 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.7276000000000001e-05 2024-04-19T01:05:07.897 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.897 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.897 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:07.897 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.702072+0000", 2024-04-19T01:05:07.897 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000185911 2024-04-19T01:05:07.897 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.897 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.897 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:07.897 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.702108+0000", 2024-04-19T01:05:07.898 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.6009e-05 2024-04-19T01:05:07.898 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.898 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.898 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:07.898 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.726458+0000", 2024-04-19T01:05:07.898 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.024349546 2024-04-19T01:05:07.898 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.899 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.899 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:07.899 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.726532+0000", 2024-04-19T01:05:07.899 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.4423000000000002e-05 2024-04-19T01:05:07.899 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.899 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.899 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.899 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.899 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.899 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDPGPush(3.7s6 97/96 [PushOp(3:e2c4c59f:::smithi15528762-35:head, version: 67'21, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:e2c4c59f:::smithi15528762-35:head@67'21, size: 3080192, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-04-19T01:05:07.899 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:07.568593+0000", 2024-04-19T01:05:07.900 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 0.18987005000000001, 2024-04-19T01:05:07.900 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.017052013000000001, 2024-04-19T01:05:07.900 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:07.900 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:07.900 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:07.900 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:07.900 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.900 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:07.900 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:07.568593+0000", 2024-04-19T01:05:07.900 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.900 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.900 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.901 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:07.901 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:07.568593+0000", 2024-04-19T01:05:07.901 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:07.901 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.901 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.901 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:07.901 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:07.568594+0000", 2024-04-19T01:05:07.901 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.3459999999999999e-06 2024-04-19T01:05:07.901 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.901 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.901 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:07.901 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:07.569322+0000", 2024-04-19T01:05:07.901 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00072777699999999998 2024-04-19T01:05:07.901 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.902 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.902 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:07.902 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:07.569353+0000", 2024-04-19T01:05:07.902 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.1347000000000003e-05 2024-04-19T01:05:07.902 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.902 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.902 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:07.902 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:07.569372+0000", 2024-04-19T01:05:07.902 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9377000000000001e-05 2024-04-19T01:05:07.902 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.902 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.903 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:07.903 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:07.569581+0000", 2024-04-19T01:05:07.903 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000208319 2024-04-19T01:05:07.903 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:07.903 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:07.903 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:07.903 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:07.585645+0000", 2024-04-19T01:05:07.904 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.016063846999999999 2024-04-19T01:05:07.904 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.904 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.904 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.904 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:07.904 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:07.904 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:05:07.905 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-04-19T01:05:07.906 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T01:05:07.906 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-04-19T01:05:07.907 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T01:05:07.907 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-04-19T01:05:07.907 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-04-19T01:05:07.908 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T01:05:07.908 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-04-19T01:05:07.908 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T01:05:07.909 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-04-19T01:05:07.909 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T01:05:07.909 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-04-19T01:05:07.909 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T01:05:07.910 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:05:07.901+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:05:07.920 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-04-19T01:05:07.920 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T01:05:07.933 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-04-19T01:05:07.933 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T01:05:07.953 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2024-04-19T01:05:07.953 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T01:05:07.973 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2024-04-19T01:05:07.973 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T01:05:07.994 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2024-04-19T01:05:07.994 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T01:05:08.006 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:05:08.001+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:05:08.017 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2024-04-19T01:05:08.017 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T01:05:08.036 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2024-04-19T01:05:08.036 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-19T01:05:08.058 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2024-04-19T01:05:08.058 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-19T01:05:08.080 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2024-04-19T01:05:08.080 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-19T01:05:08.086 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:08.086 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:08.100 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-04-19T01:05:08.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:05:08.101+0000 7f1deffff640 0 -- 172.21.15.33:0/4175348059 >> [v2:172.21.15.155:6807/741132602,v1:172.21.15.155:6810/741132602] conn(0x7f1df0224ac0 msgr2=0x7f1df0226eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-19T01:05:08.108 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2024-04-19T01:05:08.108 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:05:08.108+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:05:08.215 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:05:08.214+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:05:08.265 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:08.265 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:08.279 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-04-19T01:05:08.331 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:05:08.329+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:05:08.402 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:08.402 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:08.416 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-19T01:05:08.426 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:05:08.414+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:05:08.520 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:05:08.513+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:05:08.524 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2024-04-19T01:05:08.531 INFO:tasks.rados.rados.0.smithi155.stdout:302: done (1 left) 2024-04-19T01:05:08.531 INFO:tasks.rados.rados.0.smithi155.stdout:303: done (0 left) 2024-04-19T01:05:08.531 INFO:tasks.rados.rados.0.smithi155.stdout:304: delete oid 50 current snap is 35 2024-04-19T01:05:08.553 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:08.553 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:08.567 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-19T01:05:08.621 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:05:08.620+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:05:08.697 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:08.697 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:08.704 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-19T01:05:08.747 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:05:08.746+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:05:08.822 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:05:08.820+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:05:08.825 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:08.825 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:08.840 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-19T01:05:08.923 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:05:08.921+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:05:09.007 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:09.007 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:09.021 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-19T01:05:09.024 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:05:09.021+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:05:09.125 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:05:09.124+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:05:09.146 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:09.147 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:09.159 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-19T01:05:09.226 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:05:09.225+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:05:09.281 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:09.282 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:09.299 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-19T01:05:09.349 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:05:09.346+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:05:09.422 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:09.422 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:09.428 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:05:09.426+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:05:09.437 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-19T01:05:09.529 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:05:09.525+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:05:09.560 INFO:tasks.rados.rados.0.smithi155.stdout:304: done (0 left) 2024-04-19T01:05:09.561 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 31 current snap is 35 2024-04-19T01:05:09.561 INFO:tasks.rados.rados.0.smithi155.stdout:305: seq_num 125 ranges {0=557056,557056=475136,1032192=507904,1540096=475136,2015232=98304} 2024-04-19T01:05:09.568 INFO:tasks.rados.rados.0.smithi155.stdout:305: writing smithi15528762-31 from 0 to 557056 tid 1 2024-04-19T01:05:09.575 INFO:tasks.rados.rados.0.smithi155.stdout:305: writing smithi15528762-31 from 557056 to 1032192 tid 2 2024-04-19T01:05:09.585 INFO:tasks.rados.rados.0.smithi155.stdout:305: writing smithi15528762-31 from 1032192 to 1540096 tid 3 2024-04-19T01:05:09.593 INFO:tasks.rados.rados.0.smithi155.stdout:305: writing smithi15528762-31 from 1540096 to 2015232 tid 4 2024-04-19T01:05:09.595 INFO:tasks.rados.rados.0.smithi155.stdout:305: writing smithi15528762-31 from 2015232 to 2113536 tid 5 2024-04-19T01:05:09.595 INFO:tasks.rados.rados.0.smithi155.stdout:306: copy_from oid 17 from oid 19 current snap is 35 2024-04-19T01:05:09.595 INFO:tasks.rados.rados.0.smithi155.stdout:307: snap_remove snap 33 2024-04-19T01:05:09.609 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:09.609 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:09.629 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:05:09.625+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:05:09.630 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-19T01:05:09.730 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:05:09.725+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:05:09.750 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:09.750 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:09.765 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-04-19T01:05:09.830 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:05:09.826+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:05:09.883 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:09.884 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:09.898 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-04-19T01:05:09.939 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:05:09.938+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:05:10.032 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:05:10.030+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:05:10.061 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:10.062 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:10.077 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-04-19T01:05:10.130 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-19T01:05:10.134 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:05:10.132+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:05:10.238 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:05:10.237+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:05:10.242 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:10.242 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:10.256 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-19T01:05:10.292 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-04-19T01:05:10.293 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T01:05:10.314 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-04-19T01:05:10.314 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T01:05:10.332 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-04-19T01:05:10.333 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T01:05:10.339 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:05:10.336+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:05:10.360 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-04-19T01:05:10.360 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T01:05:10.371 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-04-19T01:05:10.372 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T01:05:10.395 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-04-19T01:05:10.395 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:05:10.396 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:05:10.396 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-04-19T01:05:10.396 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:05:10.396 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T01:05:10.407 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-19T01:05:10.416 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-04-19T01:05:10.417 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T01:05:10.436 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-04-19T01:05:10.436 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T01:05:10.454 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:05:10.452+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:05:10.455 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2024-04-19T01:05:10.456 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T01:05:10.479 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2024-04-19T01:05:10.479 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T01:05:10.504 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2024-04-19T01:05:10.504 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T01:05:10.521 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2024-04-19T01:05:10.521 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T01:05:10.540 INFO:tasks.rados.rados.0.smithi155.stdout:307: done (2 left) 2024-04-19T01:05:10.540 INFO:tasks.rados.rados.0.smithi155.stdout:308: read oid 30 snap -1 2024-04-19T01:05:10.540 INFO:tasks.rados.rados.0.smithi155.stdout:308: expect (ObjNum 119 snap 30 seq_num 119) 2024-04-19T01:05:10.540 INFO:tasks.rados.rados.0.smithi155.stdout:309: rmattr oid 23 current snap is 35 2024-04-19T01:05:10.540 INFO:tasks.rados.rados.0.smithi155.stdout:309: done (3 left) 2024-04-19T01:05:10.540 INFO:tasks.rados.rados.0.smithi155.stdout:310: snap_create 2024-04-19T01:05:10.540 INFO:tasks.rados.rados.0.smithi155.stdout:305: finishing write tid 1 to smithi15528762-31 2024-04-19T01:05:10.540 INFO:tasks.rados.rados.0.smithi155.stdout:305: finishing write tid 2 to smithi15528762-31 2024-04-19T01:05:10.541 INFO:tasks.rados.rados.0.smithi155.stdout:306: finishing copy_from racing read to smithi15528762-17 2024-04-19T01:05:10.541 INFO:tasks.rados.rados.0.smithi155.stdout:305: finishing write tid 3 to smithi15528762-31 2024-04-19T01:05:10.541 INFO:tasks.rados.rados.0.smithi155.stdout:305: finishing write tid 4 to smithi15528762-31 2024-04-19T01:05:10.541 INFO:tasks.rados.rados.0.smithi155.stdout:306: finishing copy_from to smithi15528762-17 2024-04-19T01:05:10.541 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 17 v 30 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-04-19T01:05:10.541 INFO:tasks.rados.rados.0.smithi155.stdout:305: finishing write tid 5 to smithi15528762-31 2024-04-19T01:05:10.541 INFO:tasks.rados.rados.0.smithi155.stdout:305: finishing write tid 6 to smithi15528762-31 2024-04-19T01:05:10.541 INFO:tasks.rados.rados.0.smithi155.stdout:305: finishing write tid 7 to smithi15528762-31 2024-04-19T01:05:10.541 INFO:tasks.rados.rados.0.smithi155.stdout:305: oid 31 updating version 0 to 72 2024-04-19T01:05:10.541 INFO:tasks.rados.rados.0.smithi155.stdout:305: oid 31 updating version 72 to 73 2024-04-19T01:05:10.541 INFO:tasks.rados.rados.0.smithi155.stdout:305: oid 31 version 73 is already newer than 69 2024-04-19T01:05:10.541 INFO:tasks.rados.rados.0.smithi155.stdout:305: oid 31 version 73 is already newer than 71 2024-04-19T01:05:10.541 INFO:tasks.rados.rados.0.smithi155.stdout:305: oid 31 version 73 is already newer than 68 2024-04-19T01:05:10.541 INFO:tasks.rados.rados.0.smithi155.stdout:305: oid 31 version 73 is already newer than 70 2024-04-19T01:05:10.541 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 31 v 73 (ObjNum 125 snap 35 seq_num 125) dirty exists 2024-04-19T01:05:10.542 INFO:tasks.rados.rados.0.smithi155.stdout:305: left oid 31 (ObjNum 125 snap 35 seq_num 125) 2024-04-19T01:05:10.542 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:05:10.537+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:05:10.548 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2024-04-19T01:05:10.548 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-19T01:05:10.556 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-19T01:05:10.580 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2024-04-19T01:05:10.580 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-19T01:05:10.587 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:05:10.587 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:05:10.587 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-04-19T01:05:10.587 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-04-19T01:05:10.587 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:05:10.589 INFO:tasks.rados.rados.0.smithi155.stdout:308: expect (ObjNum 119 snap 30 seq_num 119) 2024-04-19T01:05:10.601 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-19T01:05:10.602 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2024-04-19T01:05:10.602 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-19T01:05:10.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:05:10.601+0000 7fd560968640 0 -- 172.21.15.33:0/3763970234 >> [v2:172.21.15.155:6807/741132602,v1:172.21.15.155:6810/741132602] conn(0x7fd55c2241e0 msgr2=0x7fd55c2265d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-19T01:05:10.641 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:05:10.637+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:05:10.657 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [0] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-04-19T01:05:10.657 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-19T01:05:10.658 INFO:tasks.thrashosds.thrasher:inject_pause on osd.12 2024-04-19T01:05:10.658 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-19T01:05:10.658 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-19T01:05:10.658 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok config set filestore_inject_stall 3 2024-04-19T01:05:10.730 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:05:10.730 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-04-19T01:05:10.730 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-04-19T01:05:10.730 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-04-19T01:05:10.730 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.730 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.0s3 47/31 ECSubWrite(tid=32, reqid=client.4604.0:308, at_version=47'29, trim_to=0'0, roll_forward_to=47'28))", 2024-04-19T01:05:10.731 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:17.565539+0000", 2024-04-19T01:05:10.731 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 53.163975636000004, 2024-04-19T01:05:10.731 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.018109618000000001, 2024-04-19T01:05:10.731 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:10.731 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:10.731 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:10.731 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:10.731 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.731 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:10.731 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.565539+0000", 2024-04-19T01:05:10.731 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:10.731 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.731 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.731 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:10.732 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.565539+0000", 2024-04-19T01:05:10.732 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:10.732 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.732 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.732 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:10.732 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.565541+0000", 2024-04-19T01:05:10.732 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.04e-06 2024-04-19T01:05:10.732 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.732 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.732 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:10.732 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.566554+0000", 2024-04-19T01:05:10.732 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0010129029999999999 2024-04-19T01:05:10.732 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.732 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.732 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:10.733 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.566591+0000", 2024-04-19T01:05:10.733 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7061999999999999e-05 2024-04-19T01:05:10.733 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.733 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.733 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:10.733 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.566613+0000", 2024-04-19T01:05:10.733 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.145e-05 2024-04-19T01:05:10.733 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.733 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.733 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:10.733 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.566752+0000", 2024-04-19T01:05:10.733 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000138868 2024-04-19T01:05:10.733 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.733 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.734 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:10.734 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.566783+0000", 2024-04-19T01:05:10.734 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0772000000000003e-05 2024-04-19T01:05:10.734 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.734 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.734 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:10.734 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.583581+0000", 2024-04-19T01:05:10.734 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.016798124000000001 2024-04-19T01:05:10.735 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.736 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.736 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:10.736 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.583649+0000", 2024-04-19T01:05:10.736 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.8398999999999993e-05 2024-04-19T01:05:10.736 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:10.736 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:10.736 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:10.736 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.736 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.736 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.0s3 47/31 ECSubWrite(tid=33, reqid=client.4604.0:309, at_version=47'30, trim_to=0'0, roll_forward_to=47'28))", 2024-04-19T01:05:10.736 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:17.573407+0000", 2024-04-19T01:05:10.736 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 53.156108029000002, 2024-04-19T01:05:10.737 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.010864425, 2024-04-19T01:05:10.737 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:10.737 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:10.737 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:10.737 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:10.737 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.737 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:10.737 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.573407+0000", 2024-04-19T01:05:10.737 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:10.737 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.737 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.737 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:10.737 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.573407+0000", 2024-04-19T01:05:10.738 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:10.738 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.738 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.738 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:10.738 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.573408+0000", 2024-04-19T01:05:10.738 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4300000000000001e-06 2024-04-19T01:05:10.738 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.738 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.739 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:10.740 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.573474+0000", 2024-04-19T01:05:10.740 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5711999999999997e-05 2024-04-19T01:05:10.740 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.740 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.740 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:10.740 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.573506+0000", 2024-04-19T01:05:10.740 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1445999999999997e-05 2024-04-19T01:05:10.740 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.740 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.741 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:10.741 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.573526+0000", 2024-04-19T01:05:10.741 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9919999999999999e-05 2024-04-19T01:05:10.741 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.741 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.741 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:10.741 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.582034+0000", 2024-04-19T01:05:10.741 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.008508837 2024-04-19T01:05:10.741 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.741 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.741 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:10.741 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.582053+0000", 2024-04-19T01:05:10.741 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8816000000000001e-05 2024-04-19T01:05:10.742 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.742 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.742 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:10.742 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.584163+0000", 2024-04-19T01:05:10.742 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0021099439999999999 2024-04-19T01:05:10.742 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.743 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.743 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:10.743 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.584271+0000", 2024-04-19T01:05:10.743 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010832000000000001 2024-04-19T01:05:10.743 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:10.743 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:10.743 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:10.743 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.743 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.743 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as3 48/31 ECSubWrite(tid=71, reqid=client.4604.0:354, at_version=48'10, trim_to=0'0, roll_forward_to=34'8))", 2024-04-19T01:05:10.743 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.556344+0000", 2024-04-19T01:05:10.743 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 52.173170871000004, 2024-04-19T01:05:10.743 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.030027702, 2024-04-19T01:05:10.744 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:10.744 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:10.744 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:10.744 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:10.744 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.744 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:10.744 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.556344+0000", 2024-04-19T01:05:10.744 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:10.744 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.744 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.744 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:10.744 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.556344+0000", 2024-04-19T01:05:10.745 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:10.745 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.745 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.745 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:10.745 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.556345+0000", 2024-04-19T01:05:10.745 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1199999999999999e-07 2024-04-19T01:05:10.745 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.745 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.745 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:10.745 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.557285+0000", 2024-04-19T01:05:10.745 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00094096299999999996 2024-04-19T01:05:10.745 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.746 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.746 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:10.746 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.557335+0000", 2024-04-19T01:05:10.746 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9669000000000002e-05 2024-04-19T01:05:10.746 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.746 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.746 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:10.746 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.557358+0000", 2024-04-19T01:05:10.746 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3099999999999999e-05 2024-04-19T01:05:10.746 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.746 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.746 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:10.747 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.557490+0000", 2024-04-19T01:05:10.747 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013213400000000001 2024-04-19T01:05:10.749 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.749 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.749 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:10.749 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.557512+0000", 2024-04-19T01:05:10.749 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1603999999999998e-05 2024-04-19T01:05:10.750 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.750 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.750 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:10.750 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.586316+0000", 2024-04-19T01:05:10.750 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.028803714000000001 2024-04-19T01:05:10.750 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.750 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.751 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:10.751 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.586372+0000", 2024-04-19T01:05:10.751 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6205999999999997e-05 2024-04-19T01:05:10.751 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:10.751 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:10.751 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:10.751 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.751 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.751 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as3 65/31 ECSubWrite(tid=107, reqid=client.4604.0:478, at_version=65'26, trim_to=0'0, roll_forward_to=64'24))", 2024-04-19T01:05:10.751 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:35.638966+0000", 2024-04-19T01:05:10.751 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 35.090549373999998, 2024-04-19T01:05:10.751 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.01558386, 2024-04-19T01:05:10.752 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:10.752 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:10.752 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:10.752 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:10.752 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.752 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:10.752 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.638966+0000", 2024-04-19T01:05:10.752 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:10.752 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.752 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.753 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:10.753 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.638966+0000", 2024-04-19T01:05:10.753 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:10.753 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.753 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.753 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:10.753 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.638967+0000", 2024-04-19T01:05:10.753 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2500000000000001e-06 2024-04-19T01:05:10.753 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.754 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.754 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:10.754 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.639799+0000", 2024-04-19T01:05:10.754 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00083235399999999999 2024-04-19T01:05:10.754 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.754 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.754 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:10.754 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.639828+0000", 2024-04-19T01:05:10.754 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8513e-05 2024-04-19T01:05:10.754 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.754 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.754 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:10.755 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.639854+0000", 2024-04-19T01:05:10.755 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5769999999999999e-05 2024-04-19T01:05:10.755 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.755 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.755 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:10.755 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.640035+0000", 2024-04-19T01:05:10.755 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000181714 2024-04-19T01:05:10.755 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.755 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.755 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:10.755 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.640070+0000", 2024-04-19T01:05:10.755 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5179000000000003e-05 2024-04-19T01:05:10.756 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.756 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.756 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:10.756 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.654477+0000", 2024-04-19T01:05:10.756 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.014406402 2024-04-19T01:05:10.756 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.756 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.756 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:10.756 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.654550+0000", 2024-04-19T01:05:10.756 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2677999999999998e-05 2024-04-19T01:05:10.756 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:10.757 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:10.757 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:10.757 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.757 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.757 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.2s5 79/31 ECSubWrite(tid=252, reqid=client.4604.0:600, at_version=79'23, trim_to=0'0, roll_forward_to=79'21))", 2024-04-19T01:05:10.757 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:49.675205+0000", 2024-04-19T01:05:10.757 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 21.054309930999999, 2024-04-19T01:05:10.757 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.010261670000000001, 2024-04-19T01:05:10.757 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:10.757 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:10.757 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:10.757 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:10.758 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.758 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:10.758 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.675205+0000", 2024-04-19T01:05:10.758 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:10.758 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.758 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.758 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:10.758 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.675205+0000", 2024-04-19T01:05:10.758 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:10.758 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.763 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.763 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:10.763 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.675208+0000", 2024-04-19T01:05:10.763 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9739999999999998e-06 2024-04-19T01:05:10.763 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.763 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.763 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:10.763 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.675577+0000", 2024-04-19T01:05:10.763 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00036901499999999999 2024-04-19T01:05:10.763 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.764 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.764 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:10.764 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.675649+0000", 2024-04-19T01:05:10.764 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2331999999999998e-05 2024-04-19T01:05:10.764 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.764 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.764 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:10.764 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.675674+0000", 2024-04-19T01:05:10.764 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4326999999999999e-05 2024-04-19T01:05:10.764 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.764 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.765 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:10.765 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.675827+0000", 2024-04-19T01:05:10.765 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015302800000000001 2024-04-19T01:05:10.765 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.765 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.765 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:10.765 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.675854+0000", 2024-04-19T01:05:10.765 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6746000000000001e-05 2024-04-19T01:05:10.765 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.765 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.765 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:10.765 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.685384+0000", 2024-04-19T01:05:10.765 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0095306990000000001 2024-04-19T01:05:10.766 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.766 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.766 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:10.766 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.685467+0000", 2024-04-19T01:05:10.766 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.2548999999999999e-05 2024-04-19T01:05:10.766 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:10.766 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:10.766 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:10.766 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.766 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.767 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.4s3 79/31 ECSubWrite(tid=111, reqid=client.4604.0:605, at_version=79'36, trim_to=0'0, roll_forward_to=79'34))", 2024-04-19T01:05:10.767 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:49.680617+0000", 2024-04-19T01:05:10.767 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 21.048898318999999, 2024-04-19T01:05:10.767 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.010979739000000001, 2024-04-19T01:05:10.767 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:10.767 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:10.767 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:10.767 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:10.767 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.767 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:10.767 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.680617+0000", 2024-04-19T01:05:10.768 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:10.768 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.768 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.768 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:10.768 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.680617+0000", 2024-04-19T01:05:10.768 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:10.768 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.768 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.768 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:10.769 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.680619+0000", 2024-04-19T01:05:10.769 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1399999999999998e-06 2024-04-19T01:05:10.769 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.769 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.769 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:10.769 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.681174+0000", 2024-04-19T01:05:10.770 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00055461900000000001 2024-04-19T01:05:10.770 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.770 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.770 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:10.770 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.681195+0000", 2024-04-19T01:05:10.770 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1804999999999999e-05 2024-04-19T01:05:10.770 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.770 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.771 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:10.771 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.681205+0000", 2024-04-19T01:05:10.771 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6679999999999996e-06 2024-04-19T01:05:10.771 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.771 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.771 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:10.771 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.681312+0000", 2024-04-19T01:05:10.771 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000107293 2024-04-19T01:05:10.771 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.771 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.771 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:10.771 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.681327+0000", 2024-04-19T01:05:10.772 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.49e-05 2024-04-19T01:05:10.772 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.772 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.772 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:10.772 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.691558+0000", 2024-04-19T01:05:10.772 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.010231153 2024-04-19T01:05:10.772 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.772 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.772 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:10.772 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.691596+0000", 2024-04-19T01:05:10.772 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8161000000000003e-05 2024-04-19T01:05:10.772 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:10.772 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:10.773 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:10.773 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.773 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.773 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.4s3 81/31 ECSubWrite(tid=117, reqid=client.4604.0:618, at_version=81'41, trim_to=0'0, roll_forward_to=79'39))", 2024-04-19T01:05:10.773 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:51.536744+0000", 2024-04-19T01:05:10.773 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 19.192771315000002, 2024-04-19T01:05:10.773 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.018283939999999999, 2024-04-19T01:05:10.773 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:10.773 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:10.773 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:10.773 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:10.773 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.774 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:10.774 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.536744+0000", 2024-04-19T01:05:10.774 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:10.774 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.774 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.774 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:10.774 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.536744+0000", 2024-04-19T01:05:10.774 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:10.774 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.774 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.774 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:10.775 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.536744+0000", 2024-04-19T01:05:10.775 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6600000000000002e-07 2024-04-19T01:05:10.775 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.775 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.775 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:10.775 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.537473+0000", 2024-04-19T01:05:10.775 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00072840000000000003 2024-04-19T01:05:10.775 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.775 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.775 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:10.775 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.537505+0000", 2024-04-19T01:05:10.775 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2746000000000001e-05 2024-04-19T01:05:10.777 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.777 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.777 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:10.777 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.537536+0000", 2024-04-19T01:05:10.777 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0562000000000003e-05 2024-04-19T01:05:10.777 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.777 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.777 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:10.777 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.537733+0000", 2024-04-19T01:05:10.778 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019705800000000001 2024-04-19T01:05:10.778 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.778 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.778 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:10.778 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.537762+0000", 2024-04-19T01:05:10.778 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8958999999999998e-05 2024-04-19T01:05:10.778 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.778 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.778 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:10.778 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.554926+0000", 2024-04-19T01:05:10.778 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.017164338000000001 2024-04-19T01:05:10.778 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.779 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.779 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:10.779 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.555028+0000", 2024-04-19T01:05:10.779 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000101511 2024-04-19T01:05:10.779 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:10.779 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:10.779 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:10.779 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.779 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.779 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.4s3 81/31 ECSubWrite(tid=120, reqid=client.4604.0:619, at_version=81'42, trim_to=0'0, roll_forward_to=79'39))", 2024-04-19T01:05:10.779 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:51.557348+0000", 2024-04-19T01:05:10.779 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 19.172167130999998, 2024-04-19T01:05:10.780 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.015514271, 2024-04-19T01:05:10.780 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:10.780 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:10.780 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:10.780 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:10.780 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.780 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:10.780 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.557348+0000", 2024-04-19T01:05:10.780 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:10.780 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.780 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.780 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:10.780 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.557348+0000", 2024-04-19T01:05:10.781 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:10.781 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.781 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.781 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:10.781 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.557350+0000", 2024-04-19T01:05:10.781 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6279999999999999e-06 2024-04-19T01:05:10.781 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.781 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.781 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:10.781 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.557622+0000", 2024-04-19T01:05:10.781 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00027276399999999999 2024-04-19T01:05:10.781 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.782 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.782 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:10.782 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.557649+0000", 2024-04-19T01:05:10.782 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6412999999999999e-05 2024-04-19T01:05:10.782 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.782 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.782 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:10.782 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.557663+0000", 2024-04-19T01:05:10.782 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4472e-05 2024-04-19T01:05:10.782 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.782 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.783 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:10.783 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.557936+0000", 2024-04-19T01:05:10.783 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00027278099999999999 2024-04-19T01:05:10.783 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.783 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.783 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:10.783 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.557957+0000", 2024-04-19T01:05:10.783 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0625999999999999e-05 2024-04-19T01:05:10.783 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.783 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.783 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:10.783 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.572801+0000", 2024-04-19T01:05:10.783 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.014844375999999999 2024-04-19T01:05:10.784 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.784 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.784 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:10.784 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.572862+0000", 2024-04-19T01:05:10.784 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1210999999999994e-05 2024-04-19T01:05:10.784 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:10.784 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:10.784 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:10.784 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.784 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.784 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.4s3 81/31 ECSubWrite(tid=123, reqid=client.4604.0:620, at_version=81'43, trim_to=0'0, roll_forward_to=81'42))", 2024-04-19T01:05:10.784 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:51.576489+0000", 2024-04-19T01:05:10.785 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 19.153025933999999, 2024-04-19T01:05:10.785 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.010317349999999999, 2024-04-19T01:05:10.785 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:10.785 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:10.785 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:10.785 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:10.785 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.785 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:10.785 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.576489+0000", 2024-04-19T01:05:10.785 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:10.785 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.785 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.785 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:10.786 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.576489+0000", 2024-04-19T01:05:10.786 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:10.786 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.786 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.786 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:10.786 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.576491+0000", 2024-04-19T01:05:10.786 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4899999999999999e-06 2024-04-19T01:05:10.786 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.786 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.786 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:10.786 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.576704+0000", 2024-04-19T01:05:10.786 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021347099999999999 2024-04-19T01:05:10.786 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.787 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.787 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:10.787 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.576730+0000", 2024-04-19T01:05:10.787 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5499000000000001e-05 2024-04-19T01:05:10.787 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.787 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.787 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:10.787 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.576746+0000", 2024-04-19T01:05:10.787 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6036e-05 2024-04-19T01:05:10.787 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.787 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.787 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:10.788 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.576887+0000", 2024-04-19T01:05:10.788 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014133599999999999 2024-04-19T01:05:10.788 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.788 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.788 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:10.788 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.576915+0000", 2024-04-19T01:05:10.788 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7847999999999998e-05 2024-04-19T01:05:10.788 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.788 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.788 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:10.788 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.586737+0000", 2024-04-19T01:05:10.788 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0098220990000000008 2024-04-19T01:05:10.789 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.789 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.789 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:10.789 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.586806+0000", 2024-04-19T01:05:10.789 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.9571000000000002e-05 2024-04-19T01:05:10.789 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:10.789 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:10.789 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:10.789 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.789 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.789 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.4s3 85/31 ECSubWrite(tid=138, reqid=osd.10.0:138, at_version=85'58, trim_to=0'0, roll_forward_to=84'56))", 2024-04-19T01:05:10.789 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:55.549556+0000", 2024-04-19T01:05:10.789 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 15.179959324, 2024-04-19T01:05:10.790 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.010353438, 2024-04-19T01:05:10.790 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:10.790 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:10.790 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:10.790 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:10.790 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.790 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:10.790 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:55.549556+0000", 2024-04-19T01:05:10.790 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:10.790 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.790 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.790 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:10.791 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:55.549556+0000", 2024-04-19T01:05:10.791 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:10.791 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.791 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.791 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:10.791 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:55.549556+0000", 2024-04-19T01:05:10.791 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0299999999999999e-07 2024-04-19T01:05:10.791 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.791 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.791 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:10.791 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:55.549613+0000", 2024-04-19T01:05:10.791 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6502999999999998e-05 2024-04-19T01:05:10.792 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.792 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.792 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:10.792 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:55.549644+0000", 2024-04-19T01:05:10.792 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0769000000000003e-05 2024-04-19T01:05:10.792 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.792 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.792 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:10.792 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:55.549671+0000", 2024-04-19T01:05:10.792 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7158999999999999e-05 2024-04-19T01:05:10.792 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.792 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.792 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:10.793 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:55.549854+0000", 2024-04-19T01:05:10.793 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00018351500000000001 2024-04-19T01:05:10.793 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.793 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.793 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:10.793 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:55.549881+0000", 2024-04-19T01:05:10.793 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6846e-05 2024-04-19T01:05:10.793 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.793 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.793 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:10.793 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:55.559865+0000", 2024-04-19T01:05:10.793 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.009983713 2024-04-19T01:05:10.793 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.794 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.794 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:10.794 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:55.559909+0000", 2024-04-19T01:05:10.794 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4329999999999997e-05 2024-04-19T01:05:10.794 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:10.794 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:10.794 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:10.794 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.794 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.794 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.2s5 86/31 ECSubWrite(tid=265, reqid=client.4604.0:655, at_version=86'35, trim_to=0'0, roll_forward_to=86'33))", 2024-04-19T01:05:10.794 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:56.580820+0000", 2024-04-19T01:05:10.794 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.148695287000001, 2024-04-19T01:05:10.795 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.030714183999999999, 2024-04-19T01:05:10.795 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:10.795 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:10.795 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:10.795 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:10.795 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.795 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:10.795 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.580820+0000", 2024-04-19T01:05:10.795 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:10.795 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.795 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.795 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:10.795 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.580820+0000", 2024-04-19T01:05:10.796 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:10.796 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.796 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.796 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:10.796 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.580823+0000", 2024-04-19T01:05:10.796 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8090000000000001e-06 2024-04-19T01:05:10.796 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.796 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.796 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:10.796 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.581340+0000", 2024-04-19T01:05:10.796 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00051707599999999999 2024-04-19T01:05:10.796 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.797 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.797 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:10.797 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.581384+0000", 2024-04-19T01:05:10.797 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4369999999999997e-05 2024-04-19T01:05:10.797 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.797 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.797 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:10.797 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.581409+0000", 2024-04-19T01:05:10.797 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4913999999999999e-05 2024-04-19T01:05:10.797 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.797 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.797 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:10.797 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.581604+0000", 2024-04-19T01:05:10.798 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019476500000000001 2024-04-19T01:05:10.798 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.798 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.798 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:10.798 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.581631+0000", 2024-04-19T01:05:10.800 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6979e-05 2024-04-19T01:05:10.801 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.801 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.801 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:10.801 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.611484+0000", 2024-04-19T01:05:10.801 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.029853356000000001 2024-04-19T01:05:10.801 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.801 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.802 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:10.802 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.611534+0000", 2024-04-19T01:05:10.802 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9914999999999999e-05 2024-04-19T01:05:10.802 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:10.803 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:10.803 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:10.803 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.803 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.803 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.2s5 86/31 ECSubWrite(tid=266, reqid=client.4604.0:656, at_version=86'36, trim_to=0'0, roll_forward_to=86'33))", 2024-04-19T01:05:10.803 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:56.598798+0000", 2024-04-19T01:05:10.803 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.13071718, 2024-04-19T01:05:10.804 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.014313559, 2024-04-19T01:05:10.804 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:10.804 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:10.804 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:10.804 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:10.804 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.804 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:10.804 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.598798+0000", 2024-04-19T01:05:10.804 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:10.804 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.805 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.805 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:10.805 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.598798+0000", 2024-04-19T01:05:10.805 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:10.805 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.805 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.805 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:10.805 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.598800+0000", 2024-04-19T01:05:10.805 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5919999999999999e-06 2024-04-19T01:05:10.805 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.805 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.805 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:10.806 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.598988+0000", 2024-04-19T01:05:10.806 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00018729799999999999 2024-04-19T01:05:10.806 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.806 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.806 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:10.806 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.599030+0000", 2024-04-19T01:05:10.806 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2592999999999997e-05 2024-04-19T01:05:10.806 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.807 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.807 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:10.807 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.599056+0000", 2024-04-19T01:05:10.807 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5684999999999999e-05 2024-04-19T01:05:10.807 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.807 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.807 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:10.807 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.601825+0000", 2024-04-19T01:05:10.807 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0027694180000000001 2024-04-19T01:05:10.808 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.808 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.808 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:10.808 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.601849+0000", 2024-04-19T01:05:10.808 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.315e-05 2024-04-19T01:05:10.808 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.808 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.808 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:10.808 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.613040+0000", 2024-04-19T01:05:10.808 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.011191134 2024-04-19T01:05:10.808 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.808 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.809 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:10.809 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.613111+0000", 2024-04-19T01:05:10.809 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.1688999999999998e-05 2024-04-19T01:05:10.809 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:10.809 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:10.809 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:10.809 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.810 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.810 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.0s3 91/31 ECSubWrite(tid=96, reqid=client.4604.0:683, at_version=91'89, trim_to=0'0, roll_forward_to=89'87))", 2024-04-19T01:05:10.810 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:01.594726+0000", 2024-04-19T01:05:10.810 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 9.1347888909999995, 2024-04-19T01:05:10.810 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.011665662, 2024-04-19T01:05:10.810 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:10.810 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:10.810 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:10.810 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:10.811 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.811 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:10.811 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.594726+0000", 2024-04-19T01:05:10.811 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:10.811 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.811 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.812 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:10.812 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.594726+0000", 2024-04-19T01:05:10.812 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:10.812 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.813 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.814 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:10.814 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.594726+0000", 2024-04-19T01:05:10.814 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7300000000000002e-07 2024-04-19T01:05:10.814 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.814 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.814 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:10.814 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.595219+0000", 2024-04-19T01:05:10.814 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00049298400000000002 2024-04-19T01:05:10.814 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.815 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.815 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:10.815 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.595243+0000", 2024-04-19T01:05:10.815 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3645000000000001e-05 2024-04-19T01:05:10.815 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.815 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.815 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:10.816 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.595260+0000", 2024-04-19T01:05:10.816 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6739e-05 2024-04-19T01:05:10.816 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.816 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.816 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:10.816 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.595428+0000", 2024-04-19T01:05:10.817 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016768600000000001 2024-04-19T01:05:10.817 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.817 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.817 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:10.817 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.595461+0000", 2024-04-19T01:05:10.817 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3949999999999999e-05 2024-04-19T01:05:10.817 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.817 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.817 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:10.817 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.606319+0000", 2024-04-19T01:05:10.817 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.010858003 2024-04-19T01:05:10.817 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.817 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.817 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:10.818 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.606392+0000", 2024-04-19T01:05:10.818 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2381999999999999e-05 2024-04-19T01:05:10.818 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:10.818 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:10.818 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:10.818 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.818 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.818 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.0s3 91/31 ECSubWrite(tid=98, reqid=client.4604.0:685, at_version=91'91, trim_to=0'0, roll_forward_to=89'87))", 2024-04-19T01:05:10.818 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:01.618671+0000", 2024-04-19T01:05:10.818 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 9.1108442679999992, 2024-04-19T01:05:10.818 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.011554114000000001, 2024-04-19T01:05:10.818 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:10.818 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:10.818 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:10.819 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:10.819 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.819 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:10.819 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.618671+0000", 2024-04-19T01:05:10.819 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:10.819 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.819 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.819 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:10.820 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.618671+0000", 2024-04-19T01:05:10.820 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:10.820 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.820 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.820 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:10.820 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.618673+0000", 2024-04-19T01:05:10.820 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7439999999999999e-06 2024-04-19T01:05:10.820 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.820 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.820 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:10.820 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.618909+0000", 2024-04-19T01:05:10.820 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00023606699999999999 2024-04-19T01:05:10.820 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.820 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.821 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:10.821 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.618937+0000", 2024-04-19T01:05:10.821 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8163000000000001e-05 2024-04-19T01:05:10.821 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.821 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.821 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:10.821 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.618952+0000", 2024-04-19T01:05:10.821 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5173e-05 2024-04-19T01:05:10.821 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.822 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.822 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:10.822 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.619083+0000", 2024-04-19T01:05:10.822 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013083699999999999 2024-04-19T01:05:10.822 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.822 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.822 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:10.823 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.619102+0000", 2024-04-19T01:05:10.824 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8836999999999999e-05 2024-04-19T01:05:10.824 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.824 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.824 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:10.824 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.630158+0000", 2024-04-19T01:05:10.824 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.011056755 2024-04-19T01:05:10.825 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.825 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.825 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:10.825 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:01.630225+0000", 2024-04-19T01:05:10.825 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6538000000000006e-05 2024-04-19T01:05:10.825 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:10.825 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:10.825 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:10.825 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.826 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.826 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as3 97/31 ECSubWrite(tid=192, reqid=client.4604.0:722, at_version=97'88, trim_to=0'0, roll_forward_to=89'86))", 2024-04-19T01:05:10.826 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:07.582556+0000", 2024-04-19T01:05:10.826 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 3.1469589230000001, 2024-04-19T01:05:10.826 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.013295235000000001, 2024-04-19T01:05:10.826 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:10.826 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:10.826 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:10.826 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:10.826 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.826 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:10.826 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.582556+0000", 2024-04-19T01:05:10.826 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:10.827 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.827 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.827 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:10.827 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.582556+0000", 2024-04-19T01:05:10.827 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:10.827 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.827 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.827 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:10.827 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.582557+0000", 2024-04-19T01:05:10.827 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.037e-06 2024-04-19T01:05:10.827 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.827 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.827 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:10.827 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.583004+0000", 2024-04-19T01:05:10.828 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00044645399999999999 2024-04-19T01:05:10.828 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.828 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.828 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:10.828 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.583041+0000", 2024-04-19T01:05:10.828 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6986000000000001e-05 2024-04-19T01:05:10.828 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.828 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.828 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:10.828 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.583068+0000", 2024-04-19T01:05:10.828 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7140000000000001e-05 2024-04-19T01:05:10.828 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.828 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.828 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:10.829 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.583253+0000", 2024-04-19T01:05:10.829 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00018514900000000001 2024-04-19T01:05:10.829 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.829 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.829 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:10.829 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.583277+0000", 2024-04-19T01:05:10.829 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4239e-05 2024-04-19T01:05:10.829 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.829 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.829 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:10.829 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.595751+0000", 2024-04-19T01:05:10.829 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.012474004 2024-04-19T01:05:10.829 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.829 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.830 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:10.830 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.595851+0000", 2024-04-19T01:05:10.830 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000100226 2024-04-19T01:05:10.830 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:10.830 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:10.830 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:10.830 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.830 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.830 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as3 97/31 ECSubWrite(tid=194, reqid=client.4604.0:723, at_version=97'89, trim_to=0'0, roll_forward_to=89'86))", 2024-04-19T01:05:10.831 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:07.597536+0000", 2024-04-19T01:05:10.831 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 3.1319794729999999, 2024-04-19T01:05:10.831 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.012429079000000001, 2024-04-19T01:05:10.831 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:10.831 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:10.831 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:10.831 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:10.831 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.831 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:10.831 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.597536+0000", 2024-04-19T01:05:10.831 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:10.831 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.831 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.832 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:10.832 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.597536+0000", 2024-04-19T01:05:10.832 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:10.832 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.832 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.832 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:10.832 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.597538+0000", 2024-04-19T01:05:10.832 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.277e-06 2024-04-19T01:05:10.832 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.832 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.832 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:10.832 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.597844+0000", 2024-04-19T01:05:10.832 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00030648999999999997 2024-04-19T01:05:10.832 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.833 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.833 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:10.834 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.597885+0000", 2024-04-19T01:05:10.834 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0299000000000002e-05 2024-04-19T01:05:10.835 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.835 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.835 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:10.835 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.597911+0000", 2024-04-19T01:05:10.835 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6123e-05 2024-04-19T01:05:10.836 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.836 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.836 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:10.836 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.598299+0000", 2024-04-19T01:05:10.836 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00038795799999999999 2024-04-19T01:05:10.836 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.836 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.836 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:10.836 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.598327+0000", 2024-04-19T01:05:10.836 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8527999999999999e-05 2024-04-19T01:05:10.837 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.837 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.837 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:10.837 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.609892+0000", 2024-04-19T01:05:10.837 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.011564555000000001 2024-04-19T01:05:10.837 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.837 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.837 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:10.837 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.609965+0000", 2024-04-19T01:05:10.837 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2849000000000007e-05 2024-04-19T01:05:10.838 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:10.838 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:10.838 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:10.838 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.838 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.838 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as3 97/31 ECSubWrite(tid=196, reqid=client.4604.0:724, at_version=97'90, trim_to=0'0, roll_forward_to=97'88))", 2024-04-19T01:05:10.838 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:07.613835+0000", 2024-04-19T01:05:10.838 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 3.115680191, 2024-04-19T01:05:10.838 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.013393530000000001, 2024-04-19T01:05:10.838 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:10.838 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:10.838 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:10.838 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:10.839 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.839 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:10.839 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.613835+0000", 2024-04-19T01:05:10.839 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:10.839 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.839 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.839 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:10.839 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.613835+0000", 2024-04-19T01:05:10.839 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:10.839 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.839 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.839 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:10.839 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.613837+0000", 2024-04-19T01:05:10.839 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.843e-06 2024-04-19T01:05:10.840 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.840 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.840 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:10.840 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.614030+0000", 2024-04-19T01:05:10.840 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019310900000000001 2024-04-19T01:05:10.840 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.840 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.840 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:10.840 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.614062+0000", 2024-04-19T01:05:10.840 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2543000000000003e-05 2024-04-19T01:05:10.840 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.840 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.840 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:10.840 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.614079+0000", 2024-04-19T01:05:10.841 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.615e-05 2024-04-19T01:05:10.841 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.841 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.841 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:10.841 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.614248+0000", 2024-04-19T01:05:10.841 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016929600000000001 2024-04-19T01:05:10.841 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.841 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.841 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:10.841 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.614266+0000", 2024-04-19T01:05:10.841 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8623000000000001e-05 2024-04-19T01:05:10.841 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.841 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.841 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:10.842 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.627185+0000", 2024-04-19T01:05:10.842 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.012918743999999999 2024-04-19T01:05:10.842 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.842 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.842 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:10.842 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.627228+0000", 2024-04-19T01:05:10.842 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3222000000000001e-05 2024-04-19T01:05:10.842 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:10.842 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:10.842 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:10.842 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.842 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.842 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDPGPush(3.3s3 97/96 [PushOp(3:c03611b6:::smithi15528762-48:head, version: 52'44, data_included: [0~200704], data_size: 200704, omap_header_size: 0, omap_entries_size: 0, attrset_size: 44, recovery_info: ObjectRecoveryInfo(3:c03611b6:::smithi15528762-48:head@52'44, size: 802816, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 802816, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-04-19T01:05:10.842 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:07.669300+0000", 2024-04-19T01:05:10.843 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 3.0602151900000001, 2024-04-19T01:05:10.843 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0140806, 2024-04-19T01:05:10.843 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:10.843 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:10.843 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:10.843 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:10.843 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.843 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:10.844 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.669300+0000", 2024-04-19T01:05:10.844 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:10.844 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.844 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.844 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:10.844 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.669300+0000", 2024-04-19T01:05:10.845 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:10.845 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.846 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.846 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:10.846 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.669302+0000", 2024-04-19T01:05:10.846 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.64e-06 2024-04-19T01:05:10.846 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.846 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.846 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:10.847 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.669855+0000", 2024-04-19T01:05:10.847 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00055338800000000004 2024-04-19T01:05:10.847 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.847 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.847 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:10.847 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.669900+0000", 2024-04-19T01:05:10.847 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5153e-05 2024-04-19T01:05:10.847 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.847 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.847 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:10.847 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.669927+0000", 2024-04-19T01:05:10.848 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7423000000000002e-05 2024-04-19T01:05:10.848 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.848 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.848 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:10.848 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.670116+0000", 2024-04-19T01:05:10.848 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00018838500000000001 2024-04-19T01:05:10.848 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.849 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.849 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:10.849 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.683380+0000", 2024-04-19T01:05:10.849 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.013264611000000001 2024-04-19T01:05:10.849 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:10.849 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:10.849 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:10.849 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.849 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.849 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDPGPush(3.3s3 97/96 [PushOp(3:cbbd4e2f:::smithi15528762-23:head, version: 89'66, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:cbbd4e2f:::smithi15528762-23:head@89'66, size: 2129920, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-04-19T01:05:10.849 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:07.705295+0000", 2024-04-19T01:05:10.850 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 3.024220353, 2024-04-19T01:05:10.850 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.017631539000000002, 2024-04-19T01:05:10.850 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:10.850 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:10.850 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:10.850 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:10.850 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.850 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:10.850 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.705295+0000", 2024-04-19T01:05:10.850 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:10.850 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.851 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.851 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:10.851 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.705295+0000", 2024-04-19T01:05:10.851 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:10.851 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.851 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.851 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:10.851 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.705296+0000", 2024-04-19T01:05:10.851 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6470000000000001e-06 2024-04-19T01:05:10.851 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.851 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.851 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:10.851 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.705649+0000", 2024-04-19T01:05:10.851 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00035229599999999999 2024-04-19T01:05:10.852 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.852 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.852 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:10.852 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.705693+0000", 2024-04-19T01:05:10.852 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4459000000000002e-05 2024-04-19T01:05:10.852 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.852 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.852 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:10.852 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.705719+0000", 2024-04-19T01:05:10.852 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6242999999999999e-05 2024-04-19T01:05:10.852 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.852 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.852 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:10.852 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.705901+0000", 2024-04-19T01:05:10.853 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00018181999999999999 2024-04-19T01:05:10.853 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.853 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.853 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:10.853 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.722926+0000", 2024-04-19T01:05:10.853 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.017025074000000001 2024-04-19T01:05:10.853 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:10.853 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:10.853 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:10.853 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.853 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.853 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDPGPush(3.3s3 97/96 [PushOp(3:cbbd4e2f:::smithi15528762-23:head, version: 89'66, data_included: [262144~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:cbbd4e2f:::smithi15528762-23:head@89'66, size: 2129920, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-04-19T01:05:10.853 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:07.726191+0000", 2024-04-19T01:05:10.853 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 3.0033237119999998, 2024-04-19T01:05:10.854 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.017422659, 2024-04-19T01:05:10.854 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:10.854 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:10.854 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:10.855 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:10.855 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.855 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:10.855 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.726191+0000", 2024-04-19T01:05:10.855 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:10.855 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.855 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.855 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:10.855 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.726191+0000", 2024-04-19T01:05:10.855 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:10.855 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.855 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.855 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:10.856 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.726193+0000", 2024-04-19T01:05:10.856 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.683e-06 2024-04-19T01:05:10.856 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.856 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.856 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:10.856 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.726548+0000", 2024-04-19T01:05:10.856 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00035464300000000003 2024-04-19T01:05:10.856 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.856 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.856 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:10.856 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.726592+0000", 2024-04-19T01:05:10.856 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4492e-05 2024-04-19T01:05:10.856 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.856 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.856 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:10.857 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.726621+0000", 2024-04-19T01:05:10.857 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.887e-05 2024-04-19T01:05:10.857 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.857 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.857 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:10.857 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.726799+0000", 2024-04-19T01:05:10.857 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017782299999999999 2024-04-19T01:05:10.857 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:10.857 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:10.857 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:10.857 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.743614+0000", 2024-04-19T01:05:10.857 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.016815147999999999 2024-04-19T01:05:10.857 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:10.857 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:10.857 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:10.858 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:10.858 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:10.858 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:05:10.860 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:05:10.860 INFO:teuthology.orchestra.run.smithi033.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-19T01:05:10.860 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:05:10.860 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-19T01:05:10.861 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2024-04-19T01:05:10.862 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-19T01:05:10.862 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:05:10.857+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:05:10.961 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:05:10.961+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:05:10.987 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:05:10.987 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:05:10.987 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-04-19T01:05:10.987 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:05:11.002 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-19T01:05:11.062 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:05:11.057+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:05:11.123 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:05:11.123 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:05:11.123 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-04-19T01:05:11.124 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-04-19T01:05:11.124 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:05:11.137 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-19T01:05:11.163 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:05:11.162+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:05:11.263 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:05:11.263 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-04-19T01:05:11.264 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-04-19T01:05:11.264 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-04-19T01:05:11.264 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.264 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 43/31 ECSubWrite(tid=47, reqid=client.4604.0:263, at_version=43'8, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:05:11.264 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:13.999456+0000", 2024-04-19T01:05:11.264 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 57.263498947000002, 2024-04-19T01:05:11.264 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0090503839999999999, 2024-04-19T01:05:11.264 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:11.264 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:11.264 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:11.264 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:11.264 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.265 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:11.265 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.999456+0000", 2024-04-19T01:05:11.265 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.265 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.265 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.265 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:11.265 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.999456+0000", 2024-04-19T01:05:11.265 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.265 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.265 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.265 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:11.265 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.999458+0000", 2024-04-19T01:05:11.265 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.874e-06 2024-04-19T01:05:11.265 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.266 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.266 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:11.266 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.999813+0000", 2024-04-19T01:05:11.266 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00035491100000000002 2024-04-19T01:05:11.266 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.266 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.266 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:11.266 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.999832+0000", 2024-04-19T01:05:11.266 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9425000000000001e-05 2024-04-19T01:05:11.266 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.266 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.266 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:11.266 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.999844+0000", 2024-04-19T01:05:11.267 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1838000000000001e-05 2024-04-19T01:05:11.267 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.267 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.267 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.267 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.999950+0000", 2024-04-19T01:05:11.267 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000106125 2024-04-19T01:05:11.267 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.267 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.267 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:11.267 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:13.999965+0000", 2024-04-19T01:05:11.267 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4966e-05 2024-04-19T01:05:11.267 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.267 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.267 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:11.268 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.008459+0000", 2024-04-19T01:05:11.268 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0084938979999999997 2024-04-19T01:05:11.268 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.268 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.268 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:11.268 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:14.008506+0000", 2024-04-19T01:05:11.268 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7346999999999999e-05 2024-04-19T01:05:11.268 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.268 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.268 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.268 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.268 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.268 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 47/31 ECSubWrite(tid=61, reqid=client.4604.0:317, at_version=47'16, trim_to=0'0, roll_forward_to=45'14))", 2024-04-19T01:05:11.268 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:17.604486+0000", 2024-04-19T01:05:11.269 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 53.658469396999998, 2024-04-19T01:05:11.269 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.025312589, 2024-04-19T01:05:11.269 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:11.269 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:11.269 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:11.269 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:11.269 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.269 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:11.269 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.604486+0000", 2024-04-19T01:05:11.269 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.269 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.269 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.269 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:11.270 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.604486+0000", 2024-04-19T01:05:11.270 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.270 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.270 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.270 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:11.270 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.604486+0000", 2024-04-19T01:05:11.270 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.9999999999999997e-07 2024-04-19T01:05:11.270 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.270 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.270 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:11.270 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.604960+0000", 2024-04-19T01:05:11.270 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00047350300000000001 2024-04-19T01:05:11.270 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.270 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.271 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:11.271 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.605002+0000", 2024-04-19T01:05:11.271 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1949000000000001e-05 2024-04-19T01:05:11.271 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.271 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.271 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:11.271 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.605032+0000", 2024-04-19T01:05:11.271 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9865999999999998e-05 2024-04-19T01:05:11.271 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.271 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.271 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.271 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.605289+0000", 2024-04-19T01:05:11.271 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00025766500000000002 2024-04-19T01:05:11.271 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.272 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.272 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:11.272 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.605322+0000", 2024-04-19T01:05:11.272 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2845999999999997e-05 2024-04-19T01:05:11.272 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.272 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.272 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:11.272 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.629741+0000", 2024-04-19T01:05:11.272 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.024419335 2024-04-19T01:05:11.272 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.272 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.272 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:11.272 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.629798+0000", 2024-04-19T01:05:11.272 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6725000000000003e-05 2024-04-19T01:05:11.273 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.273 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.273 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.273 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.273 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.273 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 47/31 ECSubWrite(tid=62, reqid=client.4604.0:318, at_version=47'17, trim_to=0'0, roll_forward_to=45'14))", 2024-04-19T01:05:11.273 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:17.611532+0000", 2024-04-19T01:05:11.273 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 53.651422482999997, 2024-04-19T01:05:11.273 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.018327896, 2024-04-19T01:05:11.273 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:11.273 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:11.273 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:11.273 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:11.274 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.276 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:11.276 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.611532+0000", 2024-04-19T01:05:11.276 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.276 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.276 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.276 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:11.276 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.611532+0000", 2024-04-19T01:05:11.276 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.276 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.276 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.276 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:11.276 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.611534+0000", 2024-04-19T01:05:11.277 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.979e-06 2024-04-19T01:05:11.277 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.277 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.277 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:11.278 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.611863+0000", 2024-04-19T01:05:11.278 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00032853100000000002 2024-04-19T01:05:11.278 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.278 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.278 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:11.278 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.611907+0000", 2024-04-19T01:05:11.278 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4425000000000002e-05 2024-04-19T01:05:11.278 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.278 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.279 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:11.279 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.611935+0000", 2024-04-19T01:05:11.279 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7993000000000001e-05 2024-04-19T01:05:11.279 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.279 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.279 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.279 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.617071+0000", 2024-04-19T01:05:11.279 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0051354069999999998 2024-04-19T01:05:11.279 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.279 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.279 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:11.280 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.617091+0000", 2024-04-19T01:05:11.280 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0061999999999999e-05 2024-04-19T01:05:11.280 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.280 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.280 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:11.280 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.629827+0000", 2024-04-19T01:05:11.280 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.012735943 2024-04-19T01:05:11.280 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.280 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.280 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:11.280 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.629860+0000", 2024-04-19T01:05:11.280 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3556000000000002e-05 2024-04-19T01:05:11.280 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.280 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.281 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.281 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.281 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.281 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 47/31 ECSubWrite(tid=63, reqid=client.4604.0:319, at_version=47'18, trim_to=0'0, roll_forward_to=45'14))", 2024-04-19T01:05:11.281 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:17.621137+0000", 2024-04-19T01:05:11.281 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 53.641818235000002, 2024-04-19T01:05:11.281 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0092948300000000005, 2024-04-19T01:05:11.281 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:11.281 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:11.281 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:11.281 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:11.282 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.282 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:11.282 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.621137+0000", 2024-04-19T01:05:11.282 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.282 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.282 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.282 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:11.282 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.621137+0000", 2024-04-19T01:05:11.282 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.282 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.282 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.283 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:11.283 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.621139+0000", 2024-04-19T01:05:11.283 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3130000000000001e-06 2024-04-19T01:05:11.283 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.283 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.283 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:11.283 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.621276+0000", 2024-04-19T01:05:11.283 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013658400000000001 2024-04-19T01:05:11.284 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.284 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.284 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:11.284 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.621317+0000", 2024-04-19T01:05:11.284 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1152000000000002e-05 2024-04-19T01:05:11.284 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.284 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.285 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:11.285 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.621341+0000", 2024-04-19T01:05:11.285 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4600000000000002e-05 2024-04-19T01:05:11.285 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.285 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.285 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.285 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.627256+0000", 2024-04-19T01:05:11.285 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0059141890000000003 2024-04-19T01:05:11.285 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.285 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.285 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:11.285 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.627268+0000", 2024-04-19T01:05:11.285 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2903e-05 2024-04-19T01:05:11.285 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.286 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.286 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:11.286 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.630366+0000", 2024-04-19T01:05:11.286 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0030979879999999999 2024-04-19T01:05:11.286 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.286 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.286 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:11.286 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.630432+0000", 2024-04-19T01:05:11.286 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5100999999999999e-05 2024-04-19T01:05:11.286 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.286 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.286 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.286 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.287 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.287 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 47/31 ECSubWrite(tid=64, reqid=client.4604.0:320, at_version=47'19, trim_to=0'0, roll_forward_to=45'14))", 2024-04-19T01:05:11.287 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:17.623031+0000", 2024-04-19T01:05:11.287 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 53.639923500999998, 2024-04-19T01:05:11.287 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0081197209999999999, 2024-04-19T01:05:11.287 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:11.287 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:11.287 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:11.288 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:11.288 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.288 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:11.288 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.623031+0000", 2024-04-19T01:05:11.288 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.288 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.288 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.288 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:11.288 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.623031+0000", 2024-04-19T01:05:11.288 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.288 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.289 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.289 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:11.289 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.623033+0000", 2024-04-19T01:05:11.289 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.694e-06 2024-04-19T01:05:11.289 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.289 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.289 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:11.289 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.623067+0000", 2024-04-19T01:05:11.289 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3436999999999999e-05 2024-04-19T01:05:11.289 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.289 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.289 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:11.289 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.623093+0000", 2024-04-19T01:05:11.289 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6469999999999999e-05 2024-04-19T01:05:11.290 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.290 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.290 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:11.290 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.623114+0000", 2024-04-19T01:05:11.290 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1423000000000001e-05 2024-04-19T01:05:11.290 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.290 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.290 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.290 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.629946+0000", 2024-04-19T01:05:11.290 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0068313330000000002 2024-04-19T01:05:11.290 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.291 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.291 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:11.291 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.629963+0000", 2024-04-19T01:05:11.291 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7289e-05 2024-04-19T01:05:11.291 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.291 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.291 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:11.291 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.631074+0000", 2024-04-19T01:05:11.291 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011109570000000001 2024-04-19T01:05:11.291 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.291 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.291 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:11.291 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.631151+0000", 2024-04-19T01:05:11.291 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7118000000000003e-05 2024-04-19T01:05:11.292 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.292 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.292 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.292 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.292 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.292 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 48/31 ECSubWrite(tid=76, reqid=client.4604.0:345, at_version=48'22, trim_to=0'0, roll_forward_to=47'19))", 2024-04-19T01:05:11.292 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.537945+0000", 2024-04-19T01:05:11.292 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 52.725009507000003, 2024-04-19T01:05:11.292 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.034986066000000003, 2024-04-19T01:05:11.292 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:11.294 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:11.294 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:11.294 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:11.294 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.294 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:11.295 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.537945+0000", 2024-04-19T01:05:11.295 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.295 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.295 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.295 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:11.295 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.537945+0000", 2024-04-19T01:05:11.295 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.295 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.295 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.295 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:11.295 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.537948+0000", 2024-04-19T01:05:11.295 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6929999999999999e-06 2024-04-19T01:05:11.295 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.295 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.296 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:11.296 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.538485+0000", 2024-04-19T01:05:11.296 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00053661900000000001 2024-04-19T01:05:11.296 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.296 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.296 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:11.296 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.538530+0000", 2024-04-19T01:05:11.296 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5423000000000002e-05 2024-04-19T01:05:11.296 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.296 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.296 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:11.296 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.538558+0000", 2024-04-19T01:05:11.296 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7569999999999999e-05 2024-04-19T01:05:11.297 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.297 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.297 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.297 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.538778+0000", 2024-04-19T01:05:11.297 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000220478 2024-04-19T01:05:11.297 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.297 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.297 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:11.298 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.538806+0000", 2024-04-19T01:05:11.298 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8197000000000001e-05 2024-04-19T01:05:11.298 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.298 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.298 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:11.299 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.572854+0000", 2024-04-19T01:05:11.299 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.034047315000000002 2024-04-19T01:05:11.299 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.299 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.299 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:11.299 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.572931+0000", 2024-04-19T01:05:11.299 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7770999999999998e-05 2024-04-19T01:05:11.299 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.299 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.299 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.299 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.299 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.299 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 48/31 ECSubWrite(tid=78, reqid=client.4604.0:346, at_version=48'23, trim_to=0'0, roll_forward_to=48'21))", 2024-04-19T01:05:11.299 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.548797+0000", 2024-04-19T01:05:11.300 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 52.714157442000001, 2024-04-19T01:05:11.300 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.024207922, 2024-04-19T01:05:11.301 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:11.301 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:11.301 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:11.301 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:11.301 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.301 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:11.301 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.548797+0000", 2024-04-19T01:05:11.301 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.301 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.302 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.302 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:11.302 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.548797+0000", 2024-04-19T01:05:11.302 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.302 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.302 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.302 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:11.302 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.548799+0000", 2024-04-19T01:05:11.302 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.604e-06 2024-04-19T01:05:11.302 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.302 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.303 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:11.303 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.548991+0000", 2024-04-19T01:05:11.303 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019153400000000001 2024-04-19T01:05:11.303 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.303 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.303 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:11.303 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.549011+0000", 2024-04-19T01:05:11.303 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0332000000000001e-05 2024-04-19T01:05:11.303 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.303 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.303 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:11.304 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.549025+0000", 2024-04-19T01:05:11.304 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4260000000000001e-05 2024-04-19T01:05:11.304 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.304 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.304 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.304 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.552893+0000", 2024-04-19T01:05:11.304 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0038674460000000001 2024-04-19T01:05:11.304 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.304 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.304 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:11.304 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.552906+0000", 2024-04-19T01:05:11.304 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3105000000000001e-05 2024-04-19T01:05:11.304 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.305 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.305 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:11.305 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.572950+0000", 2024-04-19T01:05:11.305 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.020043787 2024-04-19T01:05:11.305 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.305 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.305 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:11.305 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.573005+0000", 2024-04-19T01:05:11.305 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.5853999999999998e-05 2024-04-19T01:05:11.305 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.305 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.305 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.306 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.306 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.306 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 48/31 ECSubWrite(tid=79, reqid=client.4604.0:347, at_version=48'24, trim_to=0'0, roll_forward_to=48'21))", 2024-04-19T01:05:11.306 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.555132+0000", 2024-04-19T01:05:11.306 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 52.707823009999998, 2024-04-19T01:05:11.306 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.018366424999999999, 2024-04-19T01:05:11.306 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:11.306 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:11.306 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:11.306 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:11.306 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.306 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:11.307 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.555132+0000", 2024-04-19T01:05:11.307 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.307 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.307 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.307 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:11.307 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.555132+0000", 2024-04-19T01:05:11.307 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.307 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.307 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.307 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:11.307 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.555133+0000", 2024-04-19T01:05:11.307 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.42e-06 2024-04-19T01:05:11.307 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.307 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.308 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:11.308 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.555208+0000", 2024-04-19T01:05:11.308 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4532999999999999e-05 2024-04-19T01:05:11.308 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.308 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.308 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:11.308 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.555254+0000", 2024-04-19T01:05:11.308 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6031e-05 2024-04-19T01:05:11.309 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.309 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.309 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:11.309 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.555268+0000", 2024-04-19T01:05:11.309 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4068e-05 2024-04-19T01:05:11.309 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.309 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.309 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.309 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.571711+0000", 2024-04-19T01:05:11.309 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.016442656999999999 2024-04-19T01:05:11.309 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.309 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.309 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:11.310 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.571720+0000", 2024-04-19T01:05:11.310 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1370000000000008e-06 2024-04-19T01:05:11.310 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.310 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.310 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:11.310 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.573461+0000", 2024-04-19T01:05:11.310 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0017408180000000001 2024-04-19T01:05:11.310 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.310 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.310 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:11.310 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.573498+0000", 2024-04-19T01:05:11.310 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7761e-05 2024-04-19T01:05:11.311 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.311 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.311 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.311 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.311 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.312 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as2 48/31 ECSubWrite(tid=71, reqid=client.4604.0:354, at_version=48'10, trim_to=0'0, roll_forward_to=34'8))", 2024-04-19T01:05:11.312 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.556963+0000", 2024-04-19T01:05:11.312 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 52.705992291999998, 2024-04-19T01:05:11.312 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.029734679, 2024-04-19T01:05:11.312 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:11.312 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:11.312 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:11.312 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:11.312 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.312 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:11.312 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.556963+0000", 2024-04-19T01:05:11.312 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.312 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.312 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.313 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:11.313 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.556963+0000", 2024-04-19T01:05:11.313 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.313 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.313 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.313 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:11.313 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.556963+0000", 2024-04-19T01:05:11.313 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5499999999999998e-07 2024-04-19T01:05:11.313 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.313 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.313 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:11.313 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.558128+0000", 2024-04-19T01:05:11.314 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011650180000000001 2024-04-19T01:05:11.314 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.314 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.314 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:11.314 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.558199+0000", 2024-04-19T01:05:11.314 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0630999999999995e-05 2024-04-19T01:05:11.314 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.314 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.314 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:11.314 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.558221+0000", 2024-04-19T01:05:11.314 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2546999999999999e-05 2024-04-19T01:05:11.314 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.314 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.315 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.315 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.558350+0000", 2024-04-19T01:05:11.315 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012838699999999999 2024-04-19T01:05:11.315 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.315 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.315 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:11.315 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.558370+0000", 2024-04-19T01:05:11.315 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0106999999999999e-05 2024-04-19T01:05:11.315 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.315 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.316 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:11.316 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.586643+0000", 2024-04-19T01:05:11.316 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.028273007999999999 2024-04-19T01:05:11.316 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.316 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.316 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:11.316 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.586697+0000", 2024-04-19T01:05:11.316 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4326000000000001e-05 2024-04-19T01:05:11.316 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.316 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.316 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.316 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.316 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.317 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 48/31 ECSubWrite(tid=80, reqid=client.4604.0:348, at_version=48'25, trim_to=0'0, roll_forward_to=48'21))", 2024-04-19T01:05:11.317 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.559574+0000", 2024-04-19T01:05:11.317 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 52.703381020000002, 2024-04-19T01:05:11.317 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.016989658000000001, 2024-04-19T01:05:11.317 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:11.317 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:11.317 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:11.317 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:11.317 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.317 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:11.317 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.559574+0000", 2024-04-19T01:05:11.318 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.318 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.318 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.318 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:11.318 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.559574+0000", 2024-04-19T01:05:11.318 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.318 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.318 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.318 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:11.318 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.559574+0000", 2024-04-19T01:05:11.319 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9400000000000001e-07 2024-04-19T01:05:11.319 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.319 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.319 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:11.319 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.559589+0000", 2024-04-19T01:05:11.320 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4873e-05 2024-04-19T01:05:11.320 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.320 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.320 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:11.320 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.559597+0000", 2024-04-19T01:05:11.320 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.1089999999999998e-06 2024-04-19T01:05:11.320 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.320 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.320 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:11.320 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.559605+0000", 2024-04-19T01:05:11.320 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.267e-06 2024-04-19T01:05:11.320 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.320 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.320 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.321 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.573085+0000", 2024-04-19T01:05:11.321 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.013480373 2024-04-19T01:05:11.321 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.321 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.321 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:11.321 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.573107+0000", 2024-04-19T01:05:11.321 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2031999999999998e-05 2024-04-19T01:05:11.322 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.322 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.322 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:11.322 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.576518+0000", 2024-04-19T01:05:11.322 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.003411458 2024-04-19T01:05:11.322 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.322 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.322 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:11.322 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.576564+0000", 2024-04-19T01:05:11.322 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5151999999999997e-05 2024-04-19T01:05:11.322 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.323 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.323 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.323 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.323 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.323 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as2 65/31 ECSubWrite(tid=107, reqid=client.4604.0:478, at_version=65'26, trim_to=0'0, roll_forward_to=64'24))", 2024-04-19T01:05:11.323 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:35.640013+0000", 2024-04-19T01:05:11.323 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 35.622942338000001, 2024-04-19T01:05:11.323 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.015040102, 2024-04-19T01:05:11.323 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:11.323 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:11.323 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:11.323 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:11.323 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.323 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:11.324 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.640013+0000", 2024-04-19T01:05:11.324 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.324 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.324 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.324 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:11.324 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.640013+0000", 2024-04-19T01:05:11.324 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.324 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.324 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.324 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:11.324 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.640014+0000", 2024-04-19T01:05:11.324 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0330000000000001e-06 2024-04-19T01:05:11.324 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.324 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.325 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:11.325 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.640436+0000", 2024-04-19T01:05:11.325 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00042256800000000001 2024-04-19T01:05:11.325 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.325 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.325 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:11.325 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.640475+0000", 2024-04-19T01:05:11.325 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9096e-05 2024-04-19T01:05:11.325 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.325 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.326 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:11.326 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.640499+0000", 2024-04-19T01:05:11.326 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4099000000000001e-05 2024-04-19T01:05:11.326 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.326 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.326 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.326 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.640651+0000", 2024-04-19T01:05:11.326 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015175799999999999 2024-04-19T01:05:11.326 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.326 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.326 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:11.326 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.640670+0000", 2024-04-19T01:05:11.326 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8519999999999999e-05 2024-04-19T01:05:11.327 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.327 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.327 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:11.327 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.654985+0000", 2024-04-19T01:05:11.327 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.014315682999999999 2024-04-19T01:05:11.327 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.327 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.327 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:11.327 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.655053+0000", 2024-04-19T01:05:11.327 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7345e-05 2024-04-19T01:05:11.327 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.327 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.327 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.327 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.328 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.328 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as2 65/31 ECSubWrite(tid=108, reqid=client.4604.0:479, at_version=65'27, trim_to=0'0, roll_forward_to=64'24))", 2024-04-19T01:05:11.328 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:35.655491+0000", 2024-04-19T01:05:11.328 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 35.607463545000002, 2024-04-19T01:05:11.328 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.010179575999999999, 2024-04-19T01:05:11.328 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:11.328 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:11.328 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:11.328 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:11.328 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.328 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:11.328 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.655491+0000", 2024-04-19T01:05:11.328 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.328 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.329 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.329 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:11.329 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.655491+0000", 2024-04-19T01:05:11.329 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.329 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.329 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.329 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:11.330 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.655493+0000", 2024-04-19T01:05:11.330 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.59e-06 2024-04-19T01:05:11.330 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.330 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.330 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:11.330 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.655680+0000", 2024-04-19T01:05:11.330 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000186937 2024-04-19T01:05:11.330 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.330 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.330 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:11.333 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.655708+0000", 2024-04-19T01:05:11.333 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8195999999999999e-05 2024-04-19T01:05:11.333 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.333 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.333 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:11.334 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.655724+0000", 2024-04-19T01:05:11.334 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5423000000000001e-05 2024-04-19T01:05:11.334 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.334 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.334 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.334 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.655856+0000", 2024-04-19T01:05:11.334 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000132528 2024-04-19T01:05:11.334 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.334 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.334 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:11.334 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.655882+0000", 2024-04-19T01:05:11.334 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5565999999999999e-05 2024-04-19T01:05:11.334 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.334 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.335 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:11.335 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.665599+0000", 2024-04-19T01:05:11.335 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0097172769999999999 2024-04-19T01:05:11.335 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.335 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.335 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:11.335 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:35.665671+0000", 2024-04-19T01:05:11.335 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2058999999999996e-05 2024-04-19T01:05:11.335 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.335 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.335 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.335 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.335 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.335 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as2 71/31 ECSubWrite(tid=134, reqid=client.4604.0:522, at_version=71'49, trim_to=0'0, roll_forward_to=70'47))", 2024-04-19T01:05:11.336 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:41.526442+0000", 2024-04-19T01:05:11.336 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 29.736513285000001, 2024-04-19T01:05:11.336 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0087350069999999995, 2024-04-19T01:05:11.336 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:11.336 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:11.336 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:11.336 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:11.336 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.336 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:11.336 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.526442+0000", 2024-04-19T01:05:11.336 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.336 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.336 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.336 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:11.337 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.526442+0000", 2024-04-19T01:05:11.337 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.337 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.337 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.340 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:11.340 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.526442+0000", 2024-04-19T01:05:11.340 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5600000000000005e-07 2024-04-19T01:05:11.340 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.340 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.340 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:11.340 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.526688+0000", 2024-04-19T01:05:11.340 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00024532699999999999 2024-04-19T01:05:11.340 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.340 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.340 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:11.341 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.526721+0000", 2024-04-19T01:05:11.341 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3853000000000003e-05 2024-04-19T01:05:11.341 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.341 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.341 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:11.341 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.526746+0000", 2024-04-19T01:05:11.341 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4439000000000001e-05 2024-04-19T01:05:11.341 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.341 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.341 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.341 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.526912+0000", 2024-04-19T01:05:11.341 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016591500000000001 2024-04-19T01:05:11.342 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.342 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.342 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:11.342 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.526938+0000", 2024-04-19T01:05:11.342 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5749999999999999e-05 2024-04-19T01:05:11.342 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.342 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.342 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:11.342 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.535082+0000", 2024-04-19T01:05:11.342 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0081442379999999998 2024-04-19T01:05:11.342 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.342 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.342 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:11.343 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.535177+0000", 2024-04-19T01:05:11.343 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4828999999999996e-05 2024-04-19T01:05:11.343 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.343 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.343 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.343 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.343 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.343 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.2s6 74/31 ECSubWrite(tid=246, reqid=osd.11.0:246, at_version=74'18, trim_to=0'0, roll_forward_to=72'16))", 2024-04-19T01:05:11.343 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:44.561627+0000", 2024-04-19T01:05:11.343 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 26.701327453000001, 2024-04-19T01:05:11.343 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0087837249999999992, 2024-04-19T01:05:11.343 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:11.343 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:11.343 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:11.344 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:11.344 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.344 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:11.344 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.561627+0000", 2024-04-19T01:05:11.344 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.344 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.344 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.344 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:11.344 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.561627+0000", 2024-04-19T01:05:11.344 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.344 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.344 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.344 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:11.344 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.561628+0000", 2024-04-19T01:05:11.344 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2e-07 2024-04-19T01:05:11.345 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.345 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.345 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:11.345 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.561685+0000", 2024-04-19T01:05:11.345 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6675999999999997e-05 2024-04-19T01:05:11.345 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.345 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.345 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:11.345 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.561730+0000", 2024-04-19T01:05:11.345 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5355999999999997e-05 2024-04-19T01:05:11.345 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.345 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.345 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:11.345 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.561754+0000", 2024-04-19T01:05:11.345 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4403e-05 2024-04-19T01:05:11.346 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.346 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.346 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.346 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.561910+0000", 2024-04-19T01:05:11.346 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015511100000000001 2024-04-19T01:05:11.346 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.346 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.346 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:11.346 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.561926+0000", 2024-04-19T01:05:11.346 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6733000000000001e-05 2024-04-19T01:05:11.347 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.347 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.347 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:11.347 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.570303+0000", 2024-04-19T01:05:11.347 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0083770240000000003 2024-04-19T01:05:11.347 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.347 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.347 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:11.347 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.570411+0000", 2024-04-19T01:05:11.347 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000107902 2024-04-19T01:05:11.347 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.347 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.347 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.347 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.347 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.348 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.1s6 86/31 ECSubWrite(tid=44, reqid=osd.6.0:44, at_version=86'38, trim_to=0'0, roll_forward_to=85'36))", 2024-04-19T01:05:11.348 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:56.528567+0000", 2024-04-19T01:05:11.348 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.734387957999999, 2024-04-19T01:05:11.348 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0083323939999999999, 2024-04-19T01:05:11.348 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:11.348 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:11.348 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:11.348 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:11.348 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.348 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:11.348 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.528567+0000", 2024-04-19T01:05:11.348 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.348 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.349 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.349 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:11.352 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.528567+0000", 2024-04-19T01:05:11.352 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.352 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.352 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.352 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:11.352 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.528568+0000", 2024-04-19T01:05:11.352 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5300000000000004e-07 2024-04-19T01:05:11.353 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.353 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.353 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:11.353 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.528623+0000", 2024-04-19T01:05:11.353 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.5166e-05 2024-04-19T01:05:11.353 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.353 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.353 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:11.353 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.528653+0000", 2024-04-19T01:05:11.353 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0654999999999997e-05 2024-04-19T01:05:11.353 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.353 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.353 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:11.353 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.528676+0000", 2024-04-19T01:05:11.354 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2237000000000001e-05 2024-04-19T01:05:11.354 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.354 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.354 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.354 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.528819+0000", 2024-04-19T01:05:11.354 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000143473 2024-04-19T01:05:11.354 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.354 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.354 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:11.354 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.528846+0000", 2024-04-19T01:05:11.354 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6502000000000001e-05 2024-04-19T01:05:11.354 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.354 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.354 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:11.355 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.536801+0000", 2024-04-19T01:05:11.355 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0079551970000000007 2024-04-19T01:05:11.355 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.355 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.355 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:11.355 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.536899+0000", 2024-04-19T01:05:11.355 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.8511000000000006e-05 2024-04-19T01:05:11.355 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.355 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.355 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.355 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.355 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.355 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.2s6 86/31 ECSubWrite(tid=265, reqid=client.4604.0:655, at_version=86'35, trim_to=0'0, roll_forward_to=86'33))", 2024-04-19T01:05:11.355 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:56.581157+0000", 2024-04-19T01:05:11.355 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.681798226, 2024-04-19T01:05:11.356 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.034472488000000003, 2024-04-19T01:05:11.356 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:11.356 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:11.356 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:11.356 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:11.356 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.356 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:11.356 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.581157+0000", 2024-04-19T01:05:11.356 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.356 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.356 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.356 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:11.356 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.581157+0000", 2024-04-19T01:05:11.356 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.357 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.357 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.357 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:11.357 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.581159+0000", 2024-04-19T01:05:11.357 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5770000000000001e-06 2024-04-19T01:05:11.358 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.358 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.358 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:11.358 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.581453+0000", 2024-04-19T01:05:11.358 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00029377899999999998 2024-04-19T01:05:11.358 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.358 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.358 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:11.358 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.581497+0000", 2024-04-19T01:05:11.358 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4053e-05 2024-04-19T01:05:11.358 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.358 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.359 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:11.359 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.581519+0000", 2024-04-19T01:05:11.359 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2090999999999999e-05 2024-04-19T01:05:11.359 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.359 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.359 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.359 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.581679+0000", 2024-04-19T01:05:11.359 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000159946 2024-04-19T01:05:11.359 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.359 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.359 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:11.359 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.581705+0000", 2024-04-19T01:05:11.359 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5462000000000001e-05 2024-04-19T01:05:11.359 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.360 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.360 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:11.360 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.615490+0000", 2024-04-19T01:05:11.360 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.033785002000000001 2024-04-19T01:05:11.360 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.360 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.360 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:11.360 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.615629+0000", 2024-04-19T01:05:11.360 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013957800000000001 2024-04-19T01:05:11.360 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.360 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.360 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.361 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.361 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.361 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.2s6 86/31 ECSubWrite(tid=266, reqid=client.4604.0:656, at_version=86'36, trim_to=0'0, roll_forward_to=86'33))", 2024-04-19T01:05:11.361 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:56.598018+0000", 2024-04-19T01:05:11.361 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.664937008999999, 2024-04-19T01:05:11.361 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.025308352999999999, 2024-04-19T01:05:11.361 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:11.361 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:11.361 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:11.361 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:11.361 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.361 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:11.361 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.598018+0000", 2024-04-19T01:05:11.361 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.361 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.362 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.362 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:11.362 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.598018+0000", 2024-04-19T01:05:11.362 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.362 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.362 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.362 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:11.362 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.598020+0000", 2024-04-19T01:05:11.362 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9700000000000002e-06 2024-04-19T01:05:11.362 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.362 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.362 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:11.362 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.598315+0000", 2024-04-19T01:05:11.363 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00029499300000000001 2024-04-19T01:05:11.363 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.363 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.363 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:11.363 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.598358+0000", 2024-04-19T01:05:11.364 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3599e-05 2024-04-19T01:05:11.364 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.364 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.364 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:11.364 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.598382+0000", 2024-04-19T01:05:11.364 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3051999999999999e-05 2024-04-19T01:05:11.364 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.364 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.364 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.365 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.601433+0000", 2024-04-19T01:05:11.365 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00305128 2024-04-19T01:05:11.365 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.365 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.365 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:11.365 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.601460+0000", 2024-04-19T01:05:11.365 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7327000000000001e-05 2024-04-19T01:05:11.365 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.365 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.365 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:11.365 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.623242+0000", 2024-04-19T01:05:11.365 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.021782341 2024-04-19T01:05:11.365 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.365 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.366 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:11.366 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.623326+0000", 2024-04-19T01:05:11.366 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.3790999999999998e-05 2024-04-19T01:05:11.366 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.366 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.366 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.366 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.366 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.366 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 86/31 ECSubWrite(tid=131, reqid=client.4604.0:660, at_version=86'57, trim_to=0'0, roll_forward_to=86'55))", 2024-04-19T01:05:11.366 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:56.635253+0000", 2024-04-19T01:05:11.366 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.627702304, 2024-04-19T01:05:11.366 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.013914375, 2024-04-19T01:05:11.366 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:11.366 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:11.367 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:11.367 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:11.367 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.367 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:11.367 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.635253+0000", 2024-04-19T01:05:11.368 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.368 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.368 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.368 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:11.369 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.635253+0000", 2024-04-19T01:05:11.369 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.369 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.369 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.369 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:11.369 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.635254+0000", 2024-04-19T01:05:11.369 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2300000000000001e-06 2024-04-19T01:05:11.369 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.369 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.369 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:11.369 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.636035+0000", 2024-04-19T01:05:11.369 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00078163400000000002 2024-04-19T01:05:11.369 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.369 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.370 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:11.370 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.636055+0000", 2024-04-19T01:05:11.370 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0006000000000001e-05 2024-04-19T01:05:11.370 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.370 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.370 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:11.370 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.636069+0000", 2024-04-19T01:05:11.370 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3760999999999999e-05 2024-04-19T01:05:11.370 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.370 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.370 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.370 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.636167+0000", 2024-04-19T01:05:11.370 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7483000000000002e-05 2024-04-19T01:05:11.371 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.371 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.371 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:11.371 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.636181+0000", 2024-04-19T01:05:11.371 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4343e-05 2024-04-19T01:05:11.371 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.371 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.371 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:11.371 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.649076+0000", 2024-04-19T01:05:11.371 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.012894526 2024-04-19T01:05:11.371 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.371 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.371 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:11.371 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.649167+0000", 2024-04-19T01:05:11.371 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1391999999999995e-05 2024-04-19T01:05:11.372 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.372 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.372 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.372 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.372 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.372 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as2 97/31 ECSubWrite(tid=192, reqid=client.4604.0:722, at_version=97'88, trim_to=0'0, roll_forward_to=89'86))", 2024-04-19T01:05:11.372 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:07.582697+0000", 2024-04-19T01:05:11.372 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 3.6802576240000002, 2024-04-19T01:05:11.372 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.012930539, 2024-04-19T01:05:11.372 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:11.372 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:11.372 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:11.372 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:11.372 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.372 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:11.373 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.582697+0000", 2024-04-19T01:05:11.373 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.373 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.373 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.373 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:11.373 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.582697+0000", 2024-04-19T01:05:11.373 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.373 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.373 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.373 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:11.373 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.582698+0000", 2024-04-19T01:05:11.373 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8699999999999995e-07 2024-04-19T01:05:11.373 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.373 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.374 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:11.374 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.583243+0000", 2024-04-19T01:05:11.374 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00054528499999999995 2024-04-19T01:05:11.374 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.374 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.374 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:11.374 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.583276+0000", 2024-04-19T01:05:11.374 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2505999999999997e-05 2024-04-19T01:05:11.374 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.374 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.374 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:11.374 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.583296+0000", 2024-04-19T01:05:11.374 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0403000000000001e-05 2024-04-19T01:05:11.375 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.375 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.375 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.375 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.583462+0000", 2024-04-19T01:05:11.375 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016562 2024-04-19T01:05:11.375 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.375 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.375 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:11.375 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.583480+0000", 2024-04-19T01:05:11.375 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8012999999999998e-05 2024-04-19T01:05:11.376 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.376 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.376 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:11.376 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.595535+0000", 2024-04-19T01:05:11.376 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.012054947 2024-04-19T01:05:11.376 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.376 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.376 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:11.376 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.595628+0000", 2024-04-19T01:05:11.376 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3177999999999995e-05 2024-04-19T01:05:11.376 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.376 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.376 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.376 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.377 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.377 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as2 97/31 ECSubWrite(tid=196, reqid=client.4604.0:724, at_version=97'90, trim_to=0'0, roll_forward_to=97'88))", 2024-04-19T01:05:11.377 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:07.614104+0000", 2024-04-19T01:05:11.377 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 3.6488508199999998, 2024-04-19T01:05:11.377 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.012938119, 2024-04-19T01:05:11.377 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:11.377 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:11.377 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:11.377 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:11.377 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.378 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:11.378 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.614104+0000", 2024-04-19T01:05:11.378 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.379 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.379 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.379 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:11.379 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.614104+0000", 2024-04-19T01:05:11.379 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.379 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.379 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.379 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:11.379 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.614106+0000", 2024-04-19T01:05:11.379 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9630000000000001e-06 2024-04-19T01:05:11.379 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.380 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.380 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:11.380 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.614363+0000", 2024-04-19T01:05:11.380 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00025654300000000003 2024-04-19T01:05:11.380 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.380 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.380 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:11.380 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.614390+0000", 2024-04-19T01:05:11.380 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7851000000000002e-05 2024-04-19T01:05:11.381 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.381 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.381 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:11.381 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.614405+0000", 2024-04-19T01:05:11.381 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4302e-05 2024-04-19T01:05:11.381 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.381 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.381 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.381 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.614532+0000", 2024-04-19T01:05:11.381 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012686699999999999 2024-04-19T01:05:11.381 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.381 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.381 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:11.382 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.614550+0000", 2024-04-19T01:05:11.382 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8502999999999999e-05 2024-04-19T01:05:11.382 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.382 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.382 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:11.382 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.627000+0000", 2024-04-19T01:05:11.382 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.012449454 2024-04-19T01:05:11.382 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.382 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.382 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:11.382 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.627042+0000", 2024-04-19T01:05:11.382 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2636000000000003e-05 2024-04-19T01:05:11.382 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.383 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.383 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.383 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.383 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.383 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:05:11.383 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:05:11.264+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:05:11.384 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-19T01:05:11.428 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:05:11.428+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:05:11.487 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:05:11.488+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:05:11.536 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2024-04-19T01:05:11.542 INFO:tasks.rados.rados.0.smithi155.stdout:305: done (3 left) 2024-04-19T01:05:11.542 INFO:tasks.rados.rados.0.smithi155.stdout:306: done (2 left) 2024-04-19T01:05:11.542 INFO:tasks.rados.rados.0.smithi155.stdout:308: done (1 left) 2024-04-19T01:05:11.542 INFO:tasks.rados.rados.0.smithi155.stdout:310: done (0 left) 2024-04-19T01:05:11.542 INFO:tasks.rados.rados.0.smithi155.stdout:311: snap_remove snap 26 2024-04-19T01:05:11.557 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:05:11.557 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:05:11.557 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-04-19T01:05:11.557 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:05:11.570 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-19T01:05:11.594 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:05:11.585+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:05:11.691 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:05:11.690+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:05:11.730 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:05:11.730 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:05:11.730 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-04-19T01:05:11.731 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-04-19T01:05:11.731 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:05:11.744 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-19T01:05:11.792 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:05:11.789+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:05:11.870 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:05:11.871 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-04-19T01:05:11.871 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-04-19T01:05:11.871 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-04-19T01:05:11.871 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.871 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4260.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-19T01:05:11.871 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:03:50.620310+0000", 2024-04-19T01:05:11.871 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 81.248048183999998, 2024-04-19T01:05:11.871 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.15782495899999999, 2024-04-19T01:05:11.871 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:11.872 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:11.872 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:11.872 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:11.872 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4260", 2024-04-19T01:05:11.872 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/1127060541", 2024-04-19T01:05:11.872 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1 2024-04-19T01:05:11.872 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.872 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:11.872 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.873 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:11.873 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.620310+0000", 2024-04-19T01:05:11.873 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.873 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.873 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.873 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:11.873 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.620310+0000", 2024-04-19T01:05:11.873 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.873 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.874 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.874 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:11.874 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.620313+0000", 2024-04-19T01:05:11.874 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.757e-06 2024-04-19T01:05:11.874 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.874 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.874 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:11.874 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.620327+0000", 2024-04-19T01:05:11.875 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4029000000000001e-05 2024-04-19T01:05:11.875 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.875 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.875 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:11.875 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.620362+0000", 2024-04-19T01:05:11.875 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5201999999999999e-05 2024-04-19T01:05:11.875 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.875 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.876 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:11.876 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.620383+0000", 2024-04-19T01:05:11.876 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1049000000000001e-05 2024-04-19T01:05:11.876 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.876 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.876 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.876 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.620512+0000", 2024-04-19T01:05:11.876 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012820900000000001 2024-04-19T01:05:11.876 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.877 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.877 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for peered", 2024-04-19T01:05:11.877 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.620542+0000", 2024-04-19T01:05:11.877 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0264999999999999e-05 2024-04-19T01:05:11.877 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.877 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.877 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.877 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.774580+0000", 2024-04-19T01:05:11.877 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.15403776799999999 2024-04-19T01:05:11.877 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.878 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.878 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:11.878 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.775052+0000", 2024-04-19T01:05:11.878 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00047277300000000001 2024-04-19T01:05:11.878 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.878 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.878 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 5", 2024-04-19T01:05:11.878 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.775495+0000", 2024-04-19T01:05:11.878 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00044209400000000003 2024-04-19T01:05:11.879 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.879 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.879 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-04-19T01:05:11.879 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.777608+0000", 2024-04-19T01:05:11.879 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0021133649999999999 2024-04-19T01:05:11.879 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.880 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.880 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-04-19T01:05:11.880 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.777957+0000", 2024-04-19T01:05:11.880 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00034961399999999999 2024-04-19T01:05:11.880 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.880 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.880 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:11.880 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.778062+0000", 2024-04-19T01:05:11.880 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000104576 2024-04-19T01:05:11.881 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.881 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.881 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:11.881 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.778135+0000", 2024-04-19T01:05:11.881 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3257999999999996e-05 2024-04-19T01:05:11.881 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.881 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.881 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.881 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.882 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.882 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:360 3.8s0 3:11aca45f:::smithi15528762-45:head [append 0~573440] snapc 9=[7] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-04-19T01:05:11.882 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:19.532000+0000", 2024-04-19T01:05:11.882 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 52.336358633000003, 2024-04-19T01:05:11.882 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.042676382999999998, 2024-04-19T01:05:11.882 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:11.882 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:11.883 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:11.883 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:11.883 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:11.883 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:11.883 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 360 2024-04-19T01:05:11.883 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.883 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:11.883 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.883 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:11.883 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.532000+0000", 2024-04-19T01:05:11.883 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.884 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.884 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.884 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:11.884 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.532000+0000", 2024-04-19T01:05:11.884 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.884 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.884 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.884 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:11.884 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.532003+0000", 2024-04-19T01:05:11.884 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3189999999999999e-06 2024-04-19T01:05:11.884 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.884 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.884 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:11.884 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.533430+0000", 2024-04-19T01:05:11.885 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001427069 2024-04-19T01:05:11.885 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.885 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.885 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:11.885 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.533464+0000", 2024-04-19T01:05:11.885 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3708999999999999e-05 2024-04-19T01:05:11.885 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.885 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.885 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:11.885 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.533485+0000", 2024-04-19T01:05:11.885 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0596000000000001e-05 2024-04-19T01:05:11.885 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.885 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.885 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.885 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.535443+0000", 2024-04-19T01:05:11.886 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0019587290000000002 2024-04-19T01:05:11.886 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.886 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.886 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:11.886 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.535618+0000", 2024-04-19T01:05:11.886 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000174518 2024-04-19T01:05:11.886 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.886 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.886 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:11.886 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.537401+0000", 2024-04-19T01:05:11.886 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0017834350000000001 2024-04-19T01:05:11.886 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.886 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.886 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:11.887 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.550350+0000", 2024-04-19T01:05:11.887 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.012948602 2024-04-19T01:05:11.887 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.887 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.887 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:11.887 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.574521+0000", 2024-04-19T01:05:11.887 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.024171392999999999 2024-04-19T01:05:11.887 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.887 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.887 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:11.887 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.574676+0000", 2024-04-19T01:05:11.887 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015501299999999999 2024-04-19T01:05:11.887 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.888 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.888 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.888 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.888 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.888 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:361 3.8s0 3:11aca45f:::smithi15528762-45:head [append 0~425984] snapc 9=[7] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-04-19T01:05:11.888 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:19.540243+0000", 2024-04-19T01:05:11.888 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 52.328115265999998, 2024-04-19T01:05:11.888 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.038864792000000002, 2024-04-19T01:05:11.888 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:11.888 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:11.888 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:11.888 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:11.888 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:11.888 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:11.888 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 361 2024-04-19T01:05:11.889 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.889 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:11.889 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.889 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:11.889 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540243+0000", 2024-04-19T01:05:11.889 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.889 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.889 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.889 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:11.889 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540243+0000", 2024-04-19T01:05:11.889 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.889 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.889 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.890 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:11.890 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540247+0000", 2024-04-19T01:05:11.890 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.506e-06 2024-04-19T01:05:11.890 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.890 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.890 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:11.890 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.541191+0000", 2024-04-19T01:05:11.890 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00094451099999999998 2024-04-19T01:05:11.891 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.891 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.891 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:11.891 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.541227+0000", 2024-04-19T01:05:11.891 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6053000000000002e-05 2024-04-19T01:05:11.891 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.891 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.891 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:11.891 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.541250+0000", 2024-04-19T01:05:11.891 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2543000000000001e-05 2024-04-19T01:05:11.891 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.891 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.891 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.892 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.550582+0000", 2024-04-19T01:05:11.892 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0093325990000000004 2024-04-19T01:05:11.892 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.892 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.892 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:11.892 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.550838+0000", 2024-04-19T01:05:11.892 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00025546199999999997 2024-04-19T01:05:11.892 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.892 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.892 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:11.892 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.552905+0000", 2024-04-19T01:05:11.892 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0020670699999999998 2024-04-19T01:05:11.892 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.892 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.893 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:11.893 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.574896+0000", 2024-04-19T01:05:11.893 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.021991301000000001 2024-04-19T01:05:11.893 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.893 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.893 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:11.893 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.578996+0000", 2024-04-19T01:05:11.893 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0040997849999999999 2024-04-19T01:05:11.893 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.893 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.893 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:11.894 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.579108+0000", 2024-04-19T01:05:11.894 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000111962 2024-04-19T01:05:11.894 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.894 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.894 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.894 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.894 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.894 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:362 3.8s0 3:11aca45f:::smithi15528762-45:head [append 0~344064] snapc 9=[7] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-04-19T01:05:11.894 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:19.547167+0000", 2024-04-19T01:05:11.894 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 52.321191263000003, 2024-04-19T01:05:11.894 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.032345455000000002, 2024-04-19T01:05:11.894 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:11.894 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:11.895 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:11.895 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:11.895 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:11.895 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:11.895 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 362 2024-04-19T01:05:11.895 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.895 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:11.895 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.895 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:11.895 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547167+0000", 2024-04-19T01:05:11.895 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.895 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.895 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.895 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:11.896 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547167+0000", 2024-04-19T01:05:11.896 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.896 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.896 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.896 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:11.896 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547171+0000", 2024-04-19T01:05:11.896 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.303e-06 2024-04-19T01:05:11.896 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.896 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.896 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:11.896 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547655+0000", 2024-04-19T01:05:11.896 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00048394099999999999 2024-04-19T01:05:11.896 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.896 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.896 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:11.897 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547684+0000", 2024-04-19T01:05:11.897 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9828999999999999e-05 2024-04-19T01:05:11.897 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.897 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.897 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:11.897 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547699+0000", 2024-04-19T01:05:11.897 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4739999999999999e-05 2024-04-19T01:05:11.897 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.897 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.897 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.897 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.564911+0000", 2024-04-19T01:05:11.897 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.017211820999999999 2024-04-19T01:05:11.897 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.897 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.897 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:11.898 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.565200+0000", 2024-04-19T01:05:11.898 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00028915700000000001 2024-04-19T01:05:11.898 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.898 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.898 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:11.898 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.567550+0000", 2024-04-19T01:05:11.898 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0023502869999999999 2024-04-19T01:05:11.898 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.898 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.898 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:11.898 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.574910+0000", 2024-04-19T01:05:11.898 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0073593690000000002 2024-04-19T01:05:11.898 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.899 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.899 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:11.899 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.579412+0000", 2024-04-19T01:05:11.899 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0045022100000000004 2024-04-19T01:05:11.899 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.899 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.899 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:11.899 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.579513+0000", 2024-04-19T01:05:11.899 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010079800000000001 2024-04-19T01:05:11.899 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.899 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.899 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.899 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.900 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.900 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:363 3.8s0 3:11aca45f:::smithi15528762-45:head [setxattr _header (56) in=7b] snapc 9=[7] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-04-19T01:05:11.900 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:19.547729+0000", 2024-04-19T01:05:11.900 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 52.320629396999998, 2024-04-19T01:05:11.900 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.033477475, 2024-04-19T01:05:11.900 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:11.900 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:11.901 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:11.901 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:11.901 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:11.901 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:11.901 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 363 2024-04-19T01:05:11.901 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.901 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:11.901 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.901 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:11.901 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547729+0000", 2024-04-19T01:05:11.901 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.901 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.901 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.901 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:11.902 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547729+0000", 2024-04-19T01:05:11.902 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.902 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.902 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.902 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:11.902 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547730+0000", 2024-04-19T01:05:11.902 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2160000000000001e-06 2024-04-19T01:05:11.902 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.902 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.902 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:11.902 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547749+0000", 2024-04-19T01:05:11.902 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8312999999999999e-05 2024-04-19T01:05:11.903 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.903 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.903 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:11.903 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547764+0000", 2024-04-19T01:05:11.903 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5279999999999999e-05 2024-04-19T01:05:11.903 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.903 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.903 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:11.903 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547773+0000", 2024-04-19T01:05:11.903 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3300000000000005e-06 2024-04-19T01:05:11.903 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.903 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.903 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.903 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.572682+0000", 2024-04-19T01:05:11.904 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.024908825999999998 2024-04-19T01:05:11.904 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.904 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.904 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:11.904 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.572817+0000", 2024-04-19T01:05:11.904 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000135026 2024-04-19T01:05:11.904 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.904 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.904 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:11.904 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.573699+0000", 2024-04-19T01:05:11.905 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00088153199999999998 2024-04-19T01:05:11.905 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.905 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.905 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:11.905 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.575580+0000", 2024-04-19T01:05:11.905 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0018816589999999999 2024-04-19T01:05:11.905 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.905 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.905 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:11.905 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.581051+0000", 2024-04-19T01:05:11.905 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0054708220000000002 2024-04-19T01:05:11.905 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.905 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.905 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:11.906 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.581207+0000", 2024-04-19T01:05:11.906 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015547099999999999 2024-04-19T01:05:11.906 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.906 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.906 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.906 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.906 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.906 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:364 3.8s0 3:11aca45f:::smithi15528762-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-04-19T01:05:11.906 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:19.547786+0000", 2024-04-19T01:05:11.906 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 52.320572128999999, 2024-04-19T01:05:11.906 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.037412188999999998, 2024-04-19T01:05:11.906 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:11.906 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:11.906 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-04-19T01:05:11.907 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:11.907 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:11.907 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:11.907 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 364 2024-04-19T01:05:11.907 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.907 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:11.907 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.907 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:11.907 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547786+0000", 2024-04-19T01:05:11.907 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.907 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.907 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.907 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:11.907 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547786+0000", 2024-04-19T01:05:11.907 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.908 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.908 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.908 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:11.908 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547787+0000", 2024-04-19T01:05:11.908 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.1399999999999996e-07 2024-04-19T01:05:11.908 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.908 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.908 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:11.908 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547791+0000", 2024-04-19T01:05:11.908 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8299999999999998e-06 2024-04-19T01:05:11.908 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.908 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.908 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:11.908 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547802+0000", 2024-04-19T01:05:11.909 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1372999999999999e-05 2024-04-19T01:05:11.909 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.909 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.909 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:11.909 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.547811+0000", 2024-04-19T01:05:11.909 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.0469999999999999e-06 2024-04-19T01:05:11.909 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.909 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.909 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.909 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.575967+0000", 2024-04-19T01:05:11.909 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.028155099999999999 2024-04-19T01:05:11.909 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.909 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.909 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-04-19T01:05:11.910 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.576164+0000", 2024-04-19T01:05:11.910 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019718099999999999 2024-04-19T01:05:11.910 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.910 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.910 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.910 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.581873+0000", 2024-04-19T01:05:11.910 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0057093359999999998 2024-04-19T01:05:11.910 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.910 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.911 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:11.911 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.582089+0000", 2024-04-19T01:05:11.911 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021609099999999999 2024-04-19T01:05:11.911 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.911 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.911 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:11.911 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.585199+0000", 2024-04-19T01:05:11.911 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0031094170000000002 2024-04-19T01:05:11.911 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.911 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.911 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.912 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.912 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.912 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.5s3 74/31 ECSubWrite(tid=87, reqid=client.4604.0:543, at_version=74'67, trim_to=0'0, roll_forward_to=72'65))", 2024-04-19T01:05:11.912 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:44.553593+0000", 2024-04-19T01:05:11.912 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 27.314765456, 2024-04-19T01:05:11.912 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.044047740000000002, 2024-04-19T01:05:11.912 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:11.912 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:11.912 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:11.912 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:11.912 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.912 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:11.912 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.553593+0000", 2024-04-19T01:05:11.912 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.912 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.913 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.913 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:11.913 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.553593+0000", 2024-04-19T01:05:11.913 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.913 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.913 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.913 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:11.913 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.553594+0000", 2024-04-19T01:05:11.913 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.327e-06 2024-04-19T01:05:11.913 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.913 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.913 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:11.914 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.554148+0000", 2024-04-19T01:05:11.914 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00055343600000000003 2024-04-19T01:05:11.914 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.914 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.914 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:11.914 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.554184+0000", 2024-04-19T01:05:11.914 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5865999999999998e-05 2024-04-19T01:05:11.914 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.914 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.914 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:11.914 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.554211+0000", 2024-04-19T01:05:11.914 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7203999999999999e-05 2024-04-19T01:05:11.914 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.914 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.914 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.915 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.554385+0000", 2024-04-19T01:05:11.915 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000174387 2024-04-19T01:05:11.915 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.915 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.915 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:11.915 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.554418+0000", 2024-04-19T01:05:11.915 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2502999999999997e-05 2024-04-19T01:05:11.915 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.915 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.915 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:11.915 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.597567+0000", 2024-04-19T01:05:11.915 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.043149383999999999 2024-04-19T01:05:11.915 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.916 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.916 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:11.916 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.597641+0000", 2024-04-19T01:05:11.916 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3633000000000005e-05 2024-04-19T01:05:11.916 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.916 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.916 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.916 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.916 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.916 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.5s3 74/31 ECSubWrite(tid=88, reqid=osd.8.0:88, at_version=74'69, trim_to=0'0, roll_forward_to=72'65))", 2024-04-19T01:05:11.916 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:44.559673+0000", 2024-04-19T01:05:11.916 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 27.30868603, 2024-04-19T01:05:11.916 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.038043423, 2024-04-19T01:05:11.916 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:11.917 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:11.917 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:11.917 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:11.917 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.917 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:11.917 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.559673+0000", 2024-04-19T01:05:11.917 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.917 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.917 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.917 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:11.917 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.559673+0000", 2024-04-19T01:05:11.917 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.917 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.917 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.917 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:11.917 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.559674+0000", 2024-04-19T01:05:11.918 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.77e-06 2024-04-19T01:05:11.918 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.918 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.918 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:11.918 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.559727+0000", 2024-04-19T01:05:11.918 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2552000000000001e-05 2024-04-19T01:05:11.918 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.918 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.918 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:11.918 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.559756+0000", 2024-04-19T01:05:11.918 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9254000000000001e-05 2024-04-19T01:05:11.918 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.919 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.919 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:11.919 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.559780+0000", 2024-04-19T01:05:11.919 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3856e-05 2024-04-19T01:05:11.919 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.919 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.919 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.919 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.571632+0000", 2024-04-19T01:05:11.919 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.011851733 2024-04-19T01:05:11.919 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.919 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.919 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:11.919 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.571658+0000", 2024-04-19T01:05:11.920 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6568999999999999e-05 2024-04-19T01:05:11.920 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.920 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.920 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:11.920 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.597662+0000", 2024-04-19T01:05:11.920 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.02600421 2024-04-19T01:05:11.920 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.920 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.920 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:11.920 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.597716+0000", 2024-04-19T01:05:11.920 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3479000000000001e-05 2024-04-19T01:05:11.920 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.920 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.920 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.921 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.921 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.921 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.5s3 74/31 ECSubWrite(tid=89, reqid=client.4604.0:544, at_version=74'70, trim_to=0'0, roll_forward_to=72'65))", 2024-04-19T01:05:11.921 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:44.565551+0000", 2024-04-19T01:05:11.921 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 27.302807183999999, 2024-04-19T01:05:11.921 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.032242209000000001, 2024-04-19T01:05:11.921 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:11.921 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:11.922 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:11.922 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:11.922 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.922 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:11.922 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.565551+0000", 2024-04-19T01:05:11.922 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.922 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.922 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.922 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:11.922 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.565551+0000", 2024-04-19T01:05:11.922 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.922 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.922 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.922 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:11.923 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.565553+0000", 2024-04-19T01:05:11.923 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.136e-06 2024-04-19T01:05:11.923 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.923 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.923 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:11.923 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.566058+0000", 2024-04-19T01:05:11.923 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00050426699999999997 2024-04-19T01:05:11.923 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.923 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.923 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:11.923 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.566105+0000", 2024-04-19T01:05:11.924 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7636000000000003e-05 2024-04-19T01:05:11.924 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.924 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.924 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:11.924 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.566129+0000", 2024-04-19T01:05:11.924 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3433000000000001e-05 2024-04-19T01:05:11.924 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.924 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.924 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.924 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.575213+0000", 2024-04-19T01:05:11.924 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0090842939999999997 2024-04-19T01:05:11.924 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.924 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.924 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:11.925 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.575244+0000", 2024-04-19T01:05:11.925 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1350000000000003e-05 2024-04-19T01:05:11.925 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.925 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.925 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:11.925 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.597722+0000", 2024-04-19T01:05:11.925 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.022477924 2024-04-19T01:05:11.925 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.925 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.925 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:11.925 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.597794+0000", 2024-04-19T01:05:11.925 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.1168999999999996e-05 2024-04-19T01:05:11.925 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.925 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.926 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.926 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.926 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.926 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.5s3 79/31 ECSubWrite(tid=115, reqid=client.4604.0:591, at_version=79'90, trim_to=0'0, roll_forward_to=79'86))", 2024-04-19T01:05:11.926 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:49.660197+0000", 2024-04-19T01:05:11.926 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 22.208161650000001, 2024-04-19T01:05:11.926 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.037045658000000002, 2024-04-19T01:05:11.926 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:11.926 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:11.926 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:11.926 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:11.926 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.926 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:11.926 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.660197+0000", 2024-04-19T01:05:11.927 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.927 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.927 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.927 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:11.927 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.660197+0000", 2024-04-19T01:05:11.927 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.927 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.927 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.927 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:11.927 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.660198+0000", 2024-04-19T01:05:11.928 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4270000000000001e-06 2024-04-19T01:05:11.928 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.928 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.928 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:11.928 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.660609+0000", 2024-04-19T01:05:11.928 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00041111799999999998 2024-04-19T01:05:11.928 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.928 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.928 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:11.928 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.660635+0000", 2024-04-19T01:05:11.928 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5939e-05 2024-04-19T01:05:11.928 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.928 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.929 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:11.929 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.660649+0000", 2024-04-19T01:05:11.929 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3737000000000001e-05 2024-04-19T01:05:11.929 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.929 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.929 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.929 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.660788+0000", 2024-04-19T01:05:11.929 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013921199999999999 2024-04-19T01:05:11.929 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.929 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.929 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:11.929 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.660808+0000", 2024-04-19T01:05:11.929 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9202999999999999e-05 2024-04-19T01:05:11.929 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.930 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.930 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:11.930 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.697159+0000", 2024-04-19T01:05:11.930 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.036351151999999998 2024-04-19T01:05:11.930 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.930 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.930 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:11.930 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.697243+0000", 2024-04-19T01:05:11.930 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.3869999999999995e-05 2024-04-19T01:05:11.930 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.930 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.930 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.930 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.931 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.931 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.5s3 79/31 ECSubWrite(tid=116, reqid=client.4604.0:592, at_version=79'91, trim_to=0'0, roll_forward_to=79'88))", 2024-04-19T01:05:11.931 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:49.665284+0000", 2024-04-19T01:05:11.931 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 22.203074927999999, 2024-04-19T01:05:11.931 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.032029100999999997, 2024-04-19T01:05:11.931 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:11.931 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:11.931 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:11.931 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:11.932 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.932 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:11.932 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.665284+0000", 2024-04-19T01:05:11.932 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.932 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.932 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.932 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:11.932 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.665284+0000", 2024-04-19T01:05:11.932 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.932 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.932 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.932 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:11.932 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.665285+0000", 2024-04-19T01:05:11.932 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.843e-06 2024-04-19T01:05:11.932 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.933 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.933 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:11.933 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.665682+0000", 2024-04-19T01:05:11.933 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00039639900000000002 2024-04-19T01:05:11.933 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.933 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.933 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:11.933 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.665717+0000", 2024-04-19T01:05:11.933 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5342999999999997e-05 2024-04-19T01:05:11.933 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.933 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.933 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:11.933 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.665740+0000", 2024-04-19T01:05:11.933 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2731999999999999e-05 2024-04-19T01:05:11.933 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.934 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.934 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.934 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.675712+0000", 2024-04-19T01:05:11.934 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0099725370000000001 2024-04-19T01:05:11.934 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.934 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.934 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:11.934 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.678072+0000", 2024-04-19T01:05:11.934 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0023595130000000001 2024-04-19T01:05:11.940 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.940 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.940 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:11.940 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.697256+0000", 2024-04-19T01:05:11.940 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.019184063000000001 2024-04-19T01:05:11.940 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.940 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.940 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:11.941 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:49.697313+0000", 2024-04-19T01:05:11.941 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6671e-05 2024-04-19T01:05:11.941 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.941 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.941 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.941 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.941 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.941 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.5s3 81/31 ECSubWrite(tid=122, reqid=client.4604.0:626, at_version=81'96, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:05:11.941 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:51.593541+0000", 2024-04-19T01:05:11.941 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 20.274817670000001, 2024-04-19T01:05:11.941 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.061054634000000003, 2024-04-19T01:05:11.941 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:11.941 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:11.942 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:11.942 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:11.942 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.942 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:11.942 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.593541+0000", 2024-04-19T01:05:11.942 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.942 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.942 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.942 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:11.942 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.593541+0000", 2024-04-19T01:05:11.942 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.942 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.943 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.943 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:11.943 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.593541+0000", 2024-04-19T01:05:11.943 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9699999999999996e-07 2024-04-19T01:05:11.943 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.943 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.943 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:11.943 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.594429+0000", 2024-04-19T01:05:11.944 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000887318 2024-04-19T01:05:11.944 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.944 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.944 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:11.944 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.594456+0000", 2024-04-19T01:05:11.944 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7509e-05 2024-04-19T01:05:11.944 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.945 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.945 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:11.945 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.594472+0000", 2024-04-19T01:05:11.945 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6107e-05 2024-04-19T01:05:11.945 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.945 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.945 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.945 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.594609+0000", 2024-04-19T01:05:11.945 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013671899999999999 2024-04-19T01:05:11.945 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.945 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.945 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:11.946 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.594628+0000", 2024-04-19T01:05:11.946 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9123e-05 2024-04-19T01:05:11.946 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.946 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.946 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:11.946 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.654485+0000", 2024-04-19T01:05:11.946 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.059857150999999997 2024-04-19T01:05:11.946 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.946 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.946 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:11.946 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.654596+0000", 2024-04-19T01:05:11.946 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011021 2024-04-19T01:05:11.946 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.946 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.947 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.947 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.947 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.947 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.5s3 81/31 ECSubWrite(tid=123, reqid=client.4604.0:627, at_version=81'97, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:05:11.947 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:51.600033+0000", 2024-04-19T01:05:11.947 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 20.268325136000001, 2024-04-19T01:05:11.947 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.054639177999999997, 2024-04-19T01:05:11.947 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:11.947 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:11.947 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:11.947 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:11.947 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.947 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:11.947 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.600033+0000", 2024-04-19T01:05:11.948 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.948 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.948 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.948 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:11.949 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.600033+0000", 2024-04-19T01:05:11.949 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.949 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.949 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.949 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:11.949 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.600036+0000", 2024-04-19T01:05:11.949 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.419e-06 2024-04-19T01:05:11.949 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.949 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.949 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:11.950 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.600416+0000", 2024-04-19T01:05:11.950 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00038017299999999998 2024-04-19T01:05:11.950 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.950 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.951 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:11.951 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.600457+0000", 2024-04-19T01:05:11.951 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1075999999999998e-05 2024-04-19T01:05:11.951 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.951 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.951 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:11.951 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.600480+0000", 2024-04-19T01:05:11.951 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2976000000000002e-05 2024-04-19T01:05:11.951 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.951 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.951 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.951 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.609097+0000", 2024-04-19T01:05:11.951 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.008617408 2024-04-19T01:05:11.952 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.952 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.952 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:11.952 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.609118+0000", 2024-04-19T01:05:11.952 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0136000000000002e-05 2024-04-19T01:05:11.952 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.952 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.952 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:11.952 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.654605+0000", 2024-04-19T01:05:11.952 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.045486985000000001 2024-04-19T01:05:11.952 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.953 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.953 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:11.953 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.654673+0000", 2024-04-19T01:05:11.953 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.8004999999999997e-05 2024-04-19T01:05:11.953 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.953 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.953 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.953 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.953 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.953 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.5s3 81/31 ECSubWrite(tid=124, reqid=client.4604.0:628, at_version=81'98, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:05:11.954 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:51.605750+0000", 2024-04-19T01:05:11.954 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 20.262608102000002, 2024-04-19T01:05:11.954 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.048993015000000001, 2024-04-19T01:05:11.954 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:11.954 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:11.954 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:11.954 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:11.954 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.954 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:11.954 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.605750+0000", 2024-04-19T01:05:11.955 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.955 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.955 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.955 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:11.955 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.605750+0000", 2024-04-19T01:05:11.955 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.955 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.955 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.955 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:11.955 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.605752+0000", 2024-04-19T01:05:11.955 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9870000000000002e-06 2024-04-19T01:05:11.955 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.955 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.955 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:11.955 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.606125+0000", 2024-04-19T01:05:11.956 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00037219999999999999 2024-04-19T01:05:11.956 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.956 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.956 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:11.956 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.606160+0000", 2024-04-19T01:05:11.956 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5534999999999998e-05 2024-04-19T01:05:11.956 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.956 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.956 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:11.956 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.606182+0000", 2024-04-19T01:05:11.956 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1999000000000001e-05 2024-04-19T01:05:11.956 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.956 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.956 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.956 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.618898+0000", 2024-04-19T01:05:11.957 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.012715888 2024-04-19T01:05:11.957 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.957 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.957 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:11.957 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.618930+0000", 2024-04-19T01:05:11.957 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1712999999999999e-05 2024-04-19T01:05:11.957 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.957 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.957 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:11.957 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.654683+0000", 2024-04-19T01:05:11.958 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.035753339000000002 2024-04-19T01:05:11.958 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.958 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.958 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:11.958 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.654743+0000", 2024-04-19T01:05:11.958 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0353999999999998e-05 2024-04-19T01:05:11.958 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.958 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.958 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.958 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.959 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.959 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.5s3 81/31 ECSubWrite(tid=125, reqid=client.4604.0:629, at_version=81'99, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:05:11.959 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:51.609931+0000", 2024-04-19T01:05:11.959 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 20.258427179999998, 2024-04-19T01:05:11.959 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.044873181999999998, 2024-04-19T01:05:11.960 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:11.960 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:11.960 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:11.960 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:11.960 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.960 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:11.961 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.609931+0000", 2024-04-19T01:05:11.961 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.961 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.961 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.961 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:11.961 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.609931+0000", 2024-04-19T01:05:11.961 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.962 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.962 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.962 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:11.962 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.609933+0000", 2024-04-19T01:05:11.962 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.753e-06 2024-04-19T01:05:11.962 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.962 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.962 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:11.962 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.609982+0000", 2024-04-19T01:05:11.962 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9330999999999999e-05 2024-04-19T01:05:11.962 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.962 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.962 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:11.962 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.610031+0000", 2024-04-19T01:05:11.963 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8718999999999999e-05 2024-04-19T01:05:11.963 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.963 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.963 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:11.963 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.610048+0000", 2024-04-19T01:05:11.963 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6923e-05 2024-04-19T01:05:11.963 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.963 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.963 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.963 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.629621+0000", 2024-04-19T01:05:11.963 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.019572523000000001 2024-04-19T01:05:11.963 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.963 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.964 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:11.964 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.629650+0000", 2024-04-19T01:05:11.964 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9104000000000001e-05 2024-04-19T01:05:11.964 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.964 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.964 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:11.964 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.654749+0000", 2024-04-19T01:05:11.964 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.025098874 2024-04-19T01:05:11.964 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.964 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.964 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:11.964 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.654805+0000", 2024-04-19T01:05:11.964 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.5955000000000002e-05 2024-04-19T01:05:11.964 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.965 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.965 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.965 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.965 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.965 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.5s3 81/31 ECSubWrite(tid=126, reqid=client.4604.0:631, at_version=81'101, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:05:11.965 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:51.618408+0000", 2024-04-19T01:05:11.965 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 20.249950672000001, 2024-04-19T01:05:11.965 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.038972772000000003, 2024-04-19T01:05:11.965 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:11.965 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:11.965 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:11.965 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:11.965 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.966 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:11.966 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.618408+0000", 2024-04-19T01:05:11.966 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.966 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.966 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.966 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:11.966 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.618408+0000", 2024-04-19T01:05:11.966 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.966 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.966 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.966 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:11.966 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.618410+0000", 2024-04-19T01:05:11.966 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.863e-06 2024-04-19T01:05:11.966 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.967 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.967 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:11.967 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.619228+0000", 2024-04-19T01:05:11.967 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00081862099999999995 2024-04-19T01:05:11.967 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.967 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.967 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:11.967 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.619258+0000", 2024-04-19T01:05:11.967 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9805e-05 2024-04-19T01:05:11.967 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.967 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.967 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:11.968 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.619280+0000", 2024-04-19T01:05:11.968 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.158e-05 2024-04-19T01:05:11.968 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.968 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.968 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.968 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.630187+0000", 2024-04-19T01:05:11.968 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.010906911 2024-04-19T01:05:11.968 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.968 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.968 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:11.968 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.630202+0000", 2024-04-19T01:05:11.968 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.56e-05 2024-04-19T01:05:11.968 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.968 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.969 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:11.969 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.657322+0000", 2024-04-19T01:05:11.969 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.027119353999999998 2024-04-19T01:05:11.969 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.969 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.969 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:11.969 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.657381+0000", 2024-04-19T01:05:11.969 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9038e-05 2024-04-19T01:05:11.969 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.969 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.969 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.969 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.969 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.970 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.es1 86/31 ECSubWrite(tid=195, reqid=client.4604.0:653, at_version=86'39, trim_to=0'0, roll_forward_to=84'38))", 2024-04-19T01:05:11.970 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:56.572357+0000", 2024-04-19T01:05:11.970 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 15.296001415999999, 2024-04-19T01:05:11.970 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.035630057999999999, 2024-04-19T01:05:11.970 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:11.970 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:11.970 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:11.971 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:11.971 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.971 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:11.972 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.572357+0000", 2024-04-19T01:05:11.972 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.972 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.972 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.972 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:11.972 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.572357+0000", 2024-04-19T01:05:11.972 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.972 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.972 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.972 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:11.972 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.572357+0000", 2024-04-19T01:05:11.973 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6399999999999998e-07 2024-04-19T01:05:11.973 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.973 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.973 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:11.973 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.573559+0000", 2024-04-19T01:05:11.973 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001201199 2024-04-19T01:05:11.973 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.973 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.973 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:11.973 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.573596+0000", 2024-04-19T01:05:11.973 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7441999999999998e-05 2024-04-19T01:05:11.974 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.974 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.974 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:11.974 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.573617+0000", 2024-04-19T01:05:11.974 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0908e-05 2024-04-19T01:05:11.974 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.974 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.974 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.974 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.573767+0000", 2024-04-19T01:05:11.974 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015031199999999999 2024-04-19T01:05:11.974 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.975 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.975 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:11.975 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.573785+0000", 2024-04-19T01:05:11.975 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7788e-05 2024-04-19T01:05:11.975 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.975 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.975 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:11.975 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.607908+0000", 2024-04-19T01:05:11.975 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.034122616000000001 2024-04-19T01:05:11.975 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.976 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.976 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:11.976 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.607987+0000", 2024-04-19T01:05:11.976 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.9429000000000001e-05 2024-04-19T01:05:11.976 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.976 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.976 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.976 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.976 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.976 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:655 3.2s0 3:47ba51fe:::smithi15528762-10:head [append 0~770048] snapc 1d=[1d,1c,1b,14] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-04-19T01:05:11.976 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:56.576580+0000", 2024-04-19T01:05:11.976 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 15.291778825, 2024-04-19T01:05:11.976 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.045487739999999999, 2024-04-19T01:05:11.976 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:11.977 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:11.977 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:11.977 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:11.977 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:11.977 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:11.977 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 655 2024-04-19T01:05:11.978 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.978 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:11.978 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.978 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:11.978 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.576580+0000", 2024-04-19T01:05:11.978 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.978 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.978 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.979 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:11.979 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.576580+0000", 2024-04-19T01:05:11.979 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.979 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.979 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.979 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:11.979 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.576583+0000", 2024-04-19T01:05:11.979 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.055e-06 2024-04-19T01:05:11.979 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.979 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.979 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:11.979 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.577087+0000", 2024-04-19T01:05:11.979 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00050413300000000003 2024-04-19T01:05:11.980 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.980 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.980 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:11.980 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.577115+0000", 2024-04-19T01:05:11.980 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7668e-05 2024-04-19T01:05:11.980 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.980 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.980 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:11.980 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.577128+0000", 2024-04-19T01:05:11.980 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3454e-05 2024-04-19T01:05:11.981 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.981 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.981 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.981 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.577281+0000", 2024-04-19T01:05:11.982 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000152583 2024-04-19T01:05:11.982 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.982 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.982 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:11.982 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.577572+0000", 2024-04-19T01:05:11.982 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00029132099999999998 2024-04-19T01:05:11.982 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.982 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.982 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:11.982 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.580336+0000", 2024-04-19T01:05:11.982 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0027644039999999998 2024-04-19T01:05:11.982 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.983 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.983 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:11.983 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.614100+0000", 2024-04-19T01:05:11.983 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.033763537000000003 2024-04-19T01:05:11.983 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.983 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.983 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:11.983 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.621899+0000", 2024-04-19T01:05:11.983 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0077987109999999998 2024-04-19T01:05:11.984 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.984 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.984 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:11.984 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.622067+0000", 2024-04-19T01:05:11.984 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016887400000000001 2024-04-19T01:05:11.984 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.984 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.984 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.984 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.984 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.984 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:656 3.2s0 3:47ba51fe:::smithi15528762-10:head [append 0~704512] snapc 1d=[1d,1c,1b,14] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-04-19T01:05:11.984 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:56.588400+0000", 2024-04-19T01:05:11.984 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 15.279958943, 2024-04-19T01:05:11.984 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.035455562000000003, 2024-04-19T01:05:11.985 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:11.985 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:11.985 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:11.985 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:11.985 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:11.985 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:11.985 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 656 2024-04-19T01:05:11.985 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.985 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:11.985 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.985 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:11.985 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.588400+0000", 2024-04-19T01:05:11.985 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.985 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.985 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.986 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:11.986 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.588400+0000", 2024-04-19T01:05:11.986 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.986 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.986 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.986 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:11.986 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.588403+0000", 2024-04-19T01:05:11.986 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3799999999999998e-06 2024-04-19T01:05:11.986 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.986 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.986 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:11.986 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.589339+0000", 2024-04-19T01:05:11.986 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00093576 2024-04-19T01:05:11.987 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.987 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.987 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:11.987 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.589373+0000", 2024-04-19T01:05:11.987 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3807999999999999e-05 2024-04-19T01:05:11.987 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.987 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.987 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:11.987 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.589391+0000", 2024-04-19T01:05:11.987 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8632999999999999e-05 2024-04-19T01:05:11.987 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.987 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.987 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.987 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.595426+0000", 2024-04-19T01:05:11.987 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0060349000000000002 2024-04-19T01:05:11.988 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.988 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.988 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:11.988 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.595673+0000", 2024-04-19T01:05:11.988 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00024695899999999999 2024-04-19T01:05:11.988 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.988 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.988 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:11.988 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.597722+0000", 2024-04-19T01:05:11.988 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.002049132 2024-04-19T01:05:11.988 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.988 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.988 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:11.988 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.620441+0000", 2024-04-19T01:05:11.988 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.022719143000000001 2024-04-19T01:05:11.989 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.989 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.989 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:11.989 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.623711+0000", 2024-04-19T01:05:11.989 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0032692530000000002 2024-04-19T01:05:11.989 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.989 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.989 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:11.989 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.623855+0000", 2024-04-19T01:05:11.989 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014459400000000001 2024-04-19T01:05:11.989 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.989 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.990 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.990 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.990 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.990 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:659 3.2s0 3:47ba51fe:::smithi15528762-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e86)", 2024-04-19T01:05:11.990 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:56.596815+0000", 2024-04-19T01:05:11.990 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 15.271543639000001, 2024-04-19T01:05:11.990 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.033534798999999997, 2024-04-19T01:05:11.990 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:11.990 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:11.990 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-04-19T01:05:11.990 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:11.990 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:11.990 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:11.990 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 659 2024-04-19T01:05:11.991 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.991 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:11.991 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.991 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:11.991 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.596815+0000", 2024-04-19T01:05:11.991 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.992 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.992 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.992 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:11.992 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.596815+0000", 2024-04-19T01:05:11.992 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:11.992 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.992 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.992 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:11.992 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.596816+0000", 2024-04-19T01:05:11.992 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1230000000000001e-06 2024-04-19T01:05:11.992 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.992 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.992 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:11.993 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.596821+0000", 2024-04-19T01:05:11.993 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2800000000000003e-06 2024-04-19T01:05:11.993 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.993 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.993 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:11.993 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.596839+0000", 2024-04-19T01:05:11.993 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7774999999999998e-05 2024-04-19T01:05:11.993 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.993 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.993 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:11.993 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.596852+0000", 2024-04-19T01:05:11.993 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2422000000000001e-05 2024-04-19T01:05:11.993 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.993 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.993 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.994 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.622321+0000", 2024-04-19T01:05:11.994 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.025469985000000001 2024-04-19T01:05:11.994 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.994 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.994 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-04-19T01:05:11.994 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.622469+0000", 2024-04-19T01:05:11.994 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014785300000000001 2024-04-19T01:05:11.994 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.994 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.994 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:11.994 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.628402+0000", 2024-04-19T01:05:11.994 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0059330559999999999 2024-04-19T01:05:11.994 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.995 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.995 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:11.995 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.628581+0000", 2024-04-19T01:05:11.995 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017904100000000001 2024-04-19T01:05:11.995 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:11.995 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:11.995 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:11.995 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.630350+0000", 2024-04-19T01:05:11.995 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0017682640000000001 2024-04-19T01:05:11.995 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.995 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.995 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.995 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:11.995 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:11.995 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:05:11.997 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-04-19T01:05:11.998 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:05:11.993+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:05:12.098 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:05:12.097+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:05:12.124 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:05:12.124 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:05:12.124 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-04-19T01:05:12.124 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:05:12.139 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-04-19T01:05:12.199 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:05:12.198+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:05:12.260 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:05:12.261 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:05:12.261 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-04-19T01:05:12.261 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-04-19T01:05:12.261 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:05:12.275 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-04-19T01:05:12.299 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:05:12.298+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:05:12.401 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:05:12.401 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-04-19T01:05:12.401 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-04-19T01:05:12.401 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-04-19T01:05:12.401 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.401 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:173 3.3s0 3:c34529b5:::smithi15528762-31:head [append 0~507904] snapc 5=[5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-19T01:05:12.402 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:08.816172+0000", 2024-04-19T01:05:12.402 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 63.582742437, 2024-04-19T01:05:12.402 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.024750307999999999, 2024-04-19T01:05:12.402 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:12.402 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:12.402 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:12.402 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:12.402 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:12.402 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:12.402 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 173 2024-04-19T01:05:12.403 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.403 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:12.403 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.403 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:12.403 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816172+0000", 2024-04-19T01:05:12.403 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:12.403 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.403 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.403 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:12.403 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816172+0000", 2024-04-19T01:05:12.404 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:12.404 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.404 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.404 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:12.404 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816175+0000", 2024-04-19T01:05:12.405 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9589999999999997e-06 2024-04-19T01:05:12.405 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.405 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.405 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:12.405 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816766+0000", 2024-04-19T01:05:12.405 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00059020999999999998 2024-04-19T01:05:12.405 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.405 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.405 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:12.406 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816802+0000", 2024-04-19T01:05:12.406 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6140999999999998e-05 2024-04-19T01:05:12.406 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.406 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.406 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:12.406 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816820+0000", 2024-04-19T01:05:12.406 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7771999999999999e-05 2024-04-19T01:05:12.406 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.407 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.407 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:12.407 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816990+0000", 2024-04-19T01:05:12.407 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016993299999999999 2024-04-19T01:05:12.407 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.407 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.407 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op must wait for map", 2024-04-19T01:05:12.407 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.817016+0000", 2024-04-19T01:05:12.407 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6857000000000001e-05 2024-04-19T01:05:12.407 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.408 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.408 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:12.408 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.821660+0000", 2024-04-19T01:05:12.408 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0046440520000000001 2024-04-19T01:05:12.408 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.408 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.408 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:12.408 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.821835+0000", 2024-04-19T01:05:12.408 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000174112 2024-04-19T01:05:12.409 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.409 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.409 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:12.409 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.824020+0000", 2024-04-19T01:05:12.409 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.002185523 2024-04-19T01:05:12.409 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.409 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.409 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:12.409 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.839423+0000", 2024-04-19T01:05:12.410 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.015403261999999999 2024-04-19T01:05:12.410 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.410 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.410 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:12.410 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.840758+0000", 2024-04-19T01:05:12.410 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001335145 2024-04-19T01:05:12.411 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.411 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.411 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:12.411 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.840922+0000", 2024-04-19T01:05:12.411 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000163342 2024-04-19T01:05:12.411 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:12.411 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:12.411 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:12.411 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.411 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.413 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:174 3.3s0 3:c34529b5:::smithi15528762-31:head [append 0~376832] snapc 5=[5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-19T01:05:12.413 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:08.821669+0000", 2024-04-19T01:05:12.414 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 63.577245189000003, 2024-04-19T01:05:12.414 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.019779560000000002, 2024-04-19T01:05:12.414 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:12.414 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:12.414 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:12.414 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:12.414 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:12.414 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:12.414 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 174 2024-04-19T01:05:12.415 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.415 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:12.415 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.415 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:12.415 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.821669+0000", 2024-04-19T01:05:12.415 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:12.415 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.415 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.416 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:12.416 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.821669+0000", 2024-04-19T01:05:12.416 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:12.416 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.416 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.416 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:12.417 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.821671+0000", 2024-04-19T01:05:12.417 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2539999999999999e-06 2024-04-19T01:05:12.417 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.417 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.417 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:12.417 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822078+0000", 2024-04-19T01:05:12.417 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00040720600000000002 2024-04-19T01:05:12.417 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.417 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.417 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:12.418 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822097+0000", 2024-04-19T01:05:12.418 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9011000000000001e-05 2024-04-19T01:05:12.418 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.418 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.418 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:12.418 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822111+0000", 2024-04-19T01:05:12.418 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4188e-05 2024-04-19T01:05:12.418 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.418 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.418 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:12.419 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.830924+0000", 2024-04-19T01:05:12.419 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0088129609999999994 2024-04-19T01:05:12.419 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.419 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.419 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:12.420 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.831084+0000", 2024-04-19T01:05:12.420 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000159889 2024-04-19T01:05:12.421 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.422 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.422 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:12.422 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.832503+0000", 2024-04-19T01:05:12.422 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001418655 2024-04-19T01:05:12.422 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.422 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.423 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:12.423 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.841099+0000", 2024-04-19T01:05:12.423 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0085957340000000007 2024-04-19T01:05:12.423 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.423 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.423 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:12.423 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.841345+0000", 2024-04-19T01:05:12.423 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00024665800000000002 2024-04-19T01:05:12.423 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.423 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.423 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:12.423 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.841448+0000", 2024-04-19T01:05:12.423 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010300400000000001 2024-04-19T01:05:12.423 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:12.424 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:12.424 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:12.424 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.424 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.424 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:175 3.3s0 3:c34529b5:::smithi15528762-31:head [setxattr _header (56) in=7b] snapc 5=[5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-19T01:05:12.424 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:08.822142+0000", 2024-04-19T01:05:12.424 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 63.576772189000003, 2024-04-19T01:05:12.424 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.020224723999999999, 2024-04-19T01:05:12.424 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:12.424 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:12.424 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:12.424 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:12.424 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:12.425 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:12.425 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 175 2024-04-19T01:05:12.425 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.425 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:12.425 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.425 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:12.425 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822142+0000", 2024-04-19T01:05:12.425 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:12.425 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.425 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.426 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:12.426 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822142+0000", 2024-04-19T01:05:12.426 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:12.426 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.426 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.426 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:12.426 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822143+0000", 2024-04-19T01:05:12.426 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2699999999999999e-06 2024-04-19T01:05:12.427 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.427 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.427 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:12.427 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822151+0000", 2024-04-19T01:05:12.427 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4760000000000001e-06 2024-04-19T01:05:12.427 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.427 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.427 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:12.427 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822165+0000", 2024-04-19T01:05:12.427 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4138000000000001e-05 2024-04-19T01:05:12.427 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.427 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.427 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:12.427 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822178+0000", 2024-04-19T01:05:12.428 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3185e-05 2024-04-19T01:05:12.428 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.428 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.428 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:12.428 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.839463+0000", 2024-04-19T01:05:12.429 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.017285038999999999 2024-04-19T01:05:12.429 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.429 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.429 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:12.429 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.839580+0000", 2024-04-19T01:05:12.429 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000117601 2024-04-19T01:05:12.429 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.429 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.429 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:12.430 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.840161+0000", 2024-04-19T01:05:12.430 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00058043199999999997 2024-04-19T01:05:12.430 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.430 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.430 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:12.430 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.841469+0000", 2024-04-19T01:05:12.430 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001308065 2024-04-19T01:05:12.430 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.430 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.430 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:12.430 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.842264+0000", 2024-04-19T01:05:12.431 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00079508400000000005 2024-04-19T01:05:12.431 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.431 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.431 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:12.431 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.842366+0000", 2024-04-19T01:05:12.431 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000102434 2024-04-19T01:05:12.431 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:12.431 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:12.431 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:12.431 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.431 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.432 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:176 3.3s0 3:c34529b5:::smithi15528762-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-19T01:05:12.432 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:08.822193+0000", 2024-04-19T01:05:12.432 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 63.576720973999997, 2024-04-19T01:05:12.432 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.023273202, 2024-04-19T01:05:12.432 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:12.432 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:12.432 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-04-19T01:05:12.432 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:12.432 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:12.432 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:12.432 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 176 2024-04-19T01:05:12.432 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.433 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:12.433 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.433 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:12.433 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822193+0000", 2024-04-19T01:05:12.433 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:12.433 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.433 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.433 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:12.433 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822193+0000", 2024-04-19T01:05:12.433 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:12.433 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.433 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.433 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:12.433 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822194+0000", 2024-04-19T01:05:12.434 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4200000000000004e-07 2024-04-19T01:05:12.434 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.434 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.434 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:12.434 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822198+0000", 2024-04-19T01:05:12.434 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.206e-06 2024-04-19T01:05:12.434 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.434 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.434 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:12.434 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822210+0000", 2024-04-19T01:05:12.434 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.217e-05 2024-04-19T01:05:12.434 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.434 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.434 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:12.435 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822220+0000", 2024-04-19T01:05:12.435 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0059e-05 2024-04-19T01:05:12.435 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.435 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.435 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:12.435 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.842930+0000", 2024-04-19T01:05:12.435 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.020709701 2024-04-19T01:05:12.435 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.435 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.435 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:12.435 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.843170+0000", 2024-04-19T01:05:12.435 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00024026700000000001 2024-04-19T01:05:12.435 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.436 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.436 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:12.436 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.845466+0000", 2024-04-19T01:05:12.436 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0022958570000000001 2024-04-19T01:05:12.436 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:12.436 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:12.436 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:12.436 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.436 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.436 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:343 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [copy-from ver 14] snapc 9=[8,7] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:05:12.436 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.511298+0000", 2024-04-19T01:05:12.436 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 53.887615975999999, 2024-04-19T01:05:12.436 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.043267551000000001, 2024-04-19T01:05:12.436 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:12.437 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:12.437 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:12.437 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:12.437 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:12.437 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:12.437 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 343 2024-04-19T01:05:12.437 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.437 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:12.437 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.437 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:12.437 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511298+0000", 2024-04-19T01:05:12.437 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:12.438 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.438 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.438 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:12.438 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511298+0000", 2024-04-19T01:05:12.438 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:12.438 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.438 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.438 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:12.438 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511302+0000", 2024-04-19T01:05:12.438 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1030000000000004e-06 2024-04-19T01:05:12.438 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.438 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.438 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:12.438 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511319+0000", 2024-04-19T01:05:12.439 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7070000000000001e-05 2024-04-19T01:05:12.439 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.439 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.439 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:12.439 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511357+0000", 2024-04-19T01:05:12.439 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8056000000000003e-05 2024-04-19T01:05:12.439 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.440 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.440 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:12.440 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511390+0000", 2024-04-19T01:05:12.440 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3213000000000002e-05 2024-04-19T01:05:12.440 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.440 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.440 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:12.440 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.516539+0000", 2024-04-19T01:05:12.440 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0051490729999999997 2024-04-19T01:05:12.440 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.440 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.440 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:12.440 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.516748+0000", 2024-04-19T01:05:12.441 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000208099 2024-04-19T01:05:12.441 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.441 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.441 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:12.441 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.541650+0000", 2024-04-19T01:05:12.441 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.024902837000000001 2024-04-19T01:05:12.441 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.441 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.441 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:12.441 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.551197+0000", 2024-04-19T01:05:12.441 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0095463240000000001 2024-04-19T01:05:12.441 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.441 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.442 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:12.442 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.554383+0000", 2024-04-19T01:05:12.442 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0031863830000000001 2024-04-19T01:05:12.442 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.442 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.442 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:12.442 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.554566+0000", 2024-04-19T01:05:12.442 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00018239299999999999 2024-04-19T01:05:12.442 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:12.442 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:12.442 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:12.442 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.442 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.442 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:344 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:05:12.443 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.511452+0000", 2024-04-19T01:05:12.443 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 53.887461725999998, 2024-04-19T01:05:12.443 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.038223957000000003, 2024-04-19T01:05:12.443 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:12.443 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:12.443 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-04-19T01:05:12.443 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:12.443 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:12.443 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:12.443 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 344 2024-04-19T01:05:12.443 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.443 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:12.444 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.444 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:12.444 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511452+0000", 2024-04-19T01:05:12.444 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:12.444 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.444 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.444 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:12.444 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511452+0000", 2024-04-19T01:05:12.444 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:12.444 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.444 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.444 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:12.444 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511454+0000", 2024-04-19T01:05:12.444 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.39e-06 2024-04-19T01:05:12.445 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.445 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.445 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:12.445 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511460+0000", 2024-04-19T01:05:12.445 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.46e-06 2024-04-19T01:05:12.445 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.445 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.445 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:12.445 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511476+0000", 2024-04-19T01:05:12.445 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6180000000000001e-05 2024-04-19T01:05:12.445 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.445 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.445 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:12.446 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511487+0000", 2024-04-19T01:05:12.446 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.049e-05 2024-04-19T01:05:12.446 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.446 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.446 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:12.446 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.517203+0000", 2024-04-19T01:05:12.446 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00571595 2024-04-19T01:05:12.446 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.446 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.446 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for blocked object", 2024-04-19T01:05:12.446 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.517341+0000", 2024-04-19T01:05:12.446 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013835099999999999 2024-04-19T01:05:12.447 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.447 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.447 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:12.447 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.549188+0000", 2024-04-19T01:05:12.447 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.031846589000000002 2024-04-19T01:05:12.447 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.447 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.447 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:12.447 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.549399+0000", 2024-04-19T01:05:12.447 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021149600000000001 2024-04-19T01:05:12.447 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.447 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.447 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:12.448 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.549676+0000", 2024-04-19T01:05:12.448 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000277051 2024-04-19T01:05:12.448 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:12.448 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:12.448 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:12.448 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.448 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.448 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.8s3 49/31 ECSubWrite(tid=203, reqid=client.4604.0:360, at_version=49'25, trim_to=0'0, roll_forward_to=49'24))", 2024-04-19T01:05:12.448 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:19.539927+0000", 2024-04-19T01:05:12.448 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 52.858986543, 2024-04-19T01:05:12.448 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.027280209, 2024-04-19T01:05:12.448 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:12.448 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:12.448 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:12.449 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:12.449 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.449 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:12.449 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.539927+0000", 2024-04-19T01:05:12.449 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:12.449 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.449 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.449 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:12.449 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.539927+0000", 2024-04-19T01:05:12.449 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:12.450 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.450 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.450 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:12.450 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.539929+0000", 2024-04-19T01:05:12.450 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3829999999999999e-06 2024-04-19T01:05:12.450 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.450 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.450 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:12.450 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540190+0000", 2024-04-19T01:05:12.450 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00026120499999999998 2024-04-19T01:05:12.450 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.451 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.451 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:12.451 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540216+0000", 2024-04-19T01:05:12.451 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5573000000000001e-05 2024-04-19T01:05:12.451 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.451 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.451 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:12.451 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540242+0000", 2024-04-19T01:05:12.451 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5939999999999999e-05 2024-04-19T01:05:12.451 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.451 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.451 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:12.451 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540387+0000", 2024-04-19T01:05:12.451 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000145983 2024-04-19T01:05:12.451 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.452 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.452 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:12.452 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540412+0000", 2024-04-19T01:05:12.452 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.442e-05 2024-04-19T01:05:12.452 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.452 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.452 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:12.452 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.567114+0000", 2024-04-19T01:05:12.452 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.026701643000000001 2024-04-19T01:05:12.452 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.452 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.452 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:12.452 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.567208+0000", 2024-04-19T01:05:12.453 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4061999999999995e-05 2024-04-19T01:05:12.453 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:12.453 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:12.453 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:12.453 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.453 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.453 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:419 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [read 0~262144,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e59)", 2024-04-19T01:05:12.453 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:29.527509+0000", 2024-04-19T01:05:12.453 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 42.871404527000003, 2024-04-19T01:05:12.453 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.015116167, 2024-04-19T01:05:12.453 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:12.453 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:12.453 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-04-19T01:05:12.453 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:12.453 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:12.454 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:12.454 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 419 2024-04-19T01:05:12.454 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.454 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:12.454 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.454 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:12.454 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527509+0000", 2024-04-19T01:05:12.454 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:12.454 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.454 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.454 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:12.454 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527509+0000", 2024-04-19T01:05:12.454 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:12.454 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.455 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.455 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:12.455 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527511+0000", 2024-04-19T01:05:12.455 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4869999999999999e-06 2024-04-19T01:05:12.455 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.455 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.455 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:12.455 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527520+0000", 2024-04-19T01:05:12.455 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3689999999999996e-06 2024-04-19T01:05:12.455 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.455 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.455 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:12.455 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527552+0000", 2024-04-19T01:05:12.456 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1316999999999998e-05 2024-04-19T01:05:12.456 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.456 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.456 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:12.456 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527566+0000", 2024-04-19T01:05:12.456 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4893e-05 2024-04-19T01:05:12.456 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.456 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.456 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:12.456 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527787+0000", 2024-04-19T01:05:12.456 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00022044999999999999 2024-04-19T01:05:12.456 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.456 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.456 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:05:12.457 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527803+0000", 2024-04-19T01:05:12.457 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6076e-05 2024-04-19T01:05:12.457 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.457 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.457 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:12.457 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.534575+0000", 2024-04-19T01:05:12.457 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0067724559999999996 2024-04-19T01:05:12.457 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.457 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.457 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:12.457 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.534811+0000", 2024-04-19T01:05:12.457 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00023533000000000001 2024-04-19T01:05:12.457 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.458 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.458 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:12.458 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.542626+0000", 2024-04-19T01:05:12.458 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0078147890000000008 2024-04-19T01:05:12.458 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:12.458 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:12.458 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:12.458 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.458 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.458 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:420 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [read 0~262144,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e59)", 2024-04-19T01:05:12.458 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:29.527610+0000", 2024-04-19T01:05:12.458 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 42.871303961999999, 2024-04-19T01:05:12.458 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.018555769999999999, 2024-04-19T01:05:12.459 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:12.459 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:12.459 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-04-19T01:05:12.459 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:12.459 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:12.459 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:12.459 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 420 2024-04-19T01:05:12.459 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.459 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:12.459 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.459 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:12.459 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527610+0000", 2024-04-19T01:05:12.459 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:12.459 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.460 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.460 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:12.460 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527610+0000", 2024-04-19T01:05:12.460 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:12.460 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.460 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.461 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:12.461 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527612+0000", 2024-04-19T01:05:12.461 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.967e-06 2024-04-19T01:05:12.461 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.461 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.461 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:12.461 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527642+0000", 2024-04-19T01:05:12.461 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9589000000000001e-05 2024-04-19T01:05:12.461 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.461 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.461 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:12.461 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527664+0000", 2024-04-19T01:05:12.461 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2143000000000001e-05 2024-04-19T01:05:12.462 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.462 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.462 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:12.462 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527679+0000", 2024-04-19T01:05:12.462 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5e-05 2024-04-19T01:05:12.462 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.462 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.462 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:12.462 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527873+0000", 2024-04-19T01:05:12.462 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000194537 2024-04-19T01:05:12.462 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.462 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.463 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:05:12.463 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.527889+0000", 2024-04-19T01:05:12.463 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5603e-05 2024-04-19T01:05:12.463 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.463 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.463 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:12.463 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.535626+0000", 2024-04-19T01:05:12.463 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0077369099999999996 2024-04-19T01:05:12.463 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.463 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.463 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:12.463 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.535845+0000", 2024-04-19T01:05:12.463 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000218827 2024-04-19T01:05:12.463 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.464 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.464 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:12.464 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:29.546166+0000", 2024-04-19T01:05:12.464 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.010321194000000001 2024-04-19T01:05:12.464 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:12.464 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:12.464 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:12.464 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.464 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.464 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.8s3 60/31 ECSubWrite(tid=225, reqid=client.4604.0:433, at_version=60'43, trim_to=0'0, roll_forward_to=59'41))", 2024-04-19T01:05:12.464 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:30.569755+0000", 2024-04-19T01:05:12.464 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 41.829159341, 2024-04-19T01:05:12.464 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.026016885999999999, 2024-04-19T01:05:12.464 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:12.465 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:12.465 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:12.465 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:12.465 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.465 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:12.465 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.569755+0000", 2024-04-19T01:05:12.465 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:12.465 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.465 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.465 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:12.465 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.569755+0000", 2024-04-19T01:05:12.465 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:12.465 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.465 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.466 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:12.466 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.569755+0000", 2024-04-19T01:05:12.466 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6599999999999996e-07 2024-04-19T01:05:12.466 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.466 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.466 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:12.466 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.570112+0000", 2024-04-19T01:05:12.466 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00035699100000000002 2024-04-19T01:05:12.466 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.467 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.467 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:12.467 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.570159+0000", 2024-04-19T01:05:12.467 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7312999999999999e-05 2024-04-19T01:05:12.467 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.467 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.467 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:12.467 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.570184+0000", 2024-04-19T01:05:12.467 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4593e-05 2024-04-19T01:05:12.467 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.467 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.467 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:12.467 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.570362+0000", 2024-04-19T01:05:12.467 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017814400000000001 2024-04-19T01:05:12.468 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.468 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.468 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:12.468 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.570381+0000", 2024-04-19T01:05:12.468 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8369999999999999e-05 2024-04-19T01:05:12.468 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.468 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.468 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:12.468 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.595729+0000", 2024-04-19T01:05:12.468 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.025348186000000002 2024-04-19T01:05:12.468 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.468 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.468 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:12.468 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.595771+0000", 2024-04-19T01:05:12.469 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2722999999999997e-05 2024-04-19T01:05:12.469 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:12.469 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:12.469 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:12.469 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.469 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.469 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.4s4 71/31 ECSubWrite(tid=89, reqid=client.4604.0:518, at_version=71'22, trim_to=0'0, roll_forward_to=52'20))", 2024-04-19T01:05:12.469 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:41.542817+0000", 2024-04-19T01:05:12.469 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 30.856097121000001, 2024-04-19T01:05:12.469 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.068188129, 2024-04-19T01:05:12.470 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:12.470 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:12.470 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:12.470 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:12.470 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.470 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:12.470 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.542817+0000", 2024-04-19T01:05:12.470 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:12.470 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.471 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.471 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:12.471 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.542817+0000", 2024-04-19T01:05:12.471 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:12.471 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.471 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.471 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:12.471 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.542819+0000", 2024-04-19T01:05:12.471 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.283e-06 2024-04-19T01:05:12.471 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.471 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.472 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:12.472 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.545764+0000", 2024-04-19T01:05:12.472 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0029444530000000001 2024-04-19T01:05:12.472 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.472 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.472 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:12.472 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.545797+0000", 2024-04-19T01:05:12.472 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3203e-05 2024-04-19T01:05:12.472 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.472 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.472 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:12.472 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.545821+0000", 2024-04-19T01:05:12.472 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4436000000000001e-05 2024-04-19T01:05:12.473 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.473 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.473 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:12.473 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.545998+0000", 2024-04-19T01:05:12.473 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017682099999999999 2024-04-19T01:05:12.473 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.473 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.473 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:12.473 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.546048+0000", 2024-04-19T01:05:12.473 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0423000000000002e-05 2024-04-19T01:05:12.473 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.473 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.473 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:12.474 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.610960+0000", 2024-04-19T01:05:12.474 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.064912005999999994 2024-04-19T01:05:12.474 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.474 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.474 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:12.474 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.611005+0000", 2024-04-19T01:05:12.474 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4503999999999999e-05 2024-04-19T01:05:12.474 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:12.474 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:12.474 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:12.474 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.474 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.474 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:612 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [append 0~704512] snapc 1b=[1b,19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-04-19T01:05:12.475 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:50.652703+0000", 2024-04-19T01:05:12.475 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 21.746210907999998, 2024-04-19T01:05:12.475 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.035173242, 2024-04-19T01:05:12.475 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:12.475 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:12.475 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:12.475 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:12.475 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:12.475 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:12.475 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 612 2024-04-19T01:05:12.475 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.475 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:12.475 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.475 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:12.476 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.652703+0000", 2024-04-19T01:05:12.476 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:12.476 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.476 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.476 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:12.476 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.652703+0000", 2024-04-19T01:05:12.476 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:12.476 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.476 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.476 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:12.476 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.652706+0000", 2024-04-19T01:05:12.476 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4070000000000002e-06 2024-04-19T01:05:12.477 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.477 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.477 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:12.477 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.653472+0000", 2024-04-19T01:05:12.477 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00076519800000000003 2024-04-19T01:05:12.477 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.477 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.477 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:12.477 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.653507+0000", 2024-04-19T01:05:12.477 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5525999999999998e-05 2024-04-19T01:05:12.477 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.477 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.477 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:12.478 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.653525+0000", 2024-04-19T01:05:12.478 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7329e-05 2024-04-19T01:05:12.478 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.478 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.478 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:12.478 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.653662+0000", 2024-04-19T01:05:12.478 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013767900000000001 2024-04-19T01:05:12.478 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.478 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.478 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op must wait for map", 2024-04-19T01:05:12.478 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.653675+0000", 2024-04-19T01:05:12.478 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2723e-05 2024-04-19T01:05:12.478 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.478 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.479 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:12.479 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.658036+0000", 2024-04-19T01:05:12.479 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0043615820000000001 2024-04-19T01:05:12.479 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.479 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.479 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:12.479 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.658195+0000", 2024-04-19T01:05:12.479 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000158448 2024-04-19T01:05:12.479 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.479 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.479 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:12.479 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.660614+0000", 2024-04-19T01:05:12.479 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0024192749999999998 2024-04-19T01:05:12.480 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.480 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.480 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:12.480 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.687106+0000", 2024-04-19T01:05:12.480 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.026491879999999999 2024-04-19T01:05:12.480 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.480 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.480 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:12.480 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.687703+0000", 2024-04-19T01:05:12.480 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00059716299999999995 2024-04-19T01:05:12.480 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.480 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.480 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:12.481 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.687876+0000", 2024-04-19T01:05:12.481 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017303199999999999 2024-04-19T01:05:12.481 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:12.481 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:12.481 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:12.481 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.481 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.481 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:613 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [append 0~655360] snapc 1b=[1b,19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-04-19T01:05:12.482 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:50.661555+0000", 2024-04-19T01:05:12.482 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 21.737358671999999, 2024-04-19T01:05:12.482 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.038292557999999997, 2024-04-19T01:05:12.482 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:12.482 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:12.482 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:12.482 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:12.482 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:12.482 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:12.482 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 613 2024-04-19T01:05:12.482 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.482 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:12.482 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.483 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:12.483 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.661555+0000", 2024-04-19T01:05:12.483 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:12.483 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.483 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.483 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:12.483 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.661555+0000", 2024-04-19T01:05:12.483 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:12.483 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.483 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.483 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:12.483 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.661559+0000", 2024-04-19T01:05:12.483 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.337e-06 2024-04-19T01:05:12.483 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.484 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.484 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:12.484 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.662402+0000", 2024-04-19T01:05:12.484 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00084358700000000003 2024-04-19T01:05:12.484 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.484 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.484 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:12.484 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.662434+0000", 2024-04-19T01:05:12.484 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2223e-05 2024-04-19T01:05:12.484 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.484 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.484 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:12.484 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.662458+0000", 2024-04-19T01:05:12.484 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3598999999999999e-05 2024-04-19T01:05:12.485 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.485 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.485 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:12.485 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.673646+0000", 2024-04-19T01:05:12.485 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.011187852999999999 2024-04-19T01:05:12.485 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.485 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.485 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:12.485 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.673820+0000", 2024-04-19T01:05:12.485 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017450199999999999 2024-04-19T01:05:12.485 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.485 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.485 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:12.486 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.675719+0000", 2024-04-19T01:05:12.486 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001898757 2024-04-19T01:05:12.486 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.486 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.486 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:12.486 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.687984+0000", 2024-04-19T01:05:12.486 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.012265009 2024-04-19T01:05:12.486 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.486 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.486 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:12.486 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.699734+0000", 2024-04-19T01:05:12.486 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.011749426 2024-04-19T01:05:12.486 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.486 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.487 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:12.487 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.699848+0000", 2024-04-19T01:05:12.487 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000114265 2024-04-19T01:05:12.487 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:12.487 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:12.487 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:12.487 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.487 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.487 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 80/31 ECSubWriteReply(tid=63, last_complete=80'51, committed=1, applied=1))", 2024-04-19T01:05:12.487 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:50.667974+0000", 2024-04-19T01:05:12.487 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 21.730939883000001, 2024-04-19T01:05:12.487 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.018952044000000001, 2024-04-19T01:05:12.487 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:12.487 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:12.488 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:12.488 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:12.488 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.488 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:12.488 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.667974+0000", 2024-04-19T01:05:12.488 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:12.488 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.488 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.488 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:12.488 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.667974+0000", 2024-04-19T01:05:12.488 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:12.488 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.488 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.489 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:12.489 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.667976+0000", 2024-04-19T01:05:12.489 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8479999999999999e-06 2024-04-19T01:05:12.489 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.489 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.489 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:12.489 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.667990+0000", 2024-04-19T01:05:12.489 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.403e-05 2024-04-19T01:05:12.489 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.489 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.489 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:12.489 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.668020+0000", 2024-04-19T01:05:12.489 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0222e-05 2024-04-19T01:05:12.490 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.490 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.490 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:12.490 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.668037+0000", 2024-04-19T01:05:12.490 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6424999999999999e-05 2024-04-19T01:05:12.490 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.490 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.490 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:12.490 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.686874+0000", 2024-04-19T01:05:12.490 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.018837596000000002 2024-04-19T01:05:12.490 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.490 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.490 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:12.491 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.686926+0000", 2024-04-19T01:05:12.491 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.1922999999999997e-05 2024-04-19T01:05:12.491 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:12.491 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:12.491 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:12.491 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.491 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.491 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:614 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [append 0~507904] snapc 1b=[1b,19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-04-19T01:05:12.492 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:50.668341+0000", 2024-04-19T01:05:12.492 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 21.730573437, 2024-04-19T01:05:12.492 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.032677876000000002, 2024-04-19T01:05:12.492 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:12.492 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:12.492 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:12.492 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:12.492 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:12.492 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:12.492 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 614 2024-04-19T01:05:12.492 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.492 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:12.493 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.493 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:12.493 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.668341+0000", 2024-04-19T01:05:12.493 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:12.493 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.493 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.493 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:12.493 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.668341+0000", 2024-04-19T01:05:12.493 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:12.493 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.493 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.493 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:12.493 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.668343+0000", 2024-04-19T01:05:12.493 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2759999999999999e-06 2024-04-19T01:05:12.494 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.494 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.494 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:12.494 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.668923+0000", 2024-04-19T01:05:12.494 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00058019699999999996 2024-04-19T01:05:12.494 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.494 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.494 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:12.494 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.668956+0000", 2024-04-19T01:05:12.494 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2539000000000001e-05 2024-04-19T01:05:12.494 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.494 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.494 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:12.495 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.668977+0000", 2024-04-19T01:05:12.495 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1903e-05 2024-04-19T01:05:12.495 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.495 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.495 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:12.495 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.688297+0000", 2024-04-19T01:05:12.495 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.019319106999999999 2024-04-19T01:05:12.495 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.495 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.495 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:12.495 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.688460+0000", 2024-04-19T01:05:12.495 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016338799999999999 2024-04-19T01:05:12.495 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.496 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.496 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:12.496 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.689733+0000", 2024-04-19T01:05:12.496 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001273499 2024-04-19T01:05:12.496 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.496 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.496 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:12.496 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.699996+0000", 2024-04-19T01:05:12.496 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.010263042 2024-04-19T01:05:12.496 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.496 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.496 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:12.496 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.700905+0000", 2024-04-19T01:05:12.496 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00090875999999999999 2024-04-19T01:05:12.496 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.497 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.497 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:12.497 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.701018+0000", 2024-04-19T01:05:12.497 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000113165 2024-04-19T01:05:12.497 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:12.497 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:12.497 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:12.497 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.497 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.497 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:615 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [setxattr _header (57) in=7b] snapc 1b=[1b,19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-04-19T01:05:12.497 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:50.669014+0000", 2024-04-19T01:05:12.497 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 21.729899959000001, 2024-04-19T01:05:12.497 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.032983169999999999, 2024-04-19T01:05:12.498 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:12.498 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:12.498 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:12.498 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:12.498 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:12.498 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:12.498 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 615 2024-04-19T01:05:12.498 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.498 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:12.498 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.498 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:12.498 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669014+0000", 2024-04-19T01:05:12.498 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:12.499 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.499 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.499 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:12.499 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669014+0000", 2024-04-19T01:05:12.499 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:12.499 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.499 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.499 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:12.499 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669016+0000", 2024-04-19T01:05:12.499 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3439999999999999e-06 2024-04-19T01:05:12.499 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.499 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.499 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:12.500 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669039+0000", 2024-04-19T01:05:12.500 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3008999999999999e-05 2024-04-19T01:05:12.500 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.500 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.500 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:12.500 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669057+0000", 2024-04-19T01:05:12.500 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.749e-05 2024-04-19T01:05:12.500 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.500 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.500 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:12.500 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669073+0000", 2024-04-19T01:05:12.500 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6492999999999999e-05 2024-04-19T01:05:12.500 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.500 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.501 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:12.501 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.698263+0000", 2024-04-19T01:05:12.501 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.029189397999999998 2024-04-19T01:05:12.501 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.501 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.501 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:12.501 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.698412+0000", 2024-04-19T01:05:12.501 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014930499999999999 2024-04-19T01:05:12.501 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.501 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.501 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:12.501 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.699113+0000", 2024-04-19T01:05:12.501 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00070141799999999999 2024-04-19T01:05:12.501 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.502 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.502 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:12.502 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.701045+0000", 2024-04-19T01:05:12.502 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001931667 2024-04-19T01:05:12.502 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.502 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.502 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:12.503 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.701847+0000", 2024-04-19T01:05:12.503 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00080155099999999996 2024-04-19T01:05:12.503 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.503 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.503 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:12.503 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.701997+0000", 2024-04-19T01:05:12.503 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000150495 2024-04-19T01:05:12.503 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:12.503 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:12.503 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:12.503 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.503 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.503 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:616 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e80)", 2024-04-19T01:05:12.503 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:50.669097+0000", 2024-04-19T01:05:12.504 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 21.729817423, 2024-04-19T01:05:12.504 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.038907733999999999, 2024-04-19T01:05:12.504 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:12.504 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:12.504 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-04-19T01:05:12.504 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:12.504 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:12.504 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:12.504 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 616 2024-04-19T01:05:12.504 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.504 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:12.504 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.504 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:12.505 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669097+0000", 2024-04-19T01:05:12.505 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:12.505 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.505 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.505 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:12.505 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669097+0000", 2024-04-19T01:05:12.505 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:12.505 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.505 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.505 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:12.506 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669098+0000", 2024-04-19T01:05:12.506 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.587e-06 2024-04-19T01:05:12.506 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.506 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.506 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:12.506 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669105+0000", 2024-04-19T01:05:12.506 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3660000000000003e-06 2024-04-19T01:05:12.506 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.506 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.506 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:12.506 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669130+0000", 2024-04-19T01:05:12.506 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4377000000000001e-05 2024-04-19T01:05:12.507 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.507 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.507 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:12.507 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669146+0000", 2024-04-19T01:05:12.507 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6399e-05 2024-04-19T01:05:12.507 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.507 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.507 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:12.507 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.702705+0000", 2024-04-19T01:05:12.507 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.033558534000000001 2024-04-19T01:05:12.507 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.507 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.507 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:12.507 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.702962+0000", 2024-04-19T01:05:12.508 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00025750700000000003 2024-04-19T01:05:12.508 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.508 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.508 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:12.508 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.708004+0000", 2024-04-19T01:05:12.508 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0050419640000000003 2024-04-19T01:05:12.508 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:12.508 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:12.508 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:12.508 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.508 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.508 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.4s4 81/31 ECSubWrite(tid=117, reqid=client.4604.0:618, at_version=81'41, trim_to=0'0, roll_forward_to=79'39))", 2024-04-19T01:05:12.508 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:51.536707+0000", 2024-04-19T01:05:12.509 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 20.862206871000001, 2024-04-19T01:05:12.509 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.019218485, 2024-04-19T01:05:12.509 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:12.509 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:12.509 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:12.509 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:12.509 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.509 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:12.509 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.536707+0000", 2024-04-19T01:05:12.509 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:12.509 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.509 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.509 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:12.509 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.536707+0000", 2024-04-19T01:05:12.510 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:12.510 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.510 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.510 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:12.510 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.536720+0000", 2024-04-19T01:05:12.510 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3267e-05 2024-04-19T01:05:12.510 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.510 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.510 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:12.510 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.537717+0000", 2024-04-19T01:05:12.510 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00099714199999999991 2024-04-19T01:05:12.510 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.510 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.510 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:12.511 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.537750+0000", 2024-04-19T01:05:12.511 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2558999999999998e-05 2024-04-19T01:05:12.511 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.511 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.511 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:12.511 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.537773+0000", 2024-04-19T01:05:12.511 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3013000000000001e-05 2024-04-19T01:05:12.511 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.511 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.511 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:12.511 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.537958+0000", 2024-04-19T01:05:12.511 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00018457400000000001 2024-04-19T01:05:12.511 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.512 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.512 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:12.512 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.537976+0000", 2024-04-19T01:05:12.512 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8734000000000001e-05 2024-04-19T01:05:12.512 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.512 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.512 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:12.512 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.555848+0000", 2024-04-19T01:05:12.513 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.017871617999999999 2024-04-19T01:05:12.513 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.513 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.513 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:12.513 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.555926+0000", 2024-04-19T01:05:12.513 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7577999999999995e-05 2024-04-19T01:05:12.513 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:12.513 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:12.513 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:12.513 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.513 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.513 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.es6 86/31 ECSubWrite(tid=195, reqid=client.4604.0:653, at_version=86'39, trim_to=0'0, roll_forward_to=84'38))", 2024-04-19T01:05:12.513 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:56.572643+0000", 2024-04-19T01:05:12.514 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 15.826271016, 2024-04-19T01:05:12.514 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.035539842000000002, 2024-04-19T01:05:12.514 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:12.514 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:12.514 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:12.514 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:12.514 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.514 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:12.514 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.572643+0000", 2024-04-19T01:05:12.514 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:12.514 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.514 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.514 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:12.514 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.572643+0000", 2024-04-19T01:05:12.514 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:12.515 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.515 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.515 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:12.515 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.572644+0000", 2024-04-19T01:05:12.515 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8999999999999996e-07 2024-04-19T01:05:12.515 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.515 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.515 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:12.515 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.573792+0000", 2024-04-19T01:05:12.515 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011481810000000001 2024-04-19T01:05:12.515 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.515 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.515 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:12.516 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.573818+0000", 2024-04-19T01:05:12.516 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5814000000000001e-05 2024-04-19T01:05:12.516 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.516 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.516 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:12.516 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.573832+0000", 2024-04-19T01:05:12.516 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4056999999999999e-05 2024-04-19T01:05:12.516 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.516 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.516 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:12.516 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.573964+0000", 2024-04-19T01:05:12.516 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013278399999999999 2024-04-19T01:05:12.516 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.517 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.517 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:12.517 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.573992+0000", 2024-04-19T01:05:12.518 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7250000000000002e-05 2024-04-19T01:05:12.518 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.518 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.518 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:12.518 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.608115+0000", 2024-04-19T01:05:12.518 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.034123819999999999 2024-04-19T01:05:12.518 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.518 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.519 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:12.519 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.608183+0000", 2024-04-19T01:05:12.519 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7346000000000002e-05 2024-04-19T01:05:12.519 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:12.519 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:12.519 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:12.519 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.519 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.519 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.es6 92/31 ECSubWrite(tid=209, reqid=client.4604.0:701, at_version=92'41, trim_to=0'0, roll_forward_to=86'39))", 2024-04-19T01:05:12.519 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:02.701844+0000", 2024-04-19T01:05:12.519 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 9.6970703409999999, 2024-04-19T01:05:12.519 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.017895945999999999, 2024-04-19T01:05:12.519 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:12.519 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:12.519 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:12.520 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:12.520 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.520 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:12.520 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:02.701844+0000", 2024-04-19T01:05:12.520 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:12.520 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.520 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.520 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:12.520 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:02.701844+0000", 2024-04-19T01:05:12.520 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:12.520 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.520 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.520 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:12.521 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:02.701844+0000", 2024-04-19T01:05:12.521 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6300000000000004e-07 2024-04-19T01:05:12.521 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.521 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.521 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:12.521 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:02.702617+0000", 2024-04-19T01:05:12.521 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00077301300000000002 2024-04-19T01:05:12.521 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.521 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.521 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:12.521 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:02.702642+0000", 2024-04-19T01:05:12.521 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4892000000000002e-05 2024-04-19T01:05:12.521 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.522 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.522 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:12.522 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:02.702658+0000", 2024-04-19T01:05:12.522 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.571e-05 2024-04-19T01:05:12.522 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.522 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.523 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:12.523 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:02.702789+0000", 2024-04-19T01:05:12.523 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000130557 2024-04-19T01:05:12.523 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.523 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.523 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:12.523 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:02.702806+0000", 2024-04-19T01:05:12.523 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6969999999999998e-05 2024-04-19T01:05:12.523 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.523 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.523 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:12.523 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:02.719639+0000", 2024-04-19T01:05:12.524 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.016833844000000001 2024-04-19T01:05:12.524 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:12.524 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:12.525 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:12.525 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:02.719740+0000", 2024-04-19T01:05:12.525 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000100197 2024-04-19T01:05:12.525 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:12.525 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:12.525 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:12.525 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:12.525 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:12.525 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:05:12.526 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:05:12.397+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:05:12.527 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-19T01:05:12.528 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:05:12.526+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:05:12.547 INFO:tasks.rados.rados.0.smithi155.stdout:311: done (0 left) 2024-04-19T01:05:12.547 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 23 current snap is 36 2024-04-19T01:05:12.547 INFO:tasks.rados.rados.0.smithi155.stdout:312: seq_num 126 ranges {2129920=573440,2703360=622592,3325952=622592,3948544=311296} 2024-04-19T01:05:12.598 INFO:tasks.rados.rados.0.smithi155.stdout:312: writing smithi15528762-23 from 2129920 to 2703360 tid 1 2024-04-19T01:05:12.608 INFO:tasks.rados.rados.0.smithi155.stdout:312: writing smithi15528762-23 from 2703360 to 3325952 tid 2 2024-04-19T01:05:12.615 INFO:tasks.rados.rados.0.smithi155.stdout:312: writing smithi15528762-23 from 3325952 to 3948544 tid 3 2024-04-19T01:05:12.621 INFO:tasks.rados.rados.0.smithi155.stdout:312: writing smithi15528762-23 from 3948544 to 4259840 tid 4 2024-04-19T01:05:12.622 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 30 current snap is 36 2024-04-19T01:05:12.622 INFO:tasks.rados.rados.0.smithi155.stdout:313: seq_num 127 ranges {5718016=589824,6307840=753664,7061504=475136,7536640=327680} 2024-04-19T01:05:12.629 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:05:12.626+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:05:12.666 INFO:tasks.rados.rados.0.smithi155.stdout:313: writing smithi15528762-30 from 5718016 to 6307840 tid 1 2024-04-19T01:05:12.670 INFO:tasks.rados.rados.0.smithi155.stdout:313: writing smithi15528762-30 from 6307840 to 7061504 tid 2 2024-04-19T01:05:12.673 INFO:tasks.rados.rados.0.smithi155.stdout:313: writing smithi15528762-30 from 7061504 to 7536640 tid 3 2024-04-19T01:05:12.675 INFO:tasks.rados.rados.0.smithi155.stdout:313: writing smithi15528762-30 from 7536640 to 7864320 tid 4 2024-04-19T01:05:12.675 INFO:tasks.rados.rados.0.smithi155.stdout:314: rmattr oid 48 current snap is 36 2024-04-19T01:05:12.675 INFO:tasks.rados.rados.0.smithi155.stdout:312: finishing write tid 1 to smithi15528762-23 2024-04-19T01:05:12.676 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 1 current snap is 36 2024-04-19T01:05:12.676 INFO:tasks.rados.rados.0.smithi155.stdout:312: finishing write tid 2 to smithi15528762-23 2024-04-19T01:05:12.676 INFO:tasks.rados.rados.0.smithi155.stdout:312: finishing write tid 3 to smithi15528762-23 2024-04-19T01:05:12.676 INFO:tasks.rados.rados.0.smithi155.stdout:315: seq_num 128 ranges {1818624=606208,2424832=507904,2932736=573440,3506176=475136} 2024-04-19T01:05:12.681 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:05:12.682 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:05:12.682 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2024-04-19T01:05:12.682 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:05:12.689 INFO:tasks.rados.rados.0.smithi155.stdout:315: writing smithi15528762-1 from 1818624 to 2424832 tid 1 2024-04-19T01:05:12.692 INFO:tasks.rados.rados.0.smithi155.stdout:315: writing smithi15528762-1 from 2424832 to 2932736 tid 2 2024-04-19T01:05:12.695 INFO:tasks.rados.rados.0.smithi155.stdout:315: writing smithi15528762-1 from 2932736 to 3506176 tid 3 2024-04-19T01:05:12.699 INFO:tasks.rados.rados.0.smithi155.stdout:315: writing smithi15528762-1 from 3506176 to 3981312 tid 4 2024-04-19T01:05:12.699 INFO:tasks.rados.rados.0.smithi155.stdout:316: rollback oid 26 current snap is 36 2024-04-19T01:05:12.699 INFO:tasks.rados.rados.0.smithi155.stdout:rollback oid 26 to 35 2024-04-19T01:05:12.699 INFO:tasks.rados.rados.0.smithi155.stdout:312: finishing write tid 4 to smithi15528762-23 2024-04-19T01:05:12.699 INFO:tasks.rados.rados.0.smithi155.stdout:317: read oid 41 snap -1 2024-04-19T01:05:12.699 INFO:tasks.rados.rados.0.smithi155.stdout:317: expect deleted 2024-04-19T01:05:12.699 INFO:tasks.rados.rados.0.smithi155.stdout:318: snap_create 2024-04-19T01:05:12.699 INFO:tasks.rados.rados.0.smithi155.stdout:312: finishing write tid 5 to smithi15528762-23 2024-04-19T01:05:12.699 INFO:tasks.rados.rados.0.smithi155.stdout:312: finishing write tid 6 to smithi15528762-23 2024-04-19T01:05:12.699 INFO:tasks.rados.rados.0.smithi155.stdout:312: oid 23 updating version 0 to 80 2024-04-19T01:05:12.700 INFO:tasks.rados.rados.0.smithi155.stdout:312: oid 23 updating version 80 to 82 2024-04-19T01:05:12.700 INFO:tasks.rados.rados.0.smithi155.stdout:312: oid 23 version 82 is already newer than 81 2024-04-19T01:05:12.700 INFO:tasks.rados.rados.0.smithi155.stdout:312: oid 23 version 82 is already newer than 79 2024-04-19T01:05:12.700 INFO:tasks.rados.rados.0.smithi155.stdout:312: oid 23 updating version 82 to 83 2024-04-19T01:05:12.700 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 23 v 83 (ObjNum 126 snap 36 seq_num 126) dirty exists 2024-04-19T01:05:12.700 INFO:tasks.rados.rados.0.smithi155.stdout:312: left oid 23 (ObjNum 126 snap 36 seq_num 126) 2024-04-19T01:05:12.700 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 48 v 85 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-04-19T01:05:12.700 INFO:tasks.rados.rados.0.smithi155.stdout:313: finishing write tid 1 to smithi15528762-30 2024-04-19T01:05:12.701 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-19T01:05:12.728 INFO:tasks.rados.rados.0.smithi155.stdout:313: finishing write tid 2 to smithi15528762-30 2024-04-19T01:05:12.728 INFO:tasks.rados.rados.0.smithi155.stdout:313: finishing write tid 3 to smithi15528762-30 2024-04-19T01:05:12.730 INFO:tasks.rados.rados.0.smithi155.stdout:313: finishing write tid 4 to smithi15528762-30 2024-04-19T01:05:12.731 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:05:12.730+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:05:12.731 INFO:tasks.rados.rados.0.smithi155.stdout:313: finishing write tid 5 to smithi15528762-30 2024-04-19T01:05:12.732 INFO:tasks.rados.rados.0.smithi155.stdout:316: finishing rollback tid 0 to smithi15528762-26 2024-04-19T01:05:12.734 INFO:tasks.rados.rados.0.smithi155.stdout:316: finishing rollback tid 1 to smithi15528762-26 2024-04-19T01:05:12.737 INFO:tasks.rados.rados.0.smithi155.stdout:313: finishing write tid 6 to smithi15528762-30 2024-04-19T01:05:12.737 INFO:tasks.rados.rados.0.smithi155.stdout:313: oid 30 updating version 0 to 106 2024-04-19T01:05:12.737 INFO:tasks.rados.rados.0.smithi155.stdout:313: oid 30 version 106 is already newer than 103 2024-04-19T01:05:12.738 INFO:tasks.rados.rados.0.smithi155.stdout:313: oid 30 version 106 is already newer than 104 2024-04-19T01:05:12.738 INFO:tasks.rados.rados.0.smithi155.stdout:313: oid 30 version 106 is already newer than 105 2024-04-19T01:05:12.738 INFO:tasks.rados.rados.0.smithi155.stdout:313: oid 30 version 106 is already newer than 102 2024-04-19T01:05:12.738 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 30 v 106 (ObjNum 127 snap 36 seq_num 127) dirty exists 2024-04-19T01:05:12.738 INFO:tasks.rados.rados.0.smithi155.stdout:313: left oid 30 (ObjNum 127 snap 36 seq_num 127) 2024-04-19T01:05:12.742 INFO:tasks.rados.rados.0.smithi155.stdout:316: finishing rollback tid 2 to smithi15528762-26 2024-04-19T01:05:12.743 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 26 v 110 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-19T01:05:12.743 INFO:tasks.rados.rados.0.smithi155.stdout:315: finishing write tid 1 to smithi15528762-1 2024-04-19T01:05:12.743 INFO:tasks.rados.rados.0.smithi155.stdout:315: finishing write tid 2 to smithi15528762-1 2024-04-19T01:05:12.743 INFO:tasks.rados.rados.0.smithi155.stdout:315: finishing write tid 3 to smithi15528762-1 2024-04-19T01:05:12.744 INFO:tasks.rados.rados.0.smithi155.stdout:315: finishing write tid 4 to smithi15528762-1 2024-04-19T01:05:12.744 INFO:tasks.rados.rados.0.smithi155.stdout:315: finishing write tid 5 to smithi15528762-1 2024-04-19T01:05:12.745 INFO:tasks.rados.rados.0.smithi155.stdout:315: finishing write tid 6 to smithi15528762-1 2024-04-19T01:05:12.745 INFO:tasks.rados.rados.0.smithi155.stdout:315: oid 1 updating version 0 to 78 2024-04-19T01:05:12.745 INFO:tasks.rados.rados.0.smithi155.stdout:315: oid 1 updating version 78 to 81 2024-04-19T01:05:12.745 INFO:tasks.rados.rados.0.smithi155.stdout:315: oid 1 version 81 is already newer than 80 2024-04-19T01:05:12.745 INFO:tasks.rados.rados.0.smithi155.stdout:315: oid 1 updating version 81 to 82 2024-04-19T01:05:12.746 INFO:tasks.rados.rados.0.smithi155.stdout:315: oid 1 version 82 is already newer than 79 2024-04-19T01:05:12.746 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 1 v 82 (ObjNum 128 snap 36 seq_num 128) dirty exists 2024-04-19T01:05:12.746 INFO:tasks.rados.rados.0.smithi155.stdout:315: left oid 1 (ObjNum 128 snap 36 seq_num 128) 2024-04-19T01:05:12.832 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:05:12.830+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:05:12.834 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:05:12.834 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:05:12.834 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2024-04-19T01:05:12.834 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2024-04-19T01:05:12.835 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:05:12.850 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-19T01:05:12.863 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-19T01:05:12.932 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:05:12.934+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:05:12.975 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:05:12.975 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2024-04-19T01:05:12.975 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2024-04-19T01:05:12.975 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-04-19T01:05:12.975 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.976 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:263 3.fs0 3:f8a60714:::smithi15528762-14:head [append 0~720896] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:05:12.976 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.990535+0000", 2024-04-19T01:05:12.976 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 58.981946512, 2024-04-19T01:05:12.976 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.043978092000000003, 2024-04-19T01:05:12.976 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:12.976 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:12.976 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:12.976 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:12.976 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:12.976 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:12.976 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 263 2024-04-19T01:05:12.976 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.976 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:12.977 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.977 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:12.977 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.990535+0000", 2024-04-19T01:05:12.977 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:12.977 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.977 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.977 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:12.977 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.990535+0000", 2024-04-19T01:05:12.977 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:12.977 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.977 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.977 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:12.977 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.990537+0000", 2024-04-19T01:05:12.977 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4160000000000001e-06 2024-04-19T01:05:12.977 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.978 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.978 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:12.978 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991425+0000", 2024-04-19T01:05:12.978 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00088775099999999999 2024-04-19T01:05:12.978 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.978 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.978 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:12.978 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991440+0000", 2024-04-19T01:05:12.978 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5305999999999999e-05 2024-04-19T01:05:12.978 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.978 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.978 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:12.978 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991447+0000", 2024-04-19T01:05:12.978 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.4760000000000001e-06 2024-04-19T01:05:12.979 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.979 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.979 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:12.979 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991516+0000", 2024-04-19T01:05:12.979 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.9041999999999994e-05 2024-04-19T01:05:12.979 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.979 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.979 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:12.979 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991620+0000", 2024-04-19T01:05:12.979 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000103282 2024-04-19T01:05:12.979 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.979 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.979 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:12.979 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993018+0000", 2024-04-19T01:05:12.980 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001397976 2024-04-19T01:05:12.980 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.980 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.980 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:12.980 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.004440+0000", 2024-04-19T01:05:12.980 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011422261 2024-04-19T01:05:12.980 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.980 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.980 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:12.980 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.034454+0000", 2024-04-19T01:05:12.980 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.030013782999999999 2024-04-19T01:05:12.980 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.980 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.980 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:12.980 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.034513+0000", 2024-04-19T01:05:12.981 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.9799000000000001e-05 2024-04-19T01:05:12.981 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:12.981 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:12.981 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:12.981 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.981 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.981 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:264 3.fs0 3:f8a60714:::smithi15528762-14:head [append 0~376832] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:05:12.981 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.992666+0000", 2024-04-19T01:05:12.981 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 58.979815739999999, 2024-04-19T01:05:12.981 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.041981560000000001, 2024-04-19T01:05:12.981 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:12.981 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:12.981 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:12.981 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:12.982 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:12.982 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:12.982 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 264 2024-04-19T01:05:12.982 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.982 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:12.982 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.982 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:12.982 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.992666+0000", 2024-04-19T01:05:12.982 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:12.982 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.983 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.983 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:12.983 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.992666+0000", 2024-04-19T01:05:12.983 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:12.983 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.983 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.983 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:12.983 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.992668+0000", 2024-04-19T01:05:12.983 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.483e-06 2024-04-19T01:05:12.983 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.983 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.983 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:12.984 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993021+0000", 2024-04-19T01:05:12.984 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000352958 2024-04-19T01:05:12.984 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.984 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.984 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:12.984 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993031+0000", 2024-04-19T01:05:12.984 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0723000000000001e-05 2024-04-19T01:05:12.984 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.984 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.984 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:12.984 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993040+0000", 2024-04-19T01:05:12.984 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.507e-06 2024-04-19T01:05:12.984 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.985 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.985 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:12.985 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.000363+0000", 2024-04-19T01:05:12.985 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0073236819999999998 2024-04-19T01:05:12.985 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.985 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.985 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:12.985 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.000482+0000", 2024-04-19T01:05:12.985 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011817299999999999 2024-04-19T01:05:12.985 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.985 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.985 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:12.985 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.001332+0000", 2024-04-19T01:05:12.985 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000849886 2024-04-19T01:05:12.986 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.986 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.986 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:12.986 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.006292+0000", 2024-04-19T01:05:12.986 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0049605090000000001 2024-04-19T01:05:12.986 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.986 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.986 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:12.986 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.034602+0000", 2024-04-19T01:05:12.986 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.028310076999999999 2024-04-19T01:05:12.986 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.986 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.986 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:12.986 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.034648+0000", 2024-04-19T01:05:12.986 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5562000000000002e-05 2024-04-19T01:05:12.987 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:12.987 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:12.987 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:12.987 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.987 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.987 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:265 3.fs0 3:f8a60714:::smithi15528762-14:head [setxattr _header (56) in=7b] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:05:12.987 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.993067+0000", 2024-04-19T01:05:12.987 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 58.979414501999997, 2024-04-19T01:05:12.987 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.041773988999999997, 2024-04-19T01:05:12.987 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:12.987 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:12.987 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:12.987 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:12.987 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:12.987 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:12.988 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 265 2024-04-19T01:05:12.988 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.988 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:12.988 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.988 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:12.988 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993067+0000", 2024-04-19T01:05:12.988 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:12.988 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.988 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.988 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:12.988 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993067+0000", 2024-04-19T01:05:12.988 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:12.988 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.988 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.988 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:12.989 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993068+0000", 2024-04-19T01:05:12.989 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2699999999999999e-07 2024-04-19T01:05:12.989 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.989 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.989 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:12.989 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993072+0000", 2024-04-19T01:05:12.989 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0539999999999996e-06 2024-04-19T01:05:12.989 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.989 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.989 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:12.989 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993080+0000", 2024-04-19T01:05:12.989 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6140000000000003e-06 2024-04-19T01:05:12.989 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.989 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.989 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:12.990 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993119+0000", 2024-04-19T01:05:12.990 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8754000000000002e-05 2024-04-19T01:05:12.990 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.990 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.990 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:12.990 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.004460+0000", 2024-04-19T01:05:12.990 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011341188 2024-04-19T01:05:12.990 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.990 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.990 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:12.990 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.004518+0000", 2024-04-19T01:05:12.990 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8306e-05 2024-04-19T01:05:12.990 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.990 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.991 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:12.991 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.004774+0000", 2024-04-19T01:05:12.991 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00025565400000000002 2024-04-19T01:05:12.991 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.991 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.991 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:12.991 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.006318+0000", 2024-04-19T01:05:12.991 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0015444409999999999 2024-04-19T01:05:12.991 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.991 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.991 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:12.991 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.034787+0000", 2024-04-19T01:05:12.991 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.028468566000000001 2024-04-19T01:05:12.991 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.992 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.992 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:12.992 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.034841+0000", 2024-04-19T01:05:12.992 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4685000000000002e-05 2024-04-19T01:05:12.992 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:12.992 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:12.992 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:12.992 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.992 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.992 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:266 3.fs0 3:f8a60714:::smithi15528762-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:05:12.992 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.993144+0000", 2024-04-19T01:05:12.992 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 58.979337448999999, 2024-04-19T01:05:12.992 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.042235719999999997, 2024-04-19T01:05:12.992 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:12.993 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:12.993 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:05:12.993 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:12.993 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:12.993 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:12.993 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 266 2024-04-19T01:05:12.993 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.993 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:12.993 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.993 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:12.993 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993144+0000", 2024-04-19T01:05:12.993 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:12.993 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.994 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.994 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:12.994 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993144+0000", 2024-04-19T01:05:12.994 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:12.994 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.994 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.994 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:12.994 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993145+0000", 2024-04-19T01:05:12.994 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5099999999999999e-07 2024-04-19T01:05:12.994 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.994 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.994 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:12.994 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993149+0000", 2024-04-19T01:05:12.994 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4539999999999999e-06 2024-04-19T01:05:12.995 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.995 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.995 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:12.995 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993157+0000", 2024-04-19T01:05:12.995 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.6349999999999995e-06 2024-04-19T01:05:12.995 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.995 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.995 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:12.995 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993166+0000", 2024-04-19T01:05:12.995 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2709999999999998e-06 2024-04-19T01:05:12.995 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.995 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.995 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:12.995 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.004924+0000", 2024-04-19T01:05:12.995 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011757834 2024-04-19T01:05:12.996 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.996 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.996 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:05:12.996 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.004971+0000", 2024-04-19T01:05:12.996 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6947000000000003e-05 2024-04-19T01:05:12.996 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.996 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.996 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:12.996 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.035366+0000", 2024-04-19T01:05:12.996 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.03039501 2024-04-19T01:05:12.996 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.996 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.996 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:12.996 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.035380+0000", 2024-04-19T01:05:12.997 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3918e-05 2024-04-19T01:05:12.997 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:12.997 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:12.997 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:12.997 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.997 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.997 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:317 3.fs0 3:f23900a8:::smithi15528762-6:head [append 0~475136] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:05:12.997 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.601883+0000", 2024-04-19T01:05:12.997 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 55.370598471000001, 2024-04-19T01:05:12.997 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.029134103000000001, 2024-04-19T01:05:12.997 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:12.997 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:12.997 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:12.998 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:12.998 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:12.998 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:12.998 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 317 2024-04-19T01:05:12.998 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.998 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:12.998 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.998 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:12.998 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.601883+0000", 2024-04-19T01:05:12.998 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:12.998 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.998 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.998 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:12.998 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.601883+0000", 2024-04-19T01:05:12.999 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:12.999 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.999 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.999 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:12.999 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.601886+0000", 2024-04-19T01:05:12.999 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.2709999999999999e-06 2024-04-19T01:05:12.999 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.999 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.999 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:12.999 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.602696+0000", 2024-04-19T01:05:12.999 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00080995700000000004 2024-04-19T01:05:12.999 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:12.999 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:12.999 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:13.000 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.602714+0000", 2024-04-19T01:05:13.000 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8168999999999998e-05 2024-04-19T01:05:13.000 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.000 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.000 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:13.000 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.602721+0000", 2024-04-19T01:05:13.000 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.554e-06 2024-04-19T01:05:13.000 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.000 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.000 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:13.000 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.602799+0000", 2024-04-19T01:05:13.000 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7804000000000002e-05 2024-04-19T01:05:13.000 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.000 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.001 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:13.001 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.602920+0000", 2024-04-19T01:05:13.001 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012105299999999999 2024-04-19T01:05:13.001 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.001 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.001 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:13.001 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.604215+0000", 2024-04-19T01:05:13.001 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0012944129999999999 2024-04-19T01:05:13.001 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.001 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.001 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:13.001 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.609630+0000", 2024-04-19T01:05:13.001 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0054153789999999997 2024-04-19T01:05:13.002 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.002 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.002 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:13.002 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.630924+0000", 2024-04-19T01:05:13.002 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021294306999999998 2024-04-19T01:05:13.002 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.002 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.002 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:13.002 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.631018+0000", 2024-04-19T01:05:13.002 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.3195999999999993e-05 2024-04-19T01:05:13.002 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:13.002 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:13.002 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:13.003 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.003 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.003 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:318 3.fs0 3:f23900a8:::smithi15528762-6:head [append 0~622592] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:05:13.003 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.609539+0000", 2024-04-19T01:05:13.003 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 55.362942877000002, 2024-04-19T01:05:13.003 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021603679000000001, 2024-04-19T01:05:13.003 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:13.003 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:13.003 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:13.003 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:13.003 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:13.003 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:13.003 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 318 2024-04-19T01:05:13.003 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.004 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:13.004 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.004 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:13.004 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.609539+0000", 2024-04-19T01:05:13.004 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:13.004 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.004 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.004 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:13.004 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.609539+0000", 2024-04-19T01:05:13.004 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:13.004 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.004 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.004 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:13.005 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.609541+0000", 2024-04-19T01:05:13.005 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.855e-06 2024-04-19T01:05:13.005 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.005 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.005 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:13.005 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610203+0000", 2024-04-19T01:05:13.005 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00066190200000000002 2024-04-19T01:05:13.005 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.005 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.005 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:13.005 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610215+0000", 2024-04-19T01:05:13.006 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2143e-05 2024-04-19T01:05:13.006 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.006 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.006 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:13.006 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610221+0000", 2024-04-19T01:05:13.006 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4590000000000003e-06 2024-04-19T01:05:13.006 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.006 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.006 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:13.006 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610283+0000", 2024-04-19T01:05:13.007 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1322999999999995e-05 2024-04-19T01:05:13.007 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.007 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.007 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:13.007 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610394+0000", 2024-04-19T01:05:13.007 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000111044 2024-04-19T01:05:13.007 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.007 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.007 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:13.007 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611327+0000", 2024-04-19T01:05:13.007 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00093369799999999995 2024-04-19T01:05:13.007 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.007 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.008 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:13.008 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.623095+0000", 2024-04-19T01:05:13.008 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011767454 2024-04-19T01:05:13.008 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.008 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.008 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:13.008 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.631101+0000", 2024-04-19T01:05:13.008 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0080059889999999998 2024-04-19T01:05:13.008 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.008 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.008 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:13.008 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.631143+0000", 2024-04-19T01:05:13.008 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1811999999999999e-05 2024-04-19T01:05:13.009 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:13.009 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:13.009 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:13.009 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.009 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.009 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:319 3.fs0 3:f23900a8:::smithi15528762-6:head [append 0~147456] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:05:13.009 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.611402+0000", 2024-04-19T01:05:13.009 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 55.361079914000001, 2024-04-19T01:05:13.009 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.020301099, 2024-04-19T01:05:13.009 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:13.009 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:13.009 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:13.009 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:13.010 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:13.010 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:13.010 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 319 2024-04-19T01:05:13.010 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.010 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:13.010 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.010 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:13.010 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611402+0000", 2024-04-19T01:05:13.010 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:13.010 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.011 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.011 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:13.011 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611402+0000", 2024-04-19T01:05:13.011 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:13.011 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.011 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.011 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:13.011 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611404+0000", 2024-04-19T01:05:13.011 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.52e-06 2024-04-19T01:05:13.011 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.011 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.011 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:13.011 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611632+0000", 2024-04-19T01:05:13.011 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000227526 2024-04-19T01:05:13.012 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.012 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.012 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:13.012 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611660+0000", 2024-04-19T01:05:13.012 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7841999999999999e-05 2024-04-19T01:05:13.012 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.012 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.012 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:13.012 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611678+0000", 2024-04-19T01:05:13.012 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8479e-05 2024-04-19T01:05:13.012 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.012 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.012 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:13.013 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.620224+0000", 2024-04-19T01:05:13.013 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0085460999999999992 2024-04-19T01:05:13.013 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.013 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.013 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:13.013 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.620334+0000", 2024-04-19T01:05:13.013 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010964 2024-04-19T01:05:13.013 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.013 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.013 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:13.013 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.620982+0000", 2024-04-19T01:05:13.013 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00064788699999999999 2024-04-19T01:05:13.013 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.013 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.014 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:13.014 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.623230+0000", 2024-04-19T01:05:13.014 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0022481950000000001 2024-04-19T01:05:13.014 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.014 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.014 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:13.014 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.631664+0000", 2024-04-19T01:05:13.014 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0084339810000000001 2024-04-19T01:05:13.014 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.014 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.014 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:13.014 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.631703+0000", 2024-04-19T01:05:13.014 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8928999999999999e-05 2024-04-19T01:05:13.014 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:13.015 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:13.015 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:13.015 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.015 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.015 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:320 3.fs0 3:f23900a8:::smithi15528762-6:head [setxattr _header (55) in=7b] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:05:13.015 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.611709+0000", 2024-04-19T01:05:13.015 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 55.360772420000004, 2024-04-19T01:05:13.015 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.020427773, 2024-04-19T01:05:13.015 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:13.015 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:13.015 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:13.015 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:13.015 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:13.015 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:13.015 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 320 2024-04-19T01:05:13.015 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.016 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:13.016 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.016 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:13.016 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611709+0000", 2024-04-19T01:05:13.016 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:13.016 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.016 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.016 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:13.016 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611709+0000", 2024-04-19T01:05:13.016 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:13.016 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.016 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.016 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:13.016 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611711+0000", 2024-04-19T01:05:13.016 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.451e-06 2024-04-19T01:05:13.017 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.017 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.017 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:13.017 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611718+0000", 2024-04-19T01:05:13.017 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5800000000000003e-06 2024-04-19T01:05:13.017 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.017 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.017 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:13.017 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611736+0000", 2024-04-19T01:05:13.017 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7339000000000001e-05 2024-04-19T01:05:13.017 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.017 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.017 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:13.017 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611749+0000", 2024-04-19T01:05:13.017 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3557000000000001e-05 2024-04-19T01:05:13.018 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.018 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.018 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:13.018 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.622598+0000", 2024-04-19T01:05:13.018 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010848315000000001 2024-04-19T01:05:13.018 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.018 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.018 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:13.018 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.622654+0000", 2024-04-19T01:05:13.018 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6311999999999999e-05 2024-04-19T01:05:13.018 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.018 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.018 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:13.019 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.622895+0000", 2024-04-19T01:05:13.019 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00024134300000000001 2024-04-19T01:05:13.019 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.019 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.019 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:13.019 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.623782+0000", 2024-04-19T01:05:13.019 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00088635099999999996 2024-04-19T01:05:13.019 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.019 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.019 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:13.019 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.632093+0000", 2024-04-19T01:05:13.019 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.008311466 2024-04-19T01:05:13.019 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.019 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.020 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:13.020 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.632137+0000", 2024-04-19T01:05:13.020 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4058999999999999e-05 2024-04-19T01:05:13.020 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:13.020 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:13.020 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:13.020 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.020 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.020 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:321 3.fs0 3:f23900a8:::smithi15528762-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:05:13.020 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.611769+0000", 2024-04-19T01:05:13.020 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 55.360712882999998, 2024-04-19T01:05:13.020 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021684730999999999, 2024-04-19T01:05:13.020 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:13.020 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:13.021 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:05:13.021 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:13.021 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:13.021 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:13.021 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 321 2024-04-19T01:05:13.021 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.021 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:13.021 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.021 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:13.021 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611769+0000", 2024-04-19T01:05:13.021 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:13.021 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.021 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.021 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:13.022 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611769+0000", 2024-04-19T01:05:13.022 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:13.022 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.022 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.023 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:13.023 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611770+0000", 2024-04-19T01:05:13.023 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.04e-06 2024-04-19T01:05:13.023 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.023 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.023 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:13.023 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611775+0000", 2024-04-19T01:05:13.023 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3310000000000003e-06 2024-04-19T01:05:13.023 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.023 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.023 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:13.023 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611793+0000", 2024-04-19T01:05:13.023 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7408e-05 2024-04-19T01:05:13.024 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.024 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.024 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:13.024 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611808+0000", 2024-04-19T01:05:13.024 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5259000000000001e-05 2024-04-19T01:05:13.024 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.024 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.024 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:13.024 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.623039+0000", 2024-04-19T01:05:13.024 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011230756 2024-04-19T01:05:13.024 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.024 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.024 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:05:13.024 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.623089+0000", 2024-04-19T01:05:13.025 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0078999999999999e-05 2024-04-19T01:05:13.025 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.025 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.025 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:13.025 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.632385+0000", 2024-04-19T01:05:13.025 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0092956649999999998 2024-04-19T01:05:13.025 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.025 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.025 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:13.025 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.632471+0000", 2024-04-19T01:05:13.025 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.6693999999999994e-05 2024-04-19T01:05:13.025 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.025 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.025 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:13.026 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.633454+0000", 2024-04-19T01:05:13.026 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00098249899999999996 2024-04-19T01:05:13.026 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:13.026 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:13.026 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:13.026 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.026 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.026 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:345 3.fs0 3:f23900a8:::smithi15528762-6:head [append 0~557056] snapc 9=[8,7] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:05:13.026 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.534632+0000", 2024-04-19T01:05:13.026 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 54.437850058999999, 2024-04-19T01:05:13.026 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.044819185999999997, 2024-04-19T01:05:13.026 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:13.026 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:13.026 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:13.026 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:13.027 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:13.027 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:13.027 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 345 2024-04-19T01:05:13.027 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.027 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:13.027 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.027 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:13.027 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.534632+0000", 2024-04-19T01:05:13.027 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:13.027 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.027 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.027 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:13.027 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.534632+0000", 2024-04-19T01:05:13.027 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:13.027 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.028 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.028 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:13.028 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.534634+0000", 2024-04-19T01:05:13.028 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4099999999999998e-06 2024-04-19T01:05:13.028 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.028 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.028 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:13.028 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.535690+0000", 2024-04-19T01:05:13.028 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010557260000000001 2024-04-19T01:05:13.028 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.028 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.028 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:13.028 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.535710+0000", 2024-04-19T01:05:13.028 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0339e-05 2024-04-19T01:05:13.028 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.029 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.029 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:13.029 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.535720+0000", 2024-04-19T01:05:13.029 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7599999999999997e-06 2024-04-19T01:05:13.029 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.029 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.029 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:13.029 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.535817+0000", 2024-04-19T01:05:13.029 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7218000000000003e-05 2024-04-19T01:05:13.029 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.029 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.029 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:13.029 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.536037+0000", 2024-04-19T01:05:13.030 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00021956699999999999 2024-04-19T01:05:13.030 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.030 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.030 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:13.030 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.537439+0000", 2024-04-19T01:05:13.030 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014022240000000001 2024-04-19T01:05:13.030 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.030 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.030 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:13.030 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.557535+0000", 2024-04-19T01:05:13.030 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.020095874999999999 2024-04-19T01:05:13.030 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.030 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.030 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:13.031 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.579388+0000", 2024-04-19T01:05:13.031 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021852772999999999 2024-04-19T01:05:13.031 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.031 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.031 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:13.031 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.579451+0000", 2024-04-19T01:05:13.031 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3293999999999994e-05 2024-04-19T01:05:13.031 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:13.031 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:13.031 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:13.031 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.031 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.031 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:346 3.fs0 3:f23900a8:::smithi15528762-6:head [append 0~671744] snapc 9=[8,7] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:05:13.033 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.540969+0000", 2024-04-19T01:05:13.033 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 54.431512531000003, 2024-04-19T01:05:13.033 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.039266644000000003, 2024-04-19T01:05:13.033 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:13.033 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:13.033 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:13.033 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:13.033 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:13.033 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:13.033 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 346 2024-04-19T01:05:13.034 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.034 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:13.034 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.034 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:13.034 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.540969+0000", 2024-04-19T01:05:13.034 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:13.034 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.034 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.034 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:13.034 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.540969+0000", 2024-04-19T01:05:13.034 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:13.034 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.034 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.035 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:13.035 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.540971+0000", 2024-04-19T01:05:13.035 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9039999999999999e-06 2024-04-19T01:05:13.035 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.035 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.035 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:13.035 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542173+0000", 2024-04-19T01:05:13.035 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00120192 2024-04-19T01:05:13.035 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.035 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.035 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:13.035 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542196+0000", 2024-04-19T01:05:13.035 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3159e-05 2024-04-19T01:05:13.036 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.036 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.036 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:13.036 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542210+0000", 2024-04-19T01:05:13.036 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3899e-05 2024-04-19T01:05:13.036 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.036 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.036 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:13.036 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.546462+0000", 2024-04-19T01:05:13.036 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0042514629999999996 2024-04-19T01:05:13.036 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.036 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.036 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:13.036 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.546596+0000", 2024-04-19T01:05:13.036 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013463900000000001 2024-04-19T01:05:13.037 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.037 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.037 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:13.037 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.547898+0000", 2024-04-19T01:05:13.037 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0013016239999999999 2024-04-19T01:05:13.037 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.037 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.037 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:13.037 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.558500+0000", 2024-04-19T01:05:13.037 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010601928 2024-04-19T01:05:13.037 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.037 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.037 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:13.037 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.580189+0000", 2024-04-19T01:05:13.038 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021688796 2024-04-19T01:05:13.038 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.038 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.038 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:13.038 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.580236+0000", 2024-04-19T01:05:13.038 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7311999999999997e-05 2024-04-19T01:05:13.038 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:13.038 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:13.038 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:13.038 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.038 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.039 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:347 3.fs0 3:f23900a8:::smithi15528762-6:head [append 0~98304] snapc 9=[8,7] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:05:13.039 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.542237+0000", 2024-04-19T01:05:13.039 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 54.430244973999997, 2024-04-19T01:05:13.039 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.038983274999999998, 2024-04-19T01:05:13.039 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:13.039 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:13.039 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:13.039 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:13.039 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:13.039 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:13.039 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 347 2024-04-19T01:05:13.039 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.039 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:13.039 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.039 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:13.040 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542237+0000", 2024-04-19T01:05:13.040 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:13.040 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.040 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.040 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:13.040 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542237+0000", 2024-04-19T01:05:13.040 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:13.040 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.040 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.040 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:13.040 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542238+0000", 2024-04-19T01:05:13.040 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3659999999999999e-06 2024-04-19T01:05:13.040 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.040 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.040 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:13.041 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542318+0000", 2024-04-19T01:05:13.041 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.0158000000000001e-05 2024-04-19T01:05:13.041 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.041 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.041 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:13.041 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542336+0000", 2024-04-19T01:05:13.041 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7836999999999999e-05 2024-04-19T01:05:13.041 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.041 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.041 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:13.041 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542348+0000", 2024-04-19T01:05:13.041 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1646e-05 2024-04-19T01:05:13.041 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.041 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.041 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:13.042 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.553869+0000", 2024-04-19T01:05:13.042 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011521164 2024-04-19T01:05:13.042 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.042 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.042 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:13.042 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.554020+0000", 2024-04-19T01:05:13.042 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015064200000000001 2024-04-19T01:05:13.042 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.042 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.042 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:13.042 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.554913+0000", 2024-04-19T01:05:13.042 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00089314099999999997 2024-04-19T01:05:13.042 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.042 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.042 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:13.043 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.558556+0000", 2024-04-19T01:05:13.043 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003642803 2024-04-19T01:05:13.043 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.043 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.043 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:13.043 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.581179+0000", 2024-04-19T01:05:13.043 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.022623306999999999 2024-04-19T01:05:13.043 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.043 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.043 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:13.043 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.581220+0000", 2024-04-19T01:05:13.043 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1211000000000002e-05 2024-04-19T01:05:13.043 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:13.043 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:13.043 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:13.044 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.044 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.044 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:348 3.fs0 3:f23900a8:::smithi15528762-6:head [setxattr _header (55) in=7b] snapc 9=[8,7] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:05:13.044 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.542369+0000", 2024-04-19T01:05:13.044 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 54.430113114000001, 2024-04-19T01:05:13.044 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.039646576000000003, 2024-04-19T01:05:13.044 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:13.044 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:13.044 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:13.044 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:13.044 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:13.044 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:13.044 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 348 2024-04-19T01:05:13.044 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.044 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:13.045 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.045 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:13.045 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542369+0000", 2024-04-19T01:05:13.045 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:13.045 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.045 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.045 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:13.045 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542369+0000", 2024-04-19T01:05:13.045 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:13.045 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.045 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.045 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:13.045 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542370+0000", 2024-04-19T01:05:13.045 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.384e-06 2024-04-19T01:05:13.045 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.046 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.046 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:13.046 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542376+0000", 2024-04-19T01:05:13.046 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.806e-06 2024-04-19T01:05:13.046 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.046 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.046 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:13.046 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542400+0000", 2024-04-19T01:05:13.046 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4218000000000001e-05 2024-04-19T01:05:13.046 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.046 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.046 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:13.046 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542411+0000", 2024-04-19T01:05:13.046 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0801999999999999e-05 2024-04-19T01:05:13.046 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.047 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.047 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:13.047 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.557611+0000", 2024-04-19T01:05:13.047 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015199724 2024-04-19T01:05:13.047 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.047 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.047 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:13.047 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.557731+0000", 2024-04-19T01:05:13.047 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012045900000000001 2024-04-19T01:05:13.047 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.047 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.047 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:13.047 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.558229+0000", 2024-04-19T01:05:13.047 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00049771099999999997 2024-04-19T01:05:13.048 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.048 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.048 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:13.048 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.559080+0000", 2024-04-19T01:05:13.048 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00085117400000000001 2024-04-19T01:05:13.048 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.048 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.048 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:13.048 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.581912+0000", 2024-04-19T01:05:13.048 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.022832086000000001 2024-04-19T01:05:13.048 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.048 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.048 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:13.048 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.582015+0000", 2024-04-19T01:05:13.048 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000103212 2024-04-19T01:05:13.049 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:13.049 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:13.049 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:13.049 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.049 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.049 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:349 3.fs0 3:f23900a8:::smithi15528762-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:05:13.049 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.542427+0000", 2024-04-19T01:05:13.049 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 54.430054943000002, 2024-04-19T01:05:13.049 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.041128310000000001, 2024-04-19T01:05:13.049 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:13.049 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:13.049 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:05:13.049 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:13.049 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:13.049 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:13.050 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 349 2024-04-19T01:05:13.050 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.050 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:13.050 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.050 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:13.050 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542427+0000", 2024-04-19T01:05:13.050 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:13.050 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.050 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.050 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:13.050 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542427+0000", 2024-04-19T01:05:13.050 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:13.050 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.050 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.050 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:13.051 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542428+0000", 2024-04-19T01:05:13.051 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.4e-07 2024-04-19T01:05:13.051 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.051 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.051 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:13.051 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542432+0000", 2024-04-19T01:05:13.051 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5499999999999996e-06 2024-04-19T01:05:13.051 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.051 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.051 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:13.051 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542446+0000", 2024-04-19T01:05:13.051 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3365e-05 2024-04-19T01:05:13.051 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.051 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.051 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:13.052 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542456+0000", 2024-04-19T01:05:13.052 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.029e-05 2024-04-19T01:05:13.052 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.052 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.052 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:13.052 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.558587+0000", 2024-04-19T01:05:13.052 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.016131106999999999 2024-04-19T01:05:13.052 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.052 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.052 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:05:13.053 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.558676+0000", 2024-04-19T01:05:13.053 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.8823e-05 2024-04-19T01:05:13.053 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.053 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.053 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:13.053 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.582441+0000", 2024-04-19T01:05:13.053 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.023764864 2024-04-19T01:05:13.053 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.053 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.053 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:13.053 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.582493+0000", 2024-04-19T01:05:13.054 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2130999999999999e-05 2024-04-19T01:05:13.054 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.054 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.054 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:13.054 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.583555+0000", 2024-04-19T01:05:13.054 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010623399999999999 2024-04-19T01:05:13.054 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:13.054 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:13.054 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:13.054 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.054 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.054 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:660 3.fs0 3:f23900a8:::smithi15528762-6:head [append 0~786432] snapc 1d=[1d,1c,1b,14] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-04-19T01:05:13.055 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:56.630857+0000", 2024-04-19T01:05:13.055 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 16.341624612, 2024-04-19T01:05:13.055 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.038376939999999998, 2024-04-19T01:05:13.055 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:13.055 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:13.055 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:13.055 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:13.055 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:13.055 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:13.055 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 660 2024-04-19T01:05:13.055 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.055 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:13.055 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.055 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:13.055 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.630857+0000", 2024-04-19T01:05:13.056 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:13.056 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.056 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.056 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:13.056 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.630857+0000", 2024-04-19T01:05:13.056 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:13.056 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.056 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.056 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:13.056 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.630859+0000", 2024-04-19T01:05:13.056 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0719999999999998e-06 2024-04-19T01:05:13.056 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.056 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.056 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:13.056 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.631959+0000", 2024-04-19T01:05:13.057 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010997380000000001 2024-04-19T01:05:13.057 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.057 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.057 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:13.057 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.631981+0000", 2024-04-19T01:05:13.057 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.2354999999999999e-05 2024-04-19T01:05:13.057 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.057 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.057 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:13.057 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.631994+0000", 2024-04-19T01:05:13.057 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2541000000000001e-05 2024-04-19T01:05:13.057 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.057 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.057 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:13.058 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.632126+0000", 2024-04-19T01:05:13.058 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000132037 2024-04-19T01:05:13.058 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.058 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.058 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:13.058 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.632292+0000", 2024-04-19T01:05:13.058 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000165635 2024-04-19T01:05:13.058 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.058 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.058 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:13.058 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.634219+0000", 2024-04-19T01:05:13.058 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0019270800000000001 2024-04-19T01:05:13.058 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.058 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.059 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:13.059 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.658365+0000", 2024-04-19T01:05:13.059 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.024145811 2024-04-19T01:05:13.059 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.059 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.059 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:13.059 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.669104+0000", 2024-04-19T01:05:13.059 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010739395000000001 2024-04-19T01:05:13.059 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.059 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.059 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:13.059 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.669234+0000", 2024-04-19T01:05:13.059 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000130276 2024-04-19T01:05:13.059 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:13.059 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:13.060 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:13.060 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.060 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.060 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:661 3.fs0 3:f23900a8:::smithi15528762-6:head [append 0~458752] snapc 1d=[1d,1c,1b,14] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-04-19T01:05:13.060 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:56.634770+0000", 2024-04-19T01:05:13.060 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 16.337711518999999, 2024-04-19T01:05:13.060 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.037192913000000001, 2024-04-19T01:05:13.060 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:13.060 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:13.060 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:13.060 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:13.060 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:13.060 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:13.060 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 661 2024-04-19T01:05:13.061 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.061 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:13.061 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.061 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:13.061 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.634770+0000", 2024-04-19T01:05:13.061 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:13.061 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.061 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.061 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:13.061 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.634770+0000", 2024-04-19T01:05:13.061 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:13.061 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.061 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.061 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:13.062 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.634772+0000", 2024-04-19T01:05:13.062 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8360000000000001e-06 2024-04-19T01:05:13.062 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.062 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.062 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:13.062 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.635445+0000", 2024-04-19T01:05:13.062 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00067238900000000002 2024-04-19T01:05:13.062 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.062 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.062 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:13.062 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.635478+0000", 2024-04-19T01:05:13.062 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.3878000000000003e-05 2024-04-19T01:05:13.062 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.062 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.063 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:13.063 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.635488+0000", 2024-04-19T01:05:13.063 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.8020000000000003e-06 2024-04-19T01:05:13.063 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.063 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.063 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:13.063 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.648319+0000", 2024-04-19T01:05:13.063 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012830879 2024-04-19T01:05:13.063 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.063 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.063 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:13.063 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.648638+0000", 2024-04-19T01:05:13.063 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00031897300000000001 2024-04-19T01:05:13.063 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.064 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.064 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:13.064 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.650056+0000", 2024-04-19T01:05:13.064 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014178319999999999 2024-04-19T01:05:13.064 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.064 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.064 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:13.064 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.664923+0000", 2024-04-19T01:05:13.064 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.01486728 2024-04-19T01:05:13.064 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.064 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.064 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:13.064 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.671903+0000", 2024-04-19T01:05:13.064 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0069797399999999999 2024-04-19T01:05:13.064 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.065 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.065 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:13.065 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.671963+0000", 2024-04-19T01:05:13.065 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0303999999999997e-05 2024-04-19T01:05:13.065 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:13.065 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:13.065 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:13.065 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.065 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.066 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:662 3.fs0 3:f23900a8:::smithi15528762-6:head [append 0~442368] snapc 1d=[1d,1c,1b,14] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-04-19T01:05:13.066 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:56.637960+0000", 2024-04-19T01:05:13.066 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 16.334522278000001, 2024-04-19T01:05:13.066 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.040497497, 2024-04-19T01:05:13.066 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:13.066 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:13.066 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:13.066 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:13.066 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:13.066 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:13.066 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 662 2024-04-19T01:05:13.066 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.066 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:13.066 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.067 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:13.067 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.637960+0000", 2024-04-19T01:05:13.067 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:13.067 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.067 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.067 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:13.067 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.637960+0000", 2024-04-19T01:05:13.067 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:13.067 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.067 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.067 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:13.067 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.637961+0000", 2024-04-19T01:05:13.067 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.48e-06 2024-04-19T01:05:13.068 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.068 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.068 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:13.068 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.638402+0000", 2024-04-19T01:05:13.068 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000440698 2024-04-19T01:05:13.068 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.068 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.068 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:13.068 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.638421+0000", 2024-04-19T01:05:13.068 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9182999999999999e-05 2024-04-19T01:05:13.068 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.068 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.068 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:13.068 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.638430+0000", 2024-04-19T01:05:13.068 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.7749999999999998e-06 2024-04-19T01:05:13.069 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.069 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.069 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:13.069 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.658705+0000", 2024-04-19T01:05:13.069 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.020275472999999999 2024-04-19T01:05:13.069 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.069 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.069 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:13.069 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.658871+0000", 2024-04-19T01:05:13.069 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00016530900000000001 2024-04-19T01:05:13.069 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.069 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.069 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:13.069 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.660103+0000", 2024-04-19T01:05:13.069 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001232885 2024-04-19T01:05:13.070 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.070 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.070 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:13.070 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.669256+0000", 2024-04-19T01:05:13.070 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0091529620000000006 2024-04-19T01:05:13.070 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.070 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.070 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:13.070 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.678392+0000", 2024-04-19T01:05:13.070 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0091357639999999993 2024-04-19T01:05:13.070 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.070 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.070 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:13.070 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.678457+0000", 2024-04-19T01:05:13.071 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4968000000000005e-05 2024-04-19T01:05:13.071 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:13.071 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:13.071 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:13.071 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.071 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.071 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:663 3.fs0 3:f23900a8:::smithi15528762-6:head [append 0~262144] snapc 1d=[1d,1c,1b,14] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-04-19T01:05:13.071 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:56.639883+0000", 2024-04-19T01:05:13.071 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 16.332599118000001, 2024-04-19T01:05:13.071 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.038981265000000001, 2024-04-19T01:05:13.071 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:13.071 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:13.071 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:13.071 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:13.071 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:13.072 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:13.072 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 663 2024-04-19T01:05:13.072 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.072 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:13.072 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.072 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:13.072 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.639883+0000", 2024-04-19T01:05:13.072 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:13.072 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.072 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.072 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:13.072 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.639883+0000", 2024-04-19T01:05:13.073 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:13.073 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.073 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.073 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:13.073 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.639884+0000", 2024-04-19T01:05:13.073 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6020000000000001e-06 2024-04-19T01:05:13.073 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.073 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.073 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:13.073 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.640138+0000", 2024-04-19T01:05:13.073 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00025372399999999999 2024-04-19T01:05:13.073 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.073 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.073 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:13.074 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.640159+0000", 2024-04-19T01:05:13.074 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0879000000000001e-05 2024-04-19T01:05:13.074 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.074 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.074 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:13.074 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.640170+0000", 2024-04-19T01:05:13.074 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0702e-05 2024-04-19T01:05:13.074 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.074 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.074 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:13.074 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.664988+0000", 2024-04-19T01:05:13.074 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.024818468999999999 2024-04-19T01:05:13.074 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.074 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.075 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:13.075 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.665133+0000", 2024-04-19T01:05:13.075 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014525599999999999 2024-04-19T01:05:13.075 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.075 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.075 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:13.075 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.666115+0000", 2024-04-19T01:05:13.075 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00098171099999999991 2024-04-19T01:05:13.075 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.075 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.075 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:13.075 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.670349+0000", 2024-04-19T01:05:13.075 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0042341560000000002 2024-04-19T01:05:13.075 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.076 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.076 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:13.076 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.678680+0000", 2024-04-19T01:05:13.076 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0083308489999999995 2024-04-19T01:05:13.076 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.076 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.077 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:13.077 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.678864+0000", 2024-04-19T01:05:13.077 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00018391699999999999 2024-04-19T01:05:13.077 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:13.077 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:13.077 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:13.077 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.077 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.077 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:664 3.fs0 3:f23900a8:::smithi15528762-6:head [setxattr _header (56) in=7b] snapc 1d=[1d,1c,1b,14] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-04-19T01:05:13.077 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:56.640187+0000", 2024-04-19T01:05:13.078 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 16.332294512000001, 2024-04-19T01:05:13.078 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.038910425999999998, 2024-04-19T01:05:13.078 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:13.078 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:13.078 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:13.078 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:13.078 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:13.078 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:13.078 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 664 2024-04-19T01:05:13.078 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.078 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:13.078 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.078 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:13.078 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.640187+0000", 2024-04-19T01:05:13.079 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:13.079 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.079 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.079 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:13.079 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.640187+0000", 2024-04-19T01:05:13.079 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:13.079 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.080 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.080 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:13.080 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.640188+0000", 2024-04-19T01:05:13.080 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.2799999999999995e-07 2024-04-19T01:05:13.080 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.080 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.080 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:13.080 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.640193+0000", 2024-04-19T01:05:13.080 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3599999999999998e-06 2024-04-19T01:05:13.081 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.081 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.081 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:13.081 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.640200+0000", 2024-04-19T01:05:13.081 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7989999999999994e-06 2024-04-19T01:05:13.081 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.081 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.081 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:13.081 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.640206+0000", 2024-04-19T01:05:13.081 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6339999999999998e-06 2024-04-19T01:05:13.081 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.081 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.081 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:13.081 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.669495+0000", 2024-04-19T01:05:13.081 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.029289440999999999 2024-04-19T01:05:13.082 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.082 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.082 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:13.082 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.669587+0000", 2024-04-19T01:05:13.082 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.1848999999999994e-05 2024-04-19T01:05:13.082 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.082 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.082 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:13.082 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.670063+0000", 2024-04-19T01:05:13.082 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00047547900000000001 2024-04-19T01:05:13.082 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.082 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.082 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:13.082 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.672096+0000", 2024-04-19T01:05:13.083 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0020328740000000001 2024-04-19T01:05:13.083 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.083 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.083 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:13.083 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.679025+0000", 2024-04-19T01:05:13.083 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.006929742 2024-04-19T01:05:13.083 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.083 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.083 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:13.083 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.679098+0000", 2024-04-19T01:05:13.083 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2420000000000001e-05 2024-04-19T01:05:13.083 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:13.083 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:13.083 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:13.083 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.084 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.084 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:665 3.fs0 3:f23900a8:::smithi15528762-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e86)", 2024-04-19T01:05:13.084 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:56.640214+0000", 2024-04-19T01:05:13.084 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 16.332268051, 2024-04-19T01:05:13.084 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.042546254999999998, 2024-04-19T01:05:13.084 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:13.084 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:13.084 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:05:13.084 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:13.084 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:13.084 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:13.084 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 665 2024-04-19T01:05:13.084 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.084 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:13.085 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.085 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:13.085 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.640214+0000", 2024-04-19T01:05:13.085 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:13.085 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.085 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.085 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:13.085 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.640214+0000", 2024-04-19T01:05:13.085 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:13.086 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.086 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.086 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:13.086 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.640214+0000", 2024-04-19T01:05:13.086 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4799999999999999e-07 2024-04-19T01:05:13.086 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.086 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.087 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:13.087 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.640217+0000", 2024-04-19T01:05:13.087 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.334e-06 2024-04-19T01:05:13.087 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.087 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.087 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:13.087 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.640224+0000", 2024-04-19T01:05:13.087 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.0530000000000001e-06 2024-04-19T01:05:13.087 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.087 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.087 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:13.087 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.640229+0000", 2024-04-19T01:05:13.087 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.48e-06 2024-04-19T01:05:13.087 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.087 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.088 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:13.088 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.670412+0000", 2024-04-19T01:05:13.088 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.030183202999999999 2024-04-19T01:05:13.088 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.088 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.088 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:05:13.088 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.670542+0000", 2024-04-19T01:05:13.088 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012960099999999999 2024-04-19T01:05:13.088 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.088 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.088 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:13.088 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.679535+0000", 2024-04-19T01:05:13.088 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0089925720000000008 2024-04-19T01:05:13.088 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.089 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.089 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:13.089 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.679696+0000", 2024-04-19T01:05:13.089 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000161013 2024-04-19T01:05:13.089 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:13.089 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:13.089 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:13.089 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.682760+0000", 2024-04-19T01:05:13.089 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003064551 2024-04-19T01:05:13.089 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:13.090 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:13.090 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:13.090 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:13.090 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:13.090 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:05:13.092 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-19T01:05:13.092 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-04-19T01:05:13.093 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T01:05:13.093 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-04-19T01:05:13.093 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T01:05:13.093 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-04-19T01:05:13.094 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T01:05:13.094 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:05:13.089+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:05:13.101 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-04-19T01:05:13.102 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T01:05:13.119 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-04-19T01:05:13.120 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T01:05:13.140 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-04-19T01:05:13.141 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T01:05:13.161 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-04-19T01:05:13.162 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T01:05:13.187 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-04-19T01:05:13.187 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T01:05:13.193 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:05:13.189+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:05:13.203 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2024-04-19T01:05:13.203 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T01:05:13.213 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:13.213 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:13.225 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2024-04-19T01:05:13.225 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T01:05:13.227 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-19T01:05:13.248 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2024-04-19T01:05:13.248 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T01:05:13.271 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2024-04-19T01:05:13.272 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T01:05:13.290 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2024-04-19T01:05:13.291 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-19T01:05:13.293 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:05:13.289+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:05:13.316 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2024-04-19T01:05:13.316 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-19T01:05:13.338 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2024-04-19T01:05:13.338 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-19T01:05:13.361 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2024-04-19T01:05:13.361 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-19T01:05:13.384 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:13.385 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:13.396 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:05:13.393+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:05:13.399 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-19T01:05:13.498 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:05:13.496+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:05:13.561 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:13.562 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:13.563 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-19T01:05:13.577 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-19T01:05:13.599 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:05:13.593+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:05:13.645 INFO:tasks.rados.rados.0.smithi155.stdout:312: done (6 left) 2024-04-19T01:05:13.646 INFO:tasks.rados.rados.0.smithi155.stdout:313: done (5 left) 2024-04-19T01:05:13.646 INFO:tasks.rados.rados.0.smithi155.stdout:314: done (4 left) 2024-04-19T01:05:13.646 INFO:tasks.rados.rados.0.smithi155.stdout:315: done (3 left) 2024-04-19T01:05:13.646 INFO:tasks.rados.rados.0.smithi155.stdout:316: done (2 left) 2024-04-19T01:05:13.646 INFO:tasks.rados.rados.0.smithi155.stdout:317: done (1 left) 2024-04-19T01:05:13.646 INFO:tasks.rados.rados.0.smithi155.stdout:318: done (0 left) 2024-04-19T01:05:13.646 INFO:tasks.rados.rados.0.smithi155.stdout:319: read oid 33 snap -1 2024-04-19T01:05:13.646 INFO:tasks.rados.rados.0.smithi155.stdout:319: expect (ObjNum 0 snap 0 seq_num 0) 2024-04-19T01:05:13.646 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 29 current snap is 37 2024-04-19T01:05:13.646 INFO:tasks.rados.rados.0.smithi155.stdout:320: seq_num 129 ranges {475136=622592,1097728=655360,1753088=688128,2441216=212992} 2024-04-19T01:05:13.650 INFO:tasks.rados.rados.0.smithi155.stdout:320: writing smithi15528762-29 from 475136 to 1097728 tid 1 2024-04-19T01:05:13.654 INFO:tasks.rados.rados.0.smithi155.stdout:320: writing smithi15528762-29 from 1097728 to 1753088 tid 2 2024-04-19T01:05:13.659 INFO:tasks.rados.rados.0.smithi155.stdout:320: writing smithi15528762-29 from 1753088 to 2441216 tid 3 2024-04-19T01:05:13.663 INFO:tasks.rados.rados.0.smithi155.stdout:320: writing smithi15528762-29 from 2441216 to 2654208 tid 4 2024-04-19T01:05:13.664 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 17 current snap is 37 2024-04-19T01:05:13.664 INFO:tasks.rados.rados.0.smithi155.stdout:321: seq_num 130 ranges {802816=638976,1441792=671744,2113536=802816,2916352=81920} 2024-04-19T01:05:13.672 INFO:tasks.rados.rados.0.smithi155.stdout:321: writing smithi15528762-17 from 802816 to 1441792 tid 1 2024-04-19T01:05:13.677 INFO:tasks.rados.rados.0.smithi155.stdout:321: writing smithi15528762-17 from 1441792 to 2113536 tid 2 2024-04-19T01:05:13.684 INFO:tasks.rados.rados.0.smithi155.stdout:321: writing smithi15528762-17 from 2113536 to 2916352 tid 3 2024-04-19T01:05:13.684 INFO:tasks.rados.rados.0.smithi155.stdout:321: writing smithi15528762-17 from 2916352 to 2998272 tid 4 2024-04-19T01:05:13.684 INFO:tasks.rados.rados.0.smithi155.stdout:322: read oid 23 snap 34 2024-04-19T01:05:13.685 INFO:tasks.rados.rados.0.smithi155.stdout:322: expect (ObjNum 111 snap 26 seq_num 111) 2024-04-19T01:05:13.685 INFO:tasks.rados.rados.0.smithi155.stdout:323: snap_remove snap 32 2024-04-19T01:05:13.699 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:05:13.697+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:05:13.753 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:13.754 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:13.768 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-19T01:05:13.800 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:05:13.800+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:05:13.886 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:13.887 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:13.899 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-19T01:05:13.901 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:05:13.900+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:05:14.002 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:05:14.001+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:05:14.010 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:14.010 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:14.023 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-04-19T01:05:14.110 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:05:14.109+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:05:14.188 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:14.188 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:14.207 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-04-19T01:05:14.208 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:05:14.208+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:05:14.311 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:05:14.310+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:05:14.370 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:14.370 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:14.384 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-04-19T01:05:14.430 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:05:14.429+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:05:14.509 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:14.509 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:14.512 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:05:14.509+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:05:14.523 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-19T01:05:14.613 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:05:14.612+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:05:14.644 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2024-04-19T01:05:14.649 INFO:tasks.rados.rados.0.smithi155.stdout:323: done (4 left) 2024-04-19T01:05:14.649 INFO:tasks.rados.rados.0.smithi155.stdout:324: read oid 35 snap 34 2024-04-19T01:05:14.650 INFO:tasks.rados.rados.0.smithi155.stdout:324: expect (ObjNum 88 snap 15 seq_num 88) 2024-04-19T01:05:14.650 INFO:tasks.rados.rados.0.smithi155.stdout:319: expect (ObjNum 0 snap 0 seq_num 0) 2024-04-19T01:05:14.651 INFO:tasks.rados.rados.0.smithi155.stdout:320: finishing write tid 1 to smithi15528762-29 2024-04-19T01:05:14.651 INFO:tasks.rados.rados.0.smithi155.stdout:322: expect (ObjNum 111 snap 26 seq_num 111) 2024-04-19T01:05:14.656 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:14.656 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:14.679 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-19T01:05:14.718 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:05:14.716+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:05:14.733 INFO:tasks.rados.rados.0.smithi155.stdout:320: finishing write tid 2 to smithi15528762-29 2024-04-19T01:05:14.733 INFO:tasks.rados.rados.0.smithi155.stdout:320: finishing write tid 3 to smithi15528762-29 2024-04-19T01:05:14.733 INFO:tasks.rados.rados.0.smithi155.stdout:320: finishing write tid 4 to smithi15528762-29 2024-04-19T01:05:14.733 INFO:tasks.rados.rados.0.smithi155.stdout:320: finishing write tid 5 to smithi15528762-29 2024-04-19T01:05:14.733 INFO:tasks.rados.rados.0.smithi155.stdout:320: finishing write tid 6 to smithi15528762-29 2024-04-19T01:05:14.733 INFO:tasks.rados.rados.0.smithi155.stdout:320: oid 29 updating version 0 to 71 2024-04-19T01:05:14.733 INFO:tasks.rados.rados.0.smithi155.stdout:320: oid 29 updating version 71 to 74 2024-04-19T01:05:14.733 INFO:tasks.rados.rados.0.smithi155.stdout:320: oid 29 version 74 is already newer than 73 2024-04-19T01:05:14.733 INFO:tasks.rados.rados.0.smithi155.stdout:320: oid 29 version 74 is already newer than 72 2024-04-19T01:05:14.733 INFO:tasks.rados.rados.0.smithi155.stdout:320: oid 29 updating version 74 to 75 2024-04-19T01:05:14.734 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 29 v 75 (ObjNum 129 snap 37 seq_num 129) dirty exists 2024-04-19T01:05:14.734 INFO:tasks.rados.rados.0.smithi155.stdout:320: left oid 29 (ObjNum 129 snap 37 seq_num 129) 2024-04-19T01:05:14.734 INFO:tasks.rados.rados.0.smithi155.stdout:321: finishing write tid 1 to smithi15528762-17 2024-04-19T01:05:14.734 INFO:tasks.rados.rados.0.smithi155.stdout:321: finishing write tid 2 to smithi15528762-17 2024-04-19T01:05:14.734 INFO:tasks.rados.rados.0.smithi155.stdout:321: finishing write tid 3 to smithi15528762-17 2024-04-19T01:05:14.734 INFO:tasks.rados.rados.0.smithi155.stdout:321: finishing write tid 4 to smithi15528762-17 2024-04-19T01:05:14.734 INFO:tasks.rados.rados.0.smithi155.stdout:321: finishing write tid 5 to smithi15528762-17 2024-04-19T01:05:14.734 INFO:tasks.rados.rados.0.smithi155.stdout:321: finishing write tid 6 to smithi15528762-17 2024-04-19T01:05:14.734 INFO:tasks.rados.rados.0.smithi155.stdout:321: oid 17 updating version 0 to 187 2024-04-19T01:05:14.734 INFO:tasks.rados.rados.0.smithi155.stdout:321: oid 17 version 187 is already newer than 185 2024-04-19T01:05:14.734 INFO:tasks.rados.rados.0.smithi155.stdout:321: oid 17 version 187 is already newer than 183 2024-04-19T01:05:14.734 INFO:tasks.rados.rados.0.smithi155.stdout:321: oid 17 version 187 is already newer than 184 2024-04-19T01:05:14.734 INFO:tasks.rados.rados.0.smithi155.stdout:321: oid 17 version 187 is already newer than 186 2024-04-19T01:05:14.734 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 17 v 187 (ObjNum 130 snap 37 seq_num 130) dirty exists 2024-04-19T01:05:14.735 INFO:tasks.rados.rados.0.smithi155.stdout:321: left oid 17 (ObjNum 130 snap 37 seq_num 130) 2024-04-19T01:05:14.735 INFO:tasks.rados.rados.0.smithi155.stdout:324: expect (ObjNum 88 snap 15 seq_num 88) 2024-04-19T01:05:14.797 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:14.797 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:14.811 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-19T01:05:14.819 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:05:14.816+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:05:14.844 INFO:tasks.rados.rados.0.smithi155.stdout:319: done (4 left) 2024-04-19T01:05:14.844 INFO:tasks.rados.rados.0.smithi155.stdout:320: done (3 left) 2024-04-19T01:05:14.844 INFO:tasks.rados.rados.0.smithi155.stdout:321: done (2 left) 2024-04-19T01:05:14.844 INFO:tasks.rados.rados.0.smithi155.stdout:322: done (1 left) 2024-04-19T01:05:14.844 INFO:tasks.rados.rados.0.smithi155.stdout:324: done (0 left) 2024-04-19T01:05:14.844 INFO:tasks.rados.rados.0.smithi155.stdout:325: snap_remove snap 29 2024-04-19T01:05:14.928 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:14.929 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:14.929 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:05:14.922+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:05:14.941 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-19T01:05:15.023 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:05:15.021+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:05:15.058 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:15.058 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:15.075 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-19T01:05:15.125 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:05:15.125+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:05:15.189 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:15.189 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:15.201 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-19T01:05:15.226 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:05:15.228+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:05:15.309 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:15.309 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:15.323 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-19T01:05:15.329 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:05:15.325+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:05:15.387 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-19T01:05:15.429 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:05:15.426+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:05:15.442 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:15.442 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:15.455 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-19T01:05:15.530 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:05:15.526+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:05:15.548 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-04-19T01:05:15.549 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T01:05:15.571 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-04-19T01:05:15.572 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T01:05:15.573 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:15.573 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:15.587 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-19T01:05:15.593 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-04-19T01:05:15.594 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T01:05:15.618 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-04-19T01:05:15.618 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T01:05:15.632 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:05:15.632+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:05:15.638 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-04-19T01:05:15.638 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T01:05:15.657 INFO:tasks.rados.rados.0.smithi155.stdout:325: done (0 left) 2024-04-19T01:05:15.657 INFO:tasks.rados.rados.0.smithi155.stdout:326: copy_from oid 8 from oid 45 current snap is 37 2024-04-19T01:05:15.657 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 2 current snap is 37 2024-04-19T01:05:15.657 INFO:tasks.rados.rados.0.smithi155.stdout:327: seq_num 131 ranges {2932736=655360,3588096=425984,4014080=638976,4653056=491520} 2024-04-19T01:05:15.665 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-04-19T01:05:15.666 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T01:05:15.699 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-04-19T01:05:15.699 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T01:05:15.715 INFO:tasks.rados.rados.0.smithi155.stdout:327: writing smithi15528762-2 from 2932736 to 3588096 tid 1 2024-04-19T01:05:15.718 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:15.718 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:15.720 INFO:tasks.rados.rados.0.smithi155.stdout:327: writing smithi15528762-2 from 3588096 to 4014080 tid 2 2024-04-19T01:05:15.724 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-04-19T01:05:15.724 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T01:05:15.726 INFO:tasks.rados.rados.0.smithi155.stdout:327: writing smithi15528762-2 from 4014080 to 4653056 tid 3 2024-04-19T01:05:15.731 INFO:tasks.rados.rados.0.smithi155.stdout:327: writing smithi15528762-2 from 4653056 to 5144576 tid 4 2024-04-19T01:05:15.732 INFO:tasks.rados.rados.0.smithi155.stdout:328: snap_remove snap 36 2024-04-19T01:05:15.732 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:05:15.733+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:05:15.736 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-04-19T01:05:15.747 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2024-04-19T01:05:15.747 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T01:05:15.767 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2024-04-19T01:05:15.768 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T01:05:15.788 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2024-04-19T01:05:15.788 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T01:05:15.809 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2024-04-19T01:05:15.809 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T01:05:15.827 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2024-04-19T01:05:15.827 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-19T01:05:15.834 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:05:15.833+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:05:15.845 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2024-04-19T01:05:15.845 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-19T01:05:15.862 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [0] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-04-19T01:05:15.862 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-19T01:05:15.862 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-04-19T01:05:15.862 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 1000 0.05 2024-04-19T01:05:15.866 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2024-04-19T01:05:15.866 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-19T01:05:15.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:05:15.865+0000 7ff072a2d640 0 -- 172.21.15.33:0/3680005623 >> [v2:172.21.15.155:6807/741132602,v1:172.21.15.155:6810/741132602] conn(0x7ff06c224520 msgr2=0x7ff06c226910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-19T01:05:15.890 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2024-04-19T01:05:15.891 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-19T01:05:15.898 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:05:15.898 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [], 2024-04-19T01:05:15.898 INFO:teuthology.orchestra.run.smithi110.stdout: "num_ops": 0 2024-04-19T01:05:15.898 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:05:15.912 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-04-19T01:05:15.955 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:05:15.957+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:05:16.035 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:05:16.036+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:05:16.072 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:05:16.072 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [], 2024-04-19T01:05:16.072 INFO:teuthology.orchestra.run.smithi110.stdout: "complaint_time": 30, 2024-04-19T01:05:16.072 INFO:teuthology.orchestra.run.smithi110.stdout: "num_blocked_ops": 0 2024-04-19T01:05:16.072 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:05:16.086 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-04-19T01:05:16.136 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:05:16.138+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:05:16.180 INFO:teuthology.orchestra.run.smithi033.stdout:moved 0 / 130 (0%) 2024-04-19T01:05:16.181 INFO:teuthology.orchestra.run.smithi033.stdout:avg 8.66667 2024-04-19T01:05:16.181 INFO:teuthology.orchestra.run.smithi033.stdout:stddev 2.52102 -> 2.52102 (expected baseline 2.8441) 2024-04-19T01:05:16.181 INFO:teuthology.orchestra.run.smithi033.stdout:min osd.12 with 4 -> 4 pgs (0.461538 -> 0.461538 * mean) 2024-04-19T01:05:16.181 INFO:teuthology.orchestra.run.smithi033.stdout:max osd.11 with 15 -> 15 pgs (1.73077 -> 1.73077 * mean) 2024-04-19T01:05:16.181 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:05:16.181 INFO:teuthology.orchestra.run.smithi033.stdout:oload 1000 2024-04-19T01:05:16.181 INFO:teuthology.orchestra.run.smithi033.stdout:max_change 0.05 2024-04-19T01:05:16.181 INFO:teuthology.orchestra.run.smithi033.stdout:max_change_osds 4 2024-04-19T01:05:16.181 INFO:teuthology.orchestra.run.smithi033.stdout:average_utilization 0.0004 2024-04-19T01:05:16.181 INFO:teuthology.orchestra.run.smithi033.stdout:overload_utilization 0.0045 2024-04-19T01:05:16.181 INFO:teuthology.orchestra.run.smithi033.stderr:no change 2024-04-19T01:05:16.200 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-04-19T01:05:16.200 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 1000 0.05 2024-04-19T01:05:16.208 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:05:16.208 INFO:teuthology.orchestra.run.smithi110.stdout: "size": 20, 2024-04-19T01:05:16.208 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 600, 2024-04-19T01:05:16.209 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [ 2024-04-19T01:05:16.209 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.209 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:235 3.9s0 3:93d266e5:::smithi15528762-34:head [append 0~688128] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:05:16.209 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.857683+0000", 2024-04-19T01:05:16.209 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 62.350555405000001, 2024-04-19T01:05:16.209 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.044416514999999997, 2024-04-19T01:05:16.209 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:16.209 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:16.209 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:16.209 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:16.209 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:16.209 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:16.209 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 235 2024-04-19T01:05:16.209 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.210 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:16.210 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.210 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:16.210 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.857683+0000", 2024-04-19T01:05:16.210 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:16.210 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.210 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.210 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:16.210 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.857683+0000", 2024-04-19T01:05:16.210 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:16.210 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.210 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.211 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:16.211 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.857687+0000", 2024-04-19T01:05:16.211 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.1130000000000001e-06 2024-04-19T01:05:16.211 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.211 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.211 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:16.211 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.859118+0000", 2024-04-19T01:05:16.211 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001431114 2024-04-19T01:05:16.211 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.211 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.211 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:16.211 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.859150+0000", 2024-04-19T01:05:16.211 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.2589000000000003e-05 2024-04-19T01:05:16.211 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.212 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.212 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:16.212 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.860377+0000", 2024-04-19T01:05:16.212 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001226878 2024-04-19T01:05:16.212 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.212 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.212 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:16.212 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.866898+0000", 2024-04-19T01:05:16.212 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0065209250000000003 2024-04-19T01:05:16.212 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.212 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.212 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:16.212 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.867163+0000", 2024-04-19T01:05:16.213 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000264901 2024-04-19T01:05:16.213 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.213 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.213 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:16.213 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.869905+0000", 2024-04-19T01:05:16.213 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0027423550000000001 2024-04-19T01:05:16.213 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.213 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.213 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:16.213 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.897422+0000", 2024-04-19T01:05:16.213 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.027516445 2024-04-19T01:05:16.213 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.213 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.214 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:16.214 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.901877+0000", 2024-04-19T01:05:16.214 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.004454965 2024-04-19T01:05:16.214 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.214 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.214 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:16.214 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.902100+0000", 2024-04-19T01:05:16.214 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00022322999999999999 2024-04-19T01:05:16.214 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:16.214 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:16.214 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:16.214 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.215 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.215 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:238 3.9s0 3:93d266e5:::smithi15528762-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:05:16.215 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.861534+0000", 2024-04-19T01:05:16.215 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 62.346704641000002, 2024-04-19T01:05:16.215 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.046352598000000002, 2024-04-19T01:05:16.215 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:16.215 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:16.215 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:05:16.215 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:16.215 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:16.215 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:16.215 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 238 2024-04-19T01:05:16.215 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.215 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:16.216 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.216 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:16.216 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861534+0000", 2024-04-19T01:05:16.216 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:16.216 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.216 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.216 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:16.216 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861534+0000", 2024-04-19T01:05:16.216 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:16.216 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.216 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.216 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:16.216 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861536+0000", 2024-04-19T01:05:16.216 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.607e-06 2024-04-19T01:05:16.217 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.217 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.217 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:16.217 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861542+0000", 2024-04-19T01:05:16.217 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.1689999999999997e-06 2024-04-19T01:05:16.217 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.217 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.217 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:16.217 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861562+0000", 2024-04-19T01:05:16.217 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0043000000000001e-05 2024-04-19T01:05:16.217 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.217 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.217 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:16.218 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861882+0000", 2024-04-19T01:05:16.218 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000320398 2024-04-19T01:05:16.218 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.218 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.218 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:16.218 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.901266+0000", 2024-04-19T01:05:16.218 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.039383373999999999 2024-04-19T01:05:16.218 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.218 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.218 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting for rw locks", 2024-04-19T01:05:16.218 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.901474+0000", 2024-04-19T01:05:16.218 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00020846999999999999 2024-04-19T01:05:16.218 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.219 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.219 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:16.219 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.904551+0000", 2024-04-19T01:05:16.219 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0030766690000000002 2024-04-19T01:05:16.219 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.219 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.219 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:16.219 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.904806+0000", 2024-04-19T01:05:16.219 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00025481899999999999 2024-04-19T01:05:16.219 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.219 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.219 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:16.219 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.907887+0000", 2024-04-19T01:05:16.219 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0030810490000000002 2024-04-19T01:05:16.220 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:16.220 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:16.220 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:16.220 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.220 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.220 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:267 3.9s0 3:93d266e5:::smithi15528762-34:head [append 0~737280] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:05:16.220 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.013235+0000", 2024-04-19T01:05:16.220 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 62.19500352, 2024-04-19T01:05:16.220 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.043578804999999998, 2024-04-19T01:05:16.220 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:16.220 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:16.220 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:16.220 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:16.220 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:16.221 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:16.221 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 267 2024-04-19T01:05:16.221 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.221 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:16.221 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.221 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:16.221 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.013235+0000", 2024-04-19T01:05:16.221 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:16.221 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.221 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.221 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:16.221 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.013235+0000", 2024-04-19T01:05:16.221 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:16.221 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.222 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.222 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:16.222 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.013239+0000", 2024-04-19T01:05:16.222 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.6200000000000001e-06 2024-04-19T01:05:16.222 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.222 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.222 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:16.222 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.014549+0000", 2024-04-19T01:05:16.222 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001310075 2024-04-19T01:05:16.222 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.222 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.222 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:16.222 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.014585+0000", 2024-04-19T01:05:16.222 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.5494999999999998e-05 2024-04-19T01:05:16.223 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.223 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.223 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:16.223 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.014607+0000", 2024-04-19T01:05:16.223 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2371999999999999e-05 2024-04-19T01:05:16.223 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.223 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.223 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:16.223 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.014744+0000", 2024-04-19T01:05:16.223 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000136988 2024-04-19T01:05:16.223 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.223 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.224 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:16.224 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.015097+0000", 2024-04-19T01:05:16.224 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00035269400000000002 2024-04-19T01:05:16.224 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.224 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.224 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:16.224 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.020447+0000", 2024-04-19T01:05:16.224 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0053503739999999998 2024-04-19T01:05:16.224 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.224 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.224 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:16.224 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.047175+0000", 2024-04-19T01:05:16.224 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.026728372 2024-04-19T01:05:16.225 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.225 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.225 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:16.225 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.056595+0000", 2024-04-19T01:05:16.225 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0094197320000000001 2024-04-19T01:05:16.225 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.225 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.225 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:16.225 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.056814+0000", 2024-04-19T01:05:16.225 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00021908299999999999 2024-04-19T01:05:16.225 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:16.225 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:16.226 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:16.226 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.226 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.226 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:270 3.9s0 3:93d266e5:::smithi15528762-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:05:16.226 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.016503+0000", 2024-04-19T01:05:16.226 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 62.191735356000002, 2024-04-19T01:05:16.226 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.044494194000000001, 2024-04-19T01:05:16.226 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:16.226 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:16.226 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:05:16.226 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:16.226 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:16.227 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:16.227 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 270 2024-04-19T01:05:16.227 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.227 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:16.227 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.227 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:16.227 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016503+0000", 2024-04-19T01:05:16.227 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:16.227 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.227 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.227 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:16.227 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016503+0000", 2024-04-19T01:05:16.227 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:16.228 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.228 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.228 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:16.228 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016505+0000", 2024-04-19T01:05:16.228 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.2729999999999999e-06 2024-04-19T01:05:16.228 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.228 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.228 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:16.228 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016511+0000", 2024-04-19T01:05:16.228 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.7570000000000001e-06 2024-04-19T01:05:16.228 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.228 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.229 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:16.229 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016530+0000", 2024-04-19T01:05:16.229 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9565999999999999e-05 2024-04-19T01:05:16.229 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.229 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.229 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:16.229 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016545+0000", 2024-04-19T01:05:16.229 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.4963000000000001e-05 2024-04-19T01:05:16.229 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.229 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.229 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:16.229 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.049233+0000", 2024-04-19T01:05:16.230 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.032687815000000002 2024-04-19T01:05:16.230 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.230 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.230 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting for rw locks", 2024-04-19T01:05:16.230 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.049365+0000", 2024-04-19T01:05:16.230 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00013258399999999999 2024-04-19T01:05:16.230 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.230 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.230 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:16.230 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.058814+0000", 2024-04-19T01:05:16.230 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0094481080000000002 2024-04-19T01:05:16.230 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.230 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.231 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:16.231 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.059010+0000", 2024-04-19T01:05:16.231 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00019622699999999999 2024-04-19T01:05:16.231 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.231 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.231 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:16.231 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.060998+0000", 2024-04-19T01:05:16.231 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0019879009999999998 2024-04-19T01:05:16.231 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:16.231 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:16.231 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:16.231 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.231 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.231 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:280 3.9s0 3:98647342:::smithi15528762-43:head [append 0~753664] snapc 7=[7,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-19T01:05:16.232 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.869239+0000", 2024-04-19T01:05:16.232 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 61.338999487000002, 2024-04-19T01:05:16.232 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.062785076999999995, 2024-04-19T01:05:16.232 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:16.232 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:16.232 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:16.232 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:16.232 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:16.232 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:16.232 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 280 2024-04-19T01:05:16.232 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.232 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:16.233 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.233 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:16.233 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.869239+0000", 2024-04-19T01:05:16.233 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:16.233 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.233 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.233 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:16.233 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.869239+0000", 2024-04-19T01:05:16.233 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:16.233 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.233 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.233 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:16.233 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.869243+0000", 2024-04-19T01:05:16.233 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.444e-06 2024-04-19T01:05:16.234 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.234 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.234 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:16.234 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.870594+0000", 2024-04-19T01:05:16.234 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001351662 2024-04-19T01:05:16.234 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.234 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.234 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:16.234 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.870629+0000", 2024-04-19T01:05:16.234 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.4329000000000003e-05 2024-04-19T01:05:16.234 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.234 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.234 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:16.234 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.870647+0000", 2024-04-19T01:05:16.235 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7768999999999999e-05 2024-04-19T01:05:16.235 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.235 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.235 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:16.235 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.870806+0000", 2024-04-19T01:05:16.235 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015962300000000001 2024-04-19T01:05:16.235 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.235 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.235 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "op must wait for map", 2024-04-19T01:05:16.235 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.870819+0000", 2024-04-19T01:05:16.235 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.3266e-05 2024-04-19T01:05:16.235 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.235 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.235 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:16.236 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875741+0000", 2024-04-19T01:05:16.236 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0049211599999999999 2024-04-19T01:05:16.236 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.236 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.236 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:16.236 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.876018+0000", 2024-04-19T01:05:16.236 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000276944 2024-04-19T01:05:16.236 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.236 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.236 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:16.237 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.880141+0000", 2024-04-19T01:05:16.237 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0041231030000000004 2024-04-19T01:05:16.237 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.237 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.237 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:16.237 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.915254+0000", 2024-04-19T01:05:16.237 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.035113610000000003 2024-04-19T01:05:16.237 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.237 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.237 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:16.237 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.931834+0000", 2024-04-19T01:05:16.237 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.016580212 2024-04-19T01:05:16.237 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.238 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.238 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:16.238 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.932024+0000", 2024-04-19T01:05:16.238 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00018995500000000001 2024-04-19T01:05:16.238 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:16.238 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:16.238 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:16.238 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.238 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.238 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:281 3.9s0 3:98647342:::smithi15528762-43:head [append 0~376832] snapc 7=[7,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-19T01:05:16.238 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.872157+0000", 2024-04-19T01:05:16.238 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 61.336081864000001, 2024-04-19T01:05:16.239 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.061221235999999998, 2024-04-19T01:05:16.239 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:16.239 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:16.239 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:16.239 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:16.239 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:16.239 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:16.239 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 281 2024-04-19T01:05:16.239 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.239 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:16.239 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.239 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:16.239 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872157+0000", 2024-04-19T01:05:16.239 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:16.240 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.240 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.240 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:16.240 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872157+0000", 2024-04-19T01:05:16.240 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:16.240 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.240 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.240 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:16.240 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872159+0000", 2024-04-19T01:05:16.240 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5129999999999999e-06 2024-04-19T01:05:16.240 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.240 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.240 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:16.240 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872435+0000", 2024-04-19T01:05:16.241 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00027559500000000001 2024-04-19T01:05:16.241 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.241 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.241 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:16.241 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872470+0000", 2024-04-19T01:05:16.241 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.5170999999999999e-05 2024-04-19T01:05:16.241 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.241 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.241 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:16.241 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872488+0000", 2024-04-19T01:05:16.242 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.8136000000000001e-05 2024-04-19T01:05:16.242 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.242 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.242 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:16.242 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872639+0000", 2024-04-19T01:05:16.242 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015092 2024-04-19T01:05:16.242 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.242 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.242 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:05:16.242 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872658+0000", 2024-04-19T01:05:16.243 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.8882999999999999e-05 2024-04-19T01:05:16.243 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.243 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.243 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:16.243 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.903660+0000", 2024-04-19T01:05:16.243 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.031001846999999999 2024-04-19T01:05:16.243 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.243 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.243 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:16.243 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.904268+0000", 2024-04-19T01:05:16.243 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00060793999999999996 2024-04-19T01:05:16.243 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.243 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.243 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:16.244 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.906536+0000", 2024-04-19T01:05:16.244 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0022680180000000001 2024-04-19T01:05:16.244 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.244 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.244 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:16.244 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.918698+0000", 2024-04-19T01:05:16.244 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.012161852000000001 2024-04-19T01:05:16.244 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.244 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.244 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:16.244 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.933228+0000", 2024-04-19T01:05:16.244 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.014529781 2024-04-19T01:05:16.244 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.244 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.245 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:16.245 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.933378+0000", 2024-04-19T01:05:16.245 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015058000000000001 2024-04-19T01:05:16.245 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:16.245 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:16.245 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:16.245 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.245 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.245 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:282 3.9s0 3:98647342:::smithi15528762-43:head [setxattr _header (56) in=7b] snapc 7=[7,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-19T01:05:16.245 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.872536+0000", 2024-04-19T01:05:16.245 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 61.335702361000003, 2024-04-19T01:05:16.245 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.062122945999999998, 2024-04-19T01:05:16.245 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:16.245 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:16.246 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:16.246 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:16.246 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:16.246 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:16.246 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 282 2024-04-19T01:05:16.246 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.246 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:16.246 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.246 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:16.246 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872536+0000", 2024-04-19T01:05:16.246 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:16.246 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.246 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.246 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:16.247 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872536+0000", 2024-04-19T01:05:16.247 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:16.247 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.247 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.247 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:16.247 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872538+0000", 2024-04-19T01:05:16.247 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.423e-06 2024-04-19T01:05:16.247 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.247 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.247 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:16.247 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872548+0000", 2024-04-19T01:05:16.247 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.8430000000000005e-06 2024-04-19T01:05:16.247 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.247 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.248 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:16.248 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872569+0000", 2024-04-19T01:05:16.248 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1336e-05 2024-04-19T01:05:16.248 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.248 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.248 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:16.248 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872585+0000", 2024-04-19T01:05:16.248 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6288999999999999e-05 2024-04-19T01:05:16.248 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.248 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.248 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:16.248 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872783+0000", 2024-04-19T01:05:16.248 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000197488 2024-04-19T01:05:16.248 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.249 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.249 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:05:16.249 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872804+0000", 2024-04-19T01:05:16.249 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0839000000000002e-05 2024-04-19T01:05:16.249 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.249 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.249 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:16.249 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.915520+0000", 2024-04-19T01:05:16.249 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.042715887000000001 2024-04-19T01:05:16.249 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.249 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.249 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:16.249 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.915852+0000", 2024-04-19T01:05:16.250 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00033199899999999998 2024-04-19T01:05:16.250 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.250 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.250 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:16.250 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.916938+0000", 2024-04-19T01:05:16.250 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0010863940000000001 2024-04-19T01:05:16.250 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.250 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.250 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:16.250 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.918701+0000", 2024-04-19T01:05:16.250 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001763351 2024-04-19T01:05:16.250 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.250 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.250 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:16.251 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.934283+0000", 2024-04-19T01:05:16.251 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.015581526 2024-04-19T01:05:16.251 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.251 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.251 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:16.251 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.934659+0000", 2024-04-19T01:05:16.251 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00037657100000000002 2024-04-19T01:05:16.251 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:16.251 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:16.251 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:16.251 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.251 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.251 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:283 3.9s0 3:98647342:::smithi15528762-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-04-19T01:05:16.251 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.872609+0000", 2024-04-19T01:05:16.251 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 61.335629511, 2024-04-19T01:05:16.252 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.067525222999999995, 2024-04-19T01:05:16.252 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:16.252 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:16.252 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:05:16.252 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:16.252 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:16.252 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:16.252 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 283 2024-04-19T01:05:16.252 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.252 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:16.252 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.252 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:16.252 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872609+0000", 2024-04-19T01:05:16.253 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:16.253 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.253 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.253 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:16.253 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872609+0000", 2024-04-19T01:05:16.253 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:16.253 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.253 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.253 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:16.253 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872611+0000", 2024-04-19T01:05:16.253 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.367e-06 2024-04-19T01:05:16.253 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.253 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.253 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:16.253 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872617+0000", 2024-04-19T01:05:16.254 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.2400000000000004e-06 2024-04-19T01:05:16.254 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.254 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.254 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:16.254 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872636+0000", 2024-04-19T01:05:16.254 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.8941999999999999e-05 2024-04-19T01:05:16.254 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.254 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.254 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:16.254 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872652+0000", 2024-04-19T01:05:16.254 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5651999999999999e-05 2024-04-19T01:05:16.254 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.255 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.255 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:16.255 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872858+0000", 2024-04-19T01:05:16.255 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00020631100000000001 2024-04-19T01:05:16.255 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.255 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.255 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:05:16.255 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872867+0000", 2024-04-19T01:05:16.255 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.5400000000000001e-06 2024-04-19T01:05:16.255 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.255 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.255 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:16.255 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.917636+0000", 2024-04-19T01:05:16.255 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.044768372000000001 2024-04-19T01:05:16.256 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.256 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.256 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting for rw locks", 2024-04-19T01:05:16.256 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.917899+0000", 2024-04-19T01:05:16.256 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00026336799999999999 2024-04-19T01:05:16.256 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.256 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.256 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:16.256 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.935711+0000", 2024-04-19T01:05:16.256 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.017811862000000001 2024-04-19T01:05:16.256 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.256 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.257 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:16.257 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.935937+0000", 2024-04-19T01:05:16.257 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00022568800000000001 2024-04-19T01:05:16.257 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.257 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.257 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:16.257 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.940135+0000", 2024-04-19T01:05:16.257 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0041978809999999997 2024-04-19T01:05:16.257 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:16.257 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:16.257 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:16.257 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.257 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.257 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:653 3.es0 3:716a2d39:::smithi15528762-27:head [copy-from ver 38] snapc 1d=[1d,1c,1b,14] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-04-19T01:05:16.258 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:56.558085+0000", 2024-04-19T01:05:16.258 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 19.650154031, 2024-04-19T01:05:16.258 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.050389499999999997, 2024-04-19T01:05:16.258 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:16.258 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:16.258 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:16.258 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:16.258 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:16.258 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:16.258 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 653 2024-04-19T01:05:16.258 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.258 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:16.258 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.258 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:16.259 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.558085+0000", 2024-04-19T01:05:16.259 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:16.259 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.259 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.259 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:16.259 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.558085+0000", 2024-04-19T01:05:16.259 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:16.259 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.259 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.259 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:16.259 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.558089+0000", 2024-04-19T01:05:16.259 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.8190000000000002e-06 2024-04-19T01:05:16.259 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.260 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.260 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:16.260 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.558106+0000", 2024-04-19T01:05:16.260 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6883000000000001e-05 2024-04-19T01:05:16.260 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.260 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.260 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:16.260 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.558148+0000", 2024-04-19T01:05:16.260 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.2970999999999999e-05 2024-04-19T01:05:16.260 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.260 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.260 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:16.260 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.558171+0000", 2024-04-19T01:05:16.261 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2728999999999999e-05 2024-04-19T01:05:16.261 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.261 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.261 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:16.261 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.558355+0000", 2024-04-19T01:05:16.261 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00018391699999999999 2024-04-19T01:05:16.261 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.261 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.261 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:16.261 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.558621+0000", 2024-04-19T01:05:16.261 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00026582199999999998 2024-04-19T01:05:16.261 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.261 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.261 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:16.262 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.571924+0000", 2024-04-19T01:05:16.262 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.013303429 2024-04-19T01:05:16.262 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.262 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.262 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:16.262 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.598434+0000", 2024-04-19T01:05:16.262 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.026509805000000001 2024-04-19T01:05:16.262 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.262 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.262 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:16.262 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.608345+0000", 2024-04-19T01:05:16.262 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0099107599999999994 2024-04-19T01:05:16.262 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.262 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.263 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:16.263 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.608474+0000", 2024-04-19T01:05:16.263 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00012936499999999999 2024-04-19T01:05:16.263 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:16.263 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:16.263 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:16.263 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.263 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.263 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:691 3.9s0 3:9a4f6987:::smithi15528762-44:head [append 0~475136] snapc 20=[20,1f,1e,1c] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e92)", 2024-04-19T01:05:16.263 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:02.623135+0000", 2024-04-19T01:05:16.263 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 13.585103733, 2024-04-19T01:05:16.263 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.063498073000000002, 2024-04-19T01:05:16.263 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:16.264 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:16.264 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:16.264 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:16.264 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:16.264 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:16.264 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 691 2024-04-19T01:05:16.264 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.264 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:16.264 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.264 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:16.264 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.623135+0000", 2024-04-19T01:05:16.264 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:16.264 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.264 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.265 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:16.265 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.623135+0000", 2024-04-19T01:05:16.265 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:16.265 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.265 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.265 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:16.265 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.623137+0000", 2024-04-19T01:05:16.265 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2060000000000001e-06 2024-04-19T01:05:16.265 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.265 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.265 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:16.265 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.624004+0000", 2024-04-19T01:05:16.266 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00086699399999999995 2024-04-19T01:05:16.266 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.266 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.266 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:16.266 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.624031+0000", 2024-04-19T01:05:16.266 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6951999999999999e-05 2024-04-19T01:05:16.266 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.266 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.266 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:16.266 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.624057+0000", 2024-04-19T01:05:16.266 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5471999999999999e-05 2024-04-19T01:05:16.266 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.267 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.267 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:16.267 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.624250+0000", 2024-04-19T01:05:16.267 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00019280499999999999 2024-04-19T01:05:16.267 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.267 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.267 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "op must wait for map", 2024-04-19T01:05:16.267 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.624267+0000", 2024-04-19T01:05:16.267 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7734999999999999e-05 2024-04-19T01:05:16.267 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.267 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.267 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:16.268 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.628695+0000", 2024-04-19T01:05:16.268 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0044277780000000003 2024-04-19T01:05:16.268 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.268 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.268 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:16.268 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.629013+0000", 2024-04-19T01:05:16.268 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00031820500000000001 2024-04-19T01:05:16.268 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.268 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.268 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:16.268 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.631338+0000", 2024-04-19T01:05:16.268 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.002324951 2024-04-19T01:05:16.268 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.268 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.269 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:16.269 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.662889+0000", 2024-04-19T01:05:16.269 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.031550750000000002 2024-04-19T01:05:16.269 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.269 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.269 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:16.269 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.686483+0000", 2024-04-19T01:05:16.269 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.023594542 2024-04-19T01:05:16.269 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.269 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.269 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:16.269 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.686633+0000", 2024-04-19T01:05:16.270 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00014968300000000001 2024-04-19T01:05:16.270 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:16.270 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:16.270 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:16.270 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.270 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.270 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:692 3.9s0 3:9a4f6987:::smithi15528762-44:head [append 0~655360] snapc 20=[20,1f,1e,1c] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-04-19T01:05:16.270 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:02.626896+0000", 2024-04-19T01:05:16.270 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 13.581342939000001, 2024-04-19T01:05:16.270 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.065157396000000006, 2024-04-19T01:05:16.270 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:16.270 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:16.270 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:16.271 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:16.271 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:16.271 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:16.271 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 692 2024-04-19T01:05:16.271 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.271 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:16.271 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.271 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:16.271 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.626896+0000", 2024-04-19T01:05:16.271 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:16.271 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.271 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.271 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:16.272 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.626896+0000", 2024-04-19T01:05:16.272 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:16.272 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.272 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.272 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:16.272 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.626899+0000", 2024-04-19T01:05:16.272 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6840000000000001e-06 2024-04-19T01:05:16.272 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.272 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.272 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:16.272 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.628053+0000", 2024-04-19T01:05:16.272 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001153933 2024-04-19T01:05:16.273 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.273 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.273 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:16.273 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.628087+0000", 2024-04-19T01:05:16.273 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.4159000000000003e-05 2024-04-19T01:05:16.273 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.273 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.273 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:16.273 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.628111+0000", 2024-04-19T01:05:16.274 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4752e-05 2024-04-19T01:05:16.275 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.275 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.275 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:16.275 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.643061+0000", 2024-04-19T01:05:16.275 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.014949234 2024-04-19T01:05:16.275 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.275 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.275 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:16.275 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.643383+0000", 2024-04-19T01:05:16.275 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00032213800000000003 2024-04-19T01:05:16.275 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.275 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.275 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:16.276 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.645699+0000", 2024-04-19T01:05:16.276 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.002316041 2024-04-19T01:05:16.276 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.276 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.280 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:16.280 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.683741+0000", 2024-04-19T01:05:16.280 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.038042377000000002 2024-04-19T01:05:16.280 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.280 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.281 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:16.281 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.691942+0000", 2024-04-19T01:05:16.281 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0082007139999999996 2024-04-19T01:05:16.281 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.281 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.281 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:16.281 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.692053+0000", 2024-04-19T01:05:16.281 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000111364 2024-04-19T01:05:16.281 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:16.281 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:16.281 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:16.281 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.281 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.281 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:693 3.9s0 3:9a4f6987:::smithi15528762-44:head [append 0~622592] snapc 20=[20,1f,1e,1c] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-04-19T01:05:16.281 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:02.632540+0000", 2024-04-19T01:05:16.282 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 13.575698807, 2024-04-19T01:05:16.282 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.061723067999999999, 2024-04-19T01:05:16.282 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:16.282 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:16.282 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:16.282 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:16.282 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:16.282 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:16.282 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 693 2024-04-19T01:05:16.282 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.282 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:16.282 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.282 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:16.282 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.632540+0000", 2024-04-19T01:05:16.283 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:16.283 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.283 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.283 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:16.283 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.632540+0000", 2024-04-19T01:05:16.283 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:16.283 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.283 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.283 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:16.283 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.632542+0000", 2024-04-19T01:05:16.283 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.04e-06 2024-04-19T01:05:16.283 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.283 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.283 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:16.283 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.633600+0000", 2024-04-19T01:05:16.284 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0010575750000000001 2024-04-19T01:05:16.284 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.284 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.284 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:16.284 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.633632+0000", 2024-04-19T01:05:16.284 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.2199000000000001e-05 2024-04-19T01:05:16.284 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.284 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.284 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:16.284 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.633648+0000", 2024-04-19T01:05:16.285 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6019e-05 2024-04-19T01:05:16.285 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.285 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.285 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:16.285 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.663173+0000", 2024-04-19T01:05:16.285 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.029525526 2024-04-19T01:05:16.285 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.285 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.285 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:16.285 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.663337+0000", 2024-04-19T01:05:16.285 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00016387300000000001 2024-04-19T01:05:16.285 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.285 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.285 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:16.286 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.664809+0000", 2024-04-19T01:05:16.286 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001471633 2024-04-19T01:05:16.286 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.286 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.286 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:16.286 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.683991+0000", 2024-04-19T01:05:16.286 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.01918226 2024-04-19T01:05:16.286 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.286 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.286 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:16.286 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.694158+0000", 2024-04-19T01:05:16.286 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.010166735999999999 2024-04-19T01:05:16.286 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.286 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.287 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:16.287 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.694263+0000", 2024-04-19T01:05:16.287 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000105207 2024-04-19T01:05:16.287 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:16.287 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:16.287 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:16.287 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.287 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.288 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:694 3.9s0 3:9a4f6987:::smithi15528762-44:head [append 0~278528] snapc 20=[20,1f,1e,1c] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-04-19T01:05:16.288 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:02.635263+0000", 2024-04-19T01:05:16.288 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 13.572976164, 2024-04-19T01:05:16.288 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.059764666000000001, 2024-04-19T01:05:16.288 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:16.288 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:16.288 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:16.288 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:16.288 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:16.288 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:16.288 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 694 2024-04-19T01:05:16.288 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.288 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:16.288 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.289 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:16.289 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.635263+0000", 2024-04-19T01:05:16.289 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:16.289 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.289 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.289 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:16.289 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.635263+0000", 2024-04-19T01:05:16.289 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:16.289 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.289 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.289 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:16.289 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.635265+0000", 2024-04-19T01:05:16.289 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6129999999999998e-06 2024-04-19T01:05:16.289 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.289 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.290 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:16.290 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.635755+0000", 2024-04-19T01:05:16.290 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00048946700000000005 2024-04-19T01:05:16.290 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.290 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.290 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:16.290 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.635792+0000", 2024-04-19T01:05:16.290 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.6789000000000003e-05 2024-04-19T01:05:16.290 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.290 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.290 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:16.290 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.635810+0000", 2024-04-19T01:05:16.290 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7999000000000002e-05 2024-04-19T01:05:16.290 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.291 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.291 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:16.291 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.676857+0000", 2024-04-19T01:05:16.291 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.041047229999999997 2024-04-19T01:05:16.291 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.291 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.291 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:16.291 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.677021+0000", 2024-04-19T01:05:16.291 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000164339 2024-04-19T01:05:16.291 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.291 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.291 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:16.291 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.678298+0000", 2024-04-19T01:05:16.291 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001277048 2024-04-19T01:05:16.291 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.292 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.292 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:16.292 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.686286+0000", 2024-04-19T01:05:16.292 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0079883290000000006 2024-04-19T01:05:16.292 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.292 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.292 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:16.292 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.694934+0000", 2024-04-19T01:05:16.292 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0086471780000000002 2024-04-19T01:05:16.292 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.292 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.292 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:16.292 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.695027+0000", 2024-04-19T01:05:16.292 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.3673999999999998e-05 2024-04-19T01:05:16.292 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:16.293 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:16.293 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:16.293 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.293 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.293 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:695 3.9s0 3:9a4f6987:::smithi15528762-44:head [setxattr _header (57) in=7b] snapc 20=[20,1f,1e,1c] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-04-19T01:05:16.293 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:02.635844+0000", 2024-04-19T01:05:16.293 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 13.572394584, 2024-04-19T01:05:16.293 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.060048563999999999, 2024-04-19T01:05:16.293 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:16.293 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:16.293 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:16.293 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:16.293 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:16.293 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:16.294 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 695 2024-04-19T01:05:16.294 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.294 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:16.294 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.294 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:16.294 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.635844+0000", 2024-04-19T01:05:16.294 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:16.294 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.294 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.294 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:16.294 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.635844+0000", 2024-04-19T01:05:16.294 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:16.294 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.294 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.295 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:16.295 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.635846+0000", 2024-04-19T01:05:16.295 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0260000000000002e-06 2024-04-19T01:05:16.295 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.295 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.295 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:16.295 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.635855+0000", 2024-04-19T01:05:16.295 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.8200000000000003e-06 2024-04-19T01:05:16.296 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.296 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.296 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:16.296 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.635880+0000", 2024-04-19T01:05:16.296 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5386000000000001e-05 2024-04-19T01:05:16.296 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.296 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.296 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:16.296 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.635896+0000", 2024-04-19T01:05:16.296 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5702e-05 2024-04-19T01:05:16.296 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.296 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.296 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:16.296 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.684339+0000", 2024-04-19T01:05:16.296 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.048443084999999997 2024-04-19T01:05:16.297 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.297 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.297 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:16.297 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.684491+0000", 2024-04-19T01:05:16.297 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015182999999999999 2024-04-19T01:05:16.297 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.297 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.297 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:16.297 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.685330+0000", 2024-04-19T01:05:16.297 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00083926499999999995 2024-04-19T01:05:16.297 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.297 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.297 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:16.298 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.686992+0000", 2024-04-19T01:05:16.298 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0016613839999999999 2024-04-19T01:05:16.298 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.298 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.298 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:16.299 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.695762+0000", 2024-04-19T01:05:16.299 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0087698370000000008 2024-04-19T01:05:16.299 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.299 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.299 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:16.299 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.695893+0000", 2024-04-19T01:05:16.299 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00013122899999999999 2024-04-19T01:05:16.299 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:16.299 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:16.299 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:16.299 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.299 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.299 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:696 3.9s0 3:9a4f6987:::smithi15528762-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e92)", 2024-04-19T01:05:16.300 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:02.635912+0000", 2024-04-19T01:05:16.300 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 13.572326546999999, 2024-04-19T01:05:16.300 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.063685439999999996, 2024-04-19T01:05:16.300 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:16.300 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:16.300 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:05:16.300 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:16.300 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:16.300 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:16.300 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 696 2024-04-19T01:05:16.300 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.300 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:16.300 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.300 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:16.300 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.635912+0000", 2024-04-19T01:05:16.300 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:16.301 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.301 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.301 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:16.301 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.635912+0000", 2024-04-19T01:05:16.301 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:16.301 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.301 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.301 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:16.301 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.635913+0000", 2024-04-19T01:05:16.301 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.7000000000000003e-07 2024-04-19T01:05:16.301 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.301 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.301 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:16.301 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.635918+0000", 2024-04-19T01:05:16.301 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.6929999999999998e-06 2024-04-19T01:05:16.302 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.302 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.302 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:16.302 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.635929+0000", 2024-04-19T01:05:16.302 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.1436e-05 2024-04-19T01:05:16.302 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.302 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.302 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:16.302 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.635938+0000", 2024-04-19T01:05:16.302 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.5799999999999992e-06 2024-04-19T01:05:16.302 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.302 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.302 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:16.303 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.685913+0000", 2024-04-19T01:05:16.303 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.049975064999999999 2024-04-19T01:05:16.303 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.303 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.303 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting for rw locks", 2024-04-19T01:05:16.303 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.686062+0000", 2024-04-19T01:05:16.303 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000149233 2024-04-19T01:05:16.303 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.303 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.303 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:16.303 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.696480+0000", 2024-04-19T01:05:16.303 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.010417493 2024-04-19T01:05:16.303 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.303 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.303 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:16.304 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.696592+0000", 2024-04-19T01:05:16.304 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000112725 2024-04-19T01:05:16.304 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.304 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.304 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:16.304 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.699598+0000", 2024-04-19T01:05:16.304 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0030052450000000001 2024-04-19T01:05:16.304 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:16.304 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:16.304 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:16.304 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.304 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.304 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:701 3.es0 3:7dbf7e9d:::smithi15528762-19:head [copy-from ver 30] snapc 20=[20,1f,1e,1c] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-04-19T01:05:16.304 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:02.648461+0000", 2024-04-19T01:05:16.304 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 13.559778195, 2024-04-19T01:05:16.305 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.078755795000000003, 2024-04-19T01:05:16.305 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:16.305 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:16.305 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:16.305 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:16.305 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:16.305 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:16.305 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 701 2024-04-19T01:05:16.306 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.306 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:16.306 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.306 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:16.306 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.648461+0000", 2024-04-19T01:05:16.306 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:16.306 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.306 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.306 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:16.306 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.648461+0000", 2024-04-19T01:05:16.306 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:16.306 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.306 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.306 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:16.307 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.648464+0000", 2024-04-19T01:05:16.307 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.2969999999999999e-06 2024-04-19T01:05:16.307 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.307 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.307 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:16.307 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.648477+0000", 2024-04-19T01:05:16.307 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.345e-05 2024-04-19T01:05:16.307 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.307 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.307 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:16.307 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.648515+0000", 2024-04-19T01:05:16.307 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.7314999999999998e-05 2024-04-19T01:05:16.307 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.307 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.308 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:16.308 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.648536+0000", 2024-04-19T01:05:16.308 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0995000000000001e-05 2024-04-19T01:05:16.308 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.310 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.310 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:16.310 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.648698+0000", 2024-04-19T01:05:16.310 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000162509 2024-04-19T01:05:16.310 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.310 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.310 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:16.310 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.648869+0000", 2024-04-19T01:05:16.310 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00017036300000000001 2024-04-19T01:05:16.310 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.310 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.310 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:16.311 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.701127+0000", 2024-04-19T01:05:16.311 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.052258617 2024-04-19T01:05:16.311 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.311 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.311 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:16.311 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.726095+0000", 2024-04-19T01:05:16.311 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.024967335 2024-04-19T01:05:16.311 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.311 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.311 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:16.311 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.727001+0000", 2024-04-19T01:05:16.311 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00090626200000000002 2024-04-19T01:05:16.311 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.311 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.311 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:16.312 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.727216+0000", 2024-04-19T01:05:16.312 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00021565199999999999 2024-04-19T01:05:16.312 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:16.312 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:16.312 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:16.312 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.312 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.312 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:702 3.es0 3:7dbf7e9d:::smithi15528762-19:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e92)", 2024-04-19T01:05:16.312 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:02.648585+0000", 2024-04-19T01:05:16.312 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 13.559654283, 2024-04-19T01:05:16.312 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.076795295999999999, 2024-04-19T01:05:16.312 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:16.312 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:16.312 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:05:16.312 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:16.313 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:16.313 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:16.313 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 702 2024-04-19T01:05:16.313 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.313 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:16.313 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.313 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:16.313 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.648585+0000", 2024-04-19T01:05:16.313 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:16.313 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.313 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.313 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:16.313 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.648585+0000", 2024-04-19T01:05:16.313 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:16.313 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.314 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.314 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:16.314 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.648586+0000", 2024-04-19T01:05:16.314 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6360000000000001e-06 2024-04-19T01:05:16.314 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.314 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.314 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:16.314 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.648594+0000", 2024-04-19T01:05:16.314 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.5900000000000002e-06 2024-04-19T01:05:16.314 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.314 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.314 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:16.314 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.648615+0000", 2024-04-19T01:05:16.314 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1274999999999999e-05 2024-04-19T01:05:16.315 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.315 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.315 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:16.315 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.648632+0000", 2024-04-19T01:05:16.315 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6555999999999999e-05 2024-04-19T01:05:16.315 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.315 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.315 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:16.315 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.649195+0000", 2024-04-19T01:05:16.315 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00056332500000000005 2024-04-19T01:05:16.315 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.315 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.315 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting for blocked object", 2024-04-19T01:05:16.315 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.649276+0000", 2024-04-19T01:05:16.316 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.1063000000000006e-05 2024-04-19T01:05:16.316 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.316 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.316 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:16.316 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.724709+0000", 2024-04-19T01:05:16.316 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.075432568000000005 2024-04-19T01:05:16.316 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.316 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.316 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:16.317 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.725009+0000", 2024-04-19T01:05:16.317 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00030049999999999999 2024-04-19T01:05:16.317 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.317 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.317 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:16.317 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.725380+0000", 2024-04-19T01:05:16.317 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00037078300000000002 2024-04-19T01:05:16.317 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:16.317 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:16.317 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:16.317 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.317 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.317 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:718 3.9s0 3:93d266e5:::smithi15528762-34:1f [read 0~3637248,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e96)", 2024-04-19T01:05:16.317 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:06.525331+0000", 2024-04-19T01:05:16.317 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 9.6829081339999998, 2024-04-19T01:05:16.318 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.043542420999999998, 2024-04-19T01:05:16.318 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:16.318 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:16.318 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:05:16.318 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:16.318 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:16.318 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:16.318 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 718 2024-04-19T01:05:16.318 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.318 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:16.318 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.318 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:16.318 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.525331+0000", 2024-04-19T01:05:16.319 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:16.319 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.319 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.319 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:16.319 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.525331+0000", 2024-04-19T01:05:16.319 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:16.319 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.319 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.319 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:16.319 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.525333+0000", 2024-04-19T01:05:16.319 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.08e-06 2024-04-19T01:05:16.320 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.320 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.320 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:16.320 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.525349+0000", 2024-04-19T01:05:16.320 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6330000000000001e-05 2024-04-19T01:05:16.320 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.320 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.321 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:16.321 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.525362+0000", 2024-04-19T01:05:16.321 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.2697999999999999e-05 2024-04-19T01:05:16.322 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.322 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.322 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:16.322 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.525373+0000", 2024-04-19T01:05:16.322 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.1348999999999999e-05 2024-04-19T01:05:16.322 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.323 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.323 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:16.323 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.525454+0000", 2024-04-19T01:05:16.323 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.0939999999999994e-05 2024-04-19T01:05:16.323 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.323 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.323 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "op must wait for map", 2024-04-19T01:05:16.323 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.525463+0000", 2024-04-19T01:05:16.323 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.7250000000000003e-06 2024-04-19T01:05:16.323 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.323 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.323 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:16.323 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.528004+0000", 2024-04-19T01:05:16.323 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0025409289999999999 2024-04-19T01:05:16.324 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.324 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.324 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:16.324 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.528118+0000", 2024-04-19T01:05:16.324 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000114113 2024-04-19T01:05:16.324 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.324 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.324 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:16.324 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.568873+0000", 2024-04-19T01:05:16.324 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.040755257000000003 2024-04-19T01:05:16.324 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:16.324 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:16.324 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:16.324 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.324 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.325 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:719 3.9s0 3:93d266e5:::smithi15528762-34:1f [read 0~3637248,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e96)", 2024-04-19T01:05:16.325 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:06.527295+0000", 2024-04-19T01:05:16.325 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 9.6809438520000004, 2024-04-19T01:05:16.325 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.044122639999999998, 2024-04-19T01:05:16.325 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:16.325 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:16.325 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:05:16.325 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:16.325 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:16.325 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:16.325 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 719 2024-04-19T01:05:16.325 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.325 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:16.325 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.325 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:16.326 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.527295+0000", 2024-04-19T01:05:16.326 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:16.326 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.326 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.326 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:16.326 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.527295+0000", 2024-04-19T01:05:16.326 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:16.326 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.327 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.327 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:16.327 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.527296+0000", 2024-04-19T01:05:16.327 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.9999999999999995e-07 2024-04-19T01:05:16.327 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.327 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.327 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:16.327 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.528753+0000", 2024-04-19T01:05:16.327 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001457042 2024-04-19T01:05:16.327 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.327 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.327 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:16.327 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.528767+0000", 2024-04-19T01:05:16.327 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.3534e-05 2024-04-19T01:05:16.328 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.328 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.328 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:16.328 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.528775+0000", 2024-04-19T01:05:16.328 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.8939999999999999e-06 2024-04-19T01:05:16.328 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.328 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.328 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:16.328 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.528944+0000", 2024-04-19T01:05:16.328 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00016899699999999999 2024-04-19T01:05:16.328 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.328 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.328 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:16.328 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.529052+0000", 2024-04-19T01:05:16.328 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000107044 2024-04-19T01:05:16.329 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.329 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.329 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:16.329 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.571418+0000", 2024-04-19T01:05:16.329 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.042366129000000002 2024-04-19T01:05:16.329 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:16.329 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:16.329 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:16.329 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.329 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.329 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:720 3.9s0 3:93d266e5:::smithi15528762-34:1f [read 0~3637248,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e96)", 2024-04-19T01:05:16.329 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:06.528800+0000", 2024-04-19T01:05:16.329 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 9.6794384210000004, 2024-04-19T01:05:16.329 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.044470728000000001, 2024-04-19T01:05:16.329 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:16.329 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:16.330 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:05:16.330 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:16.330 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:16.330 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:16.330 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 720 2024-04-19T01:05:16.330 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.330 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:16.330 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.330 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:16.331 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.528800+0000", 2024-04-19T01:05:16.331 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:16.331 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.331 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.331 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:16.331 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.528800+0000", 2024-04-19T01:05:16.331 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:16.331 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.331 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.332 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:16.332 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.528801+0000", 2024-04-19T01:05:16.332 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.7499999999999999e-07 2024-04-19T01:05:16.332 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.332 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.332 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:16.332 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.528806+0000", 2024-04-19T01:05:16.332 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.9529999999999998e-06 2024-04-19T01:05:16.332 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.332 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.332 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:16.333 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.528815+0000", 2024-04-19T01:05:16.333 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.5350000000000004e-06 2024-04-19T01:05:16.333 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.333 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.333 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:16.333 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.528821+0000", 2024-04-19T01:05:16.333 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.6950000000000003e-06 2024-04-19T01:05:16.333 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.333 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.333 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:16.333 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.529427+0000", 2024-04-19T01:05:16.333 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00060544199999999998 2024-04-19T01:05:16.333 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.334 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.334 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:16.334 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.529519+0000", 2024-04-19T01:05:16.334 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.1864e-05 2024-04-19T01:05:16.334 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:16.334 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:16.334 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:16.334 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.573271+0000", 2024-04-19T01:05:16.334 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.043752463999999998 2024-04-19T01:05:16.334 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:16.334 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:16.334 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:16.334 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:16.334 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:16.334 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:05:16.337 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-19T01:05:16.337 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:05:16.336+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:05:16.437 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:05:16.434+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:05:16.468 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:05:16.469 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-04-19T01:05:16.469 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-04-19T01:05:16.469 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:05:16.482 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-19T01:05:16.510 INFO:teuthology.orchestra.run.smithi033.stdout:moved 0 / 130 (0%) 2024-04-19T01:05:16.510 INFO:teuthology.orchestra.run.smithi033.stdout:avg 8.66667 2024-04-19T01:05:16.510 INFO:teuthology.orchestra.run.smithi033.stdout:stddev 2.52102 -> 2.52102 (expected baseline 2.8441) 2024-04-19T01:05:16.510 INFO:teuthology.orchestra.run.smithi033.stdout:min osd.12 with 4 -> 4 pgs (0.461538 -> 0.461538 * mean) 2024-04-19T01:05:16.510 INFO:teuthology.orchestra.run.smithi033.stdout:max osd.11 with 15 -> 15 pgs (1.73077 -> 1.73077 * mean) 2024-04-19T01:05:16.510 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:05:16.511 INFO:teuthology.orchestra.run.smithi033.stdout:oload 1000 2024-04-19T01:05:16.511 INFO:teuthology.orchestra.run.smithi033.stdout:max_change 0.05 2024-04-19T01:05:16.511 INFO:teuthology.orchestra.run.smithi033.stdout:max_change_osds 4 2024-04-19T01:05:16.511 INFO:teuthology.orchestra.run.smithi033.stdout:average_utilization 0.0004 2024-04-19T01:05:16.511 INFO:teuthology.orchestra.run.smithi033.stdout:overload_utilization 0.0045 2024-04-19T01:05:16.513 INFO:teuthology.orchestra.run.smithi033.stderr:no change 2024-04-19T01:05:16.528 INFO:tasks.rados.rados.0.smithi155.stdout:328: done (2 left) 2024-04-19T01:05:16.528 INFO:tasks.rados.rados.0.smithi155.stdout:329: snap_create 2024-04-19T01:05:16.528 INFO:tasks.rados.rados.0.smithi155.stdout:326: finishing copy_from racing read to smithi15528762-8 2024-04-19T01:05:16.528 INFO:tasks.rados.rados.0.smithi155.stdout:327: finishing write tid 1 to smithi15528762-2 2024-04-19T01:05:16.528 INFO:tasks.rados.rados.0.smithi155.stdout:327: finishing write tid 2 to smithi15528762-2 2024-04-19T01:05:16.528 INFO:tasks.rados.rados.0.smithi155.stdout:326: finishing copy_from to smithi15528762-8 2024-04-19T01:05:16.528 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 8 v 45 (ObjNum 85 snap 14 seq_num 85) dirty exists 2024-04-19T01:05:16.528 INFO:tasks.rados.rados.0.smithi155.stdout:327: finishing write tid 3 to smithi15528762-2 2024-04-19T01:05:16.529 INFO:tasks.rados.rados.0.smithi155.stdout:327: finishing write tid 4 to smithi15528762-2 2024-04-19T01:05:16.529 INFO:tasks.rados.rados.0.smithi155.stdout:327: finishing write tid 5 to smithi15528762-2 2024-04-19T01:05:16.529 INFO:tasks.rados.rados.0.smithi155.stdout:327: finishing write tid 6 to smithi15528762-2 2024-04-19T01:05:16.529 INFO:tasks.rados.rados.0.smithi155.stdout:327: oid 2 updating version 0 to 89 2024-04-19T01:05:16.529 INFO:tasks.rados.rados.0.smithi155.stdout:327: oid 2 updating version 89 to 92 2024-04-19T01:05:16.529 INFO:tasks.rados.rados.0.smithi155.stdout:327: oid 2 version 92 is already newer than 88 2024-04-19T01:05:16.529 INFO:tasks.rados.rados.0.smithi155.stdout:327: oid 2 version 92 is already newer than 91 2024-04-19T01:05:16.529 INFO:tasks.rados.rados.0.smithi155.stdout:327: oid 2 version 92 is already newer than 90 2024-04-19T01:05:16.529 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 2 v 92 (ObjNum 131 snap 37 seq_num 131) dirty exists 2024-04-19T01:05:16.529 INFO:tasks.rados.rados.0.smithi155.stdout:327: left oid 2 (ObjNum 131 snap 37 seq_num 131) 2024-04-19T01:05:16.532 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-04-19T01:05:16.532 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 1000 0.05 2024-04-19T01:05:16.538 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:05:16.537+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:05:16.639 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:05:16.637+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:05:16.646 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:05:16.646 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-04-19T01:05:16.646 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-04-19T01:05:16.646 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-04-19T01:05:16.646 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:05:16.666 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-19T01:05:16.679 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-19T01:05:16.740 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:05:16.740+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:05:16.790 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:05:16.790 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-04-19T01:05:16.790 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-04-19T01:05:16.790 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-04-19T01:05:16.790 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.791 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 43/31 ECSubWrite(tid=43, reqid=client.4604.0:251, at_version=43'29, trim_to=0'0, roll_forward_to=43'27))", 2024-04-19T01:05:16.791 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.986035+0000", 2024-04-19T01:05:16.791 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 62.803733195, 2024-04-19T01:05:16.791 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024180535999999999, 2024-04-19T01:05:16.791 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:16.791 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:16.791 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:16.791 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:16.791 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.791 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:16.791 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986035+0000", 2024-04-19T01:05:16.791 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:16.791 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.791 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.792 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:16.792 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986035+0000", 2024-04-19T01:05:16.792 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:16.792 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.792 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.792 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:16.792 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986058+0000", 2024-04-19T01:05:16.792 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2903000000000001e-05 2024-04-19T01:05:16.792 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.792 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.792 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:16.792 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986841+0000", 2024-04-19T01:05:16.792 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00078285599999999996 2024-04-19T01:05:16.792 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.792 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.793 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:16.793 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986887+0000", 2024-04-19T01:05:16.793 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6202000000000003e-05 2024-04-19T01:05:16.793 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.793 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.793 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:16.793 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986913+0000", 2024-04-19T01:05:16.793 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6162000000000001e-05 2024-04-19T01:05:16.793 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.793 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.793 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:16.793 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987069+0000", 2024-04-19T01:05:16.793 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015625099999999999 2024-04-19T01:05:16.793 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.794 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.794 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:16.794 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987099+0000", 2024-04-19T01:05:16.794 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9578999999999999e-05 2024-04-19T01:05:16.794 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.794 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.795 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:16.795 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.010110+0000", 2024-04-19T01:05:16.795 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023011357999999999 2024-04-19T01:05:16.795 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.795 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.795 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:16.795 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.010215+0000", 2024-04-19T01:05:16.795 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000105225 2024-04-19T01:05:16.795 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:16.795 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:16.795 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:16.796 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.796 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.796 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds5 43/31 ECSubWrite(tid=38, reqid=client.4604.0:253, at_version=43'14, trim_to=0'0, roll_forward_to=43'12))", 2024-04-19T01:05:16.796 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.987245+0000", 2024-04-19T01:05:16.796 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 62.802523172999997, 2024-04-19T01:05:16.796 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022838027, 2024-04-19T01:05:16.796 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:16.796 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:16.796 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:16.796 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:16.796 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.796 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:16.796 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987245+0000", 2024-04-19T01:05:16.796 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:16.796 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.797 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.797 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:16.797 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987245+0000", 2024-04-19T01:05:16.797 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:16.797 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.797 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.797 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:16.797 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987247+0000", 2024-04-19T01:05:16.797 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2800000000000002e-06 2024-04-19T01:05:16.797 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.797 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.797 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:16.797 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988142+0000", 2024-04-19T01:05:16.797 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00089449199999999999 2024-04-19T01:05:16.798 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.798 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.798 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:16.798 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988180+0000", 2024-04-19T01:05:16.798 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8615000000000002e-05 2024-04-19T01:05:16.798 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.798 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.798 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:16.798 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988206+0000", 2024-04-19T01:05:16.798 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6023000000000001e-05 2024-04-19T01:05:16.798 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.799 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.799 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:16.799 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988349+0000", 2024-04-19T01:05:16.799 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014252199999999999 2024-04-19T01:05:16.799 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.799 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.799 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:16.799 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988365+0000", 2024-04-19T01:05:16.799 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5855e-05 2024-04-19T01:05:16.799 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.799 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.799 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:16.799 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.010029+0000", 2024-04-19T01:05:16.799 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.021664454999999999 2024-04-19T01:05:16.800 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.800 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.800 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:16.800 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.010083+0000", 2024-04-19T01:05:16.800 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3785000000000001e-05 2024-04-19T01:05:16.800 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:16.800 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:16.800 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:16.800 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.800 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.800 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.fs6 43/31 ECSubWrite(tid=47, reqid=client.4604.0:263, at_version=43'8, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:05:16.800 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.994092+0000", 2024-04-19T01:05:16.800 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 62.795676188000002, 2024-04-19T01:05:16.800 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.027264745999999999, 2024-04-19T01:05:16.801 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:16.801 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:16.801 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:16.801 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:16.801 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.801 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:16.801 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.994092+0000", 2024-04-19T01:05:16.801 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:16.801 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.801 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.801 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:16.801 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.994092+0000", 2024-04-19T01:05:16.801 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:16.802 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.802 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:16.806 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.994094+0000", 2024-04-19T01:05:16.806 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1399999999999998e-06 2024-04-19T01:05:16.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:16.806 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.994786+0000", 2024-04-19T01:05:16.806 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00069247999999999998 2024-04-19T01:05:16.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.807 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.807 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:16.807 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.994827+0000", 2024-04-19T01:05:16.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0105000000000003e-05 2024-04-19T01:05:16.807 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.808 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.808 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:16.808 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.994853+0000", 2024-04-19T01:05:16.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6259e-05 2024-04-19T01:05:16.809 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.809 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.809 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:16.809 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.995043+0000", 2024-04-19T01:05:16.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0001906 2024-04-19T01:05:16.809 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.809 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.809 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:16.809 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.995069+0000", 2024-04-19T01:05:16.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5633e-05 2024-04-19T01:05:16.809 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.809 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.809 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:16.810 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.021288+0000", 2024-04-19T01:05:16.810 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.026218865000000001 2024-04-19T01:05:16.810 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.810 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.810 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:16.810 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.021357+0000", 2024-04-19T01:05:16.810 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8664000000000005e-05 2024-04-19T01:05:16.810 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:16.810 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:16.810 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:16.810 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.810 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.810 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.es4 47/31 ECSubWrite(tid=104, reqid=client.4604.0:312, at_version=47'18, trim_to=0'0, roll_forward_to=45'16))", 2024-04-19T01:05:16.810 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.582390+0000", 2024-04-19T01:05:16.811 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 59.207378407, 2024-04-19T01:05:16.811 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022795626999999999, 2024-04-19T01:05:16.811 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:16.811 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:16.811 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:16.811 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:16.811 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.811 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:16.811 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.582390+0000", 2024-04-19T01:05:16.811 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:16.811 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.811 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.811 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:16.811 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.582390+0000", 2024-04-19T01:05:16.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:16.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.812 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:16.812 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.582390+0000", 2024-04-19T01:05:16.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8299999999999996e-07 2024-04-19T01:05:16.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.812 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:16.812 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.582798+0000", 2024-04-19T01:05:16.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00040746499999999998 2024-04-19T01:05:16.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.812 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:16.812 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.582829+0000", 2024-04-19T01:05:16.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0988999999999998e-05 2024-04-19T01:05:16.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:16.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.582846+0000", 2024-04-19T01:05:16.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7252e-05 2024-04-19T01:05:16.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:16.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.582989+0000", 2024-04-19T01:05:16.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014342000000000001 2024-04-19T01:05:16.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:16.814 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.583006+0000", 2024-04-19T01:05:16.814 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6671999999999999e-05 2024-04-19T01:05:16.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:16.814 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.605104+0000", 2024-04-19T01:05:16.814 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022098222000000001 2024-04-19T01:05:16.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:16.814 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.605185+0000", 2024-04-19T01:05:16.814 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0823999999999997e-05 2024-04-19T01:05:16.814 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:16.814 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:16.815 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:16.815 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.815 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.815 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds5 47/31 ECSubWrite(tid=61, reqid=client.4604.0:330, at_version=47'32, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:05:16.815 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.677579+0000", 2024-04-19T01:05:16.815 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 59.112189348000001, 2024-04-19T01:05:16.815 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.033702862, 2024-04-19T01:05:16.815 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:16.815 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:16.815 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:16.815 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:16.815 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.815 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:16.815 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.677579+0000", 2024-04-19T01:05:16.815 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:16.816 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.816 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.816 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:16.816 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.677579+0000", 2024-04-19T01:05:16.816 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:16.816 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.816 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.816 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:16.817 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.677581+0000", 2024-04-19T01:05:16.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2800000000000002e-06 2024-04-19T01:05:16.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.817 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:16.817 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678062+0000", 2024-04-19T01:05:16.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00048067900000000002 2024-04-19T01:05:16.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.817 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:16.817 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678096+0000", 2024-04-19T01:05:16.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4381999999999997e-05 2024-04-19T01:05:16.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:16.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678117+0000", 2024-04-19T01:05:16.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0679999999999999e-05 2024-04-19T01:05:16.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:16.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678249+0000", 2024-04-19T01:05:16.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013211899999999999 2024-04-19T01:05:16.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.819 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:16.819 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678270+0000", 2024-04-19T01:05:16.819 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1409000000000001e-05 2024-04-19T01:05:16.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.819 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:16.819 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711228+0000", 2024-04-19T01:05:16.819 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.032957575000000003 2024-04-19T01:05:16.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.819 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:16.819 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711281+0000", 2024-04-19T01:05:16.820 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3737999999999999e-05 2024-04-19T01:05:16.820 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:16.820 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:16.820 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:16.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.820 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds5 47/31 ECSubWrite(tid=62, reqid=client.4604.0:331, at_version=47'33, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:05:16.820 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.682545+0000", 2024-04-19T01:05:16.820 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 59.107223306999998, 2024-04-19T01:05:16.820 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.028771769999999999, 2024-04-19T01:05:16.820 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:16.820 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:16.820 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:16.820 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:16.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.821 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:16.827 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682545+0000", 2024-04-19T01:05:16.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:16.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.827 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:16.827 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682545+0000", 2024-04-19T01:05:16.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:16.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.827 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:16.827 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682546+0000", 2024-04-19T01:05:16.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.567e-06 2024-04-19T01:05:16.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.828 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:16.828 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682965+0000", 2024-04-19T01:05:16.828 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00041881399999999999 2024-04-19T01:05:16.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.828 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:16.828 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682999+0000", 2024-04-19T01:05:16.828 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3738999999999997e-05 2024-04-19T01:05:16.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.828 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:16.828 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.683021+0000", 2024-04-19T01:05:16.828 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2592e-05 2024-04-19T01:05:16.829 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.829 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.829 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:16.829 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.691707+0000", 2024-04-19T01:05:16.829 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0086853629999999998 2024-04-19T01:05:16.829 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.829 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.830 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:16.830 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.691732+0000", 2024-04-19T01:05:16.830 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5366000000000001e-05 2024-04-19T01:05:16.830 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.830 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.830 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:16.830 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711288+0000", 2024-04-19T01:05:16.830 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019555922999999999 2024-04-19T01:05:16.830 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.830 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.830 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:16.830 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711316+0000", 2024-04-19T01:05:16.830 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8405999999999999e-05 2024-04-19T01:05:16.830 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:16.831 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:16.831 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:16.831 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.831 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.831 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds5 47/31 ECSubWrite(tid=63, reqid=client.4604.0:332, at_version=47'34, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:05:16.831 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.688708+0000", 2024-04-19T01:05:16.831 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 59.101060410000002, 2024-04-19T01:05:16.831 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.02327926, 2024-04-19T01:05:16.831 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:16.831 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:16.831 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:16.831 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:16.831 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.831 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:16.831 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.688708+0000", 2024-04-19T01:05:16.832 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:16.832 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.832 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.832 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:16.832 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.688708+0000", 2024-04-19T01:05:16.832 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:16.832 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.832 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.832 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:16.832 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.688709+0000", 2024-04-19T01:05:16.832 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5e-06 2024-04-19T01:05:16.832 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.832 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.832 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:16.833 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.688989+0000", 2024-04-19T01:05:16.833 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00028006800000000002 2024-04-19T01:05:16.833 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.833 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.833 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:16.833 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.689021+0000", 2024-04-19T01:05:16.833 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2026000000000001e-05 2024-04-19T01:05:16.833 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.833 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.833 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:16.833 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.689047+0000", 2024-04-19T01:05:16.833 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5809999999999999e-05 2024-04-19T01:05:16.833 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.833 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.833 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:16.834 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.704765+0000", 2024-04-19T01:05:16.834 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015717913 2024-04-19T01:05:16.834 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.834 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:16.834 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.704793+0000", 2024-04-19T01:05:16.834 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8143000000000001e-05 2024-04-19T01:05:16.834 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.834 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:16.834 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711907+0000", 2024-04-19T01:05:16.834 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071141879999999996 2024-04-19T01:05:16.834 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.835 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:16.835 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711987+0000", 2024-04-19T01:05:16.835 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9611999999999996e-05 2024-04-19T01:05:16.835 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:16.835 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:16.835 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:16.835 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.835 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.835 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 59/31 ECSubWrite(tid=66, reqid=client.4604.0:423, at_version=59'51, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:05:16.835 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.541317+0000", 2024-04-19T01:05:16.835 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 47.248450804000001, 2024-04-19T01:05:16.835 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.047133948000000002, 2024-04-19T01:05:16.835 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:16.835 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:16.835 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:16.836 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:16.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.836 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:16.836 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.541317+0000", 2024-04-19T01:05:16.836 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:16.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.836 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:16.836 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.541317+0000", 2024-04-19T01:05:16.836 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:16.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.836 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:16.836 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.541318+0000", 2024-04-19T01:05:16.836 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3399999999999999e-07 2024-04-19T01:05:16.837 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.837 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.837 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:16.837 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.543831+0000", 2024-04-19T01:05:16.837 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025130999999999999 2024-04-19T01:05:16.837 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.837 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.837 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:16.838 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.543879+0000", 2024-04-19T01:05:16.838 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8208999999999999e-05 2024-04-19T01:05:16.838 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.838 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.838 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:16.838 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.543903+0000", 2024-04-19T01:05:16.838 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.404e-05 2024-04-19T01:05:16.838 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.838 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.838 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:16.838 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.544070+0000", 2024-04-19T01:05:16.838 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000166372 2024-04-19T01:05:16.838 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.838 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.839 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:16.839 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.544097+0000", 2024-04-19T01:05:16.839 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7256000000000001e-05 2024-04-19T01:05:16.839 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.839 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.839 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:16.839 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.588382+0000", 2024-04-19T01:05:16.839 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.044284780000000003 2024-04-19T01:05:16.840 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.840 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.840 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:16.840 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.588451+0000", 2024-04-19T01:05:16.840 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9556999999999998e-05 2024-04-19T01:05:16.840 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:16.840 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:16.840 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:16.840 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.840 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.840 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 59/31 ECSubWrite(tid=67, reqid=client.4604.0:424, at_version=59'52, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:05:16.840 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.548963+0000", 2024-04-19T01:05:16.841 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 47.240805037999998, 2024-04-19T01:05:16.841 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.039534443000000002, 2024-04-19T01:05:16.841 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:16.841 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:16.841 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:16.843 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:16.843 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.843 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:16.843 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.548963+0000", 2024-04-19T01:05:16.843 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:16.844 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.845 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.845 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:16.845 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.548963+0000", 2024-04-19T01:05:16.845 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:16.845 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.845 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.845 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:16.846 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.548965+0000", 2024-04-19T01:05:16.846 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1500000000000002e-06 2024-04-19T01:05:16.846 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.846 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.846 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:16.846 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549551+0000", 2024-04-19T01:05:16.846 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00058640299999999999 2024-04-19T01:05:16.846 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.846 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.846 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:16.846 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549596+0000", 2024-04-19T01:05:16.846 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4604999999999997e-05 2024-04-19T01:05:16.846 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.846 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.846 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:16.847 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549631+0000", 2024-04-19T01:05:16.847 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4725999999999999e-05 2024-04-19T01:05:16.847 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.847 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.847 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:16.847 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.567909+0000", 2024-04-19T01:05:16.847 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018277755999999999 2024-04-19T01:05:16.847 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.848 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.848 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:16.848 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.567944+0000", 2024-04-19T01:05:16.848 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5862999999999999e-05 2024-04-19T01:05:16.848 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.848 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.848 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:16.848 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.588471+0000", 2024-04-19T01:05:16.848 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020526902999999999 2024-04-19T01:05:16.848 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.848 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.848 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:16.848 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.588497+0000", 2024-04-19T01:05:16.848 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6037000000000002e-05 2024-04-19T01:05:16.848 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:16.849 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:16.849 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:16.849 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.849 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.849 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 59/31 ECSubWrite(tid=68, reqid=client.4604.0:425, at_version=59'53, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:05:16.849 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.556523+0000", 2024-04-19T01:05:16.849 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 47.233244626999998, 2024-04-19T01:05:16.849 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.032109275999999999, 2024-04-19T01:05:16.849 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:16.849 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:16.849 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:16.849 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:16.849 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.849 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:16.849 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556523+0000", 2024-04-19T01:05:16.850 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:16.850 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.850 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.850 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:16.850 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556523+0000", 2024-04-19T01:05:16.850 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:16.851 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.851 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.851 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:16.851 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556526+0000", 2024-04-19T01:05:16.851 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4930000000000001e-06 2024-04-19T01:05:16.851 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.851 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.851 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:16.851 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556650+0000", 2024-04-19T01:05:16.851 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000124487 2024-04-19T01:05:16.851 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.851 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.851 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:16.851 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556701+0000", 2024-04-19T01:05:16.852 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1158e-05 2024-04-19T01:05:16.852 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.852 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.852 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:16.852 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556727+0000", 2024-04-19T01:05:16.852 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5843999999999999e-05 2024-04-19T01:05:16.852 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.852 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.852 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:16.852 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.585945+0000", 2024-04-19T01:05:16.852 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0292181 2024-04-19T01:05:16.852 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.852 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.852 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:16.853 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.585972+0000", 2024-04-19T01:05:16.853 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6356e-05 2024-04-19T01:05:16.853 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.853 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.853 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:16.853 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.588601+0000", 2024-04-19T01:05:16.853 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0026295839999999999 2024-04-19T01:05:16.853 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.853 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.853 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:16.853 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.588633+0000", 2024-04-19T01:05:16.853 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1254000000000003e-05 2024-04-19T01:05:16.853 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:16.853 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:16.854 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:16.854 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.854 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.854 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 59/31 ECSubWrite(tid=69, reqid=client.4604.0:426, at_version=59'54, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:05:16.854 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.558033+0000", 2024-04-19T01:05:16.854 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 47.231734676999999, 2024-04-19T01:05:16.854 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.030934886000000002, 2024-04-19T01:05:16.854 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:16.854 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:16.854 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:16.854 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:16.854 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.854 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:16.854 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558033+0000", 2024-04-19T01:05:16.855 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:16.855 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.855 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.855 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:16.855 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558033+0000", 2024-04-19T01:05:16.855 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:16.855 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.855 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.855 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:16.855 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558035+0000", 2024-04-19T01:05:16.855 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7370000000000001e-06 2024-04-19T01:05:16.855 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.855 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.855 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:16.855 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558080+0000", 2024-04-19T01:05:16.856 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5089000000000001e-05 2024-04-19T01:05:16.856 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.856 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.856 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:16.856 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558126+0000", 2024-04-19T01:05:16.856 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5902000000000002e-05 2024-04-19T01:05:16.856 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.856 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.856 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:16.856 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558149+0000", 2024-04-19T01:05:16.856 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3136e-05 2024-04-19T01:05:16.856 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.856 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.856 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:16.856 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.588071+0000", 2024-04-19T01:05:16.857 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.029921515999999999 2024-04-19T01:05:16.857 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.857 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.857 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:16.857 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.588089+0000", 2024-04-19T01:05:16.857 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8788e-05 2024-04-19T01:05:16.857 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.857 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.857 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:16.857 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.588944+0000", 2024-04-19T01:05:16.857 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00085441300000000005 2024-04-19T01:05:16.857 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.858 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.858 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:16.858 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.588968+0000", 2024-04-19T01:05:16.858 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4304999999999999e-05 2024-04-19T01:05:16.858 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:16.858 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:16.858 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:16.858 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.858 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.858 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 79/31 ECSubWrite(tid=115, reqid=client.4604.0:591, at_version=79'90, trim_to=0'0, roll_forward_to=79'86))", 2024-04-19T01:05:16.859 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:49.657667+0000", 2024-04-19T01:05:16.859 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 27.132100967, 2024-04-19T01:05:16.859 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.042570781000000002, 2024-04-19T01:05:16.859 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:16.859 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:16.859 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:16.859 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:16.859 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.859 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:16.859 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.657667+0000", 2024-04-19T01:05:16.859 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:16.859 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.859 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.859 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:16.859 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.657667+0000", 2024-04-19T01:05:16.860 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:16.860 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.860 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.860 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:16.860 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.657669+0000", 2024-04-19T01:05:16.861 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.564e-06 2024-04-19T01:05:16.861 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.861 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.861 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:16.861 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658422+0000", 2024-04-19T01:05:16.861 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00075376200000000005 2024-04-19T01:05:16.861 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.861 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.861 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:16.861 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658456+0000", 2024-04-19T01:05:16.861 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4059e-05 2024-04-19T01:05:16.861 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.862 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.862 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:16.862 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658479+0000", 2024-04-19T01:05:16.862 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2235e-05 2024-04-19T01:05:16.862 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.862 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.862 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:16.862 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658610+0000", 2024-04-19T01:05:16.862 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013109600000000001 2024-04-19T01:05:16.862 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.862 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.862 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:16.862 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658624+0000", 2024-04-19T01:05:16.862 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4756000000000001e-05 2024-04-19T01:05:16.863 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.863 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.863 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:16.863 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.699665+0000", 2024-04-19T01:05:16.863 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.041040764 2024-04-19T01:05:16.863 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.863 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.863 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:16.863 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.700238+0000", 2024-04-19T01:05:16.863 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00057254500000000004 2024-04-19T01:05:16.863 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:16.863 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:16.863 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:16.863 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.863 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.864 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 79/31 ECSubWrite(tid=116, reqid=client.4604.0:592, at_version=79'91, trim_to=0'0, roll_forward_to=79'88))", 2024-04-19T01:05:16.864 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:49.665950+0000", 2024-04-19T01:05:16.864 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 27.123818382, 2024-04-19T01:05:16.864 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.037031065000000002, 2024-04-19T01:05:16.864 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:16.864 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:16.864 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:16.864 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:16.864 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.864 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:16.864 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.665950+0000", 2024-04-19T01:05:16.864 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:16.864 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.864 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.864 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:16.865 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.665950+0000", 2024-04-19T01:05:16.865 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:16.865 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.865 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.865 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:16.865 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.665952+0000", 2024-04-19T01:05:16.865 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0360000000000001e-06 2024-04-19T01:05:16.865 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.865 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.865 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:16.865 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.666593+0000", 2024-04-19T01:05:16.865 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00064100300000000002 2024-04-19T01:05:16.865 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.865 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.865 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:16.866 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.666640+0000", 2024-04-19T01:05:16.866 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7817e-05 2024-04-19T01:05:16.866 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.867 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.867 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:16.867 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.666676+0000", 2024-04-19T01:05:16.867 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5222000000000003e-05 2024-04-19T01:05:16.867 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.867 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.867 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:16.867 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.676934+0000", 2024-04-19T01:05:16.867 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010258801999999999 2024-04-19T01:05:16.867 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.868 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.869 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:16.869 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.676964+0000", 2024-04-19T01:05:16.869 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9108999999999999e-05 2024-04-19T01:05:16.869 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.869 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.869 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:16.869 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.702830+0000", 2024-04-19T01:05:16.870 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025866865999999999 2024-04-19T01:05:16.870 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.870 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.870 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:16.870 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.702981+0000", 2024-04-19T01:05:16.870 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015020999999999999 2024-04-19T01:05:16.870 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:16.870 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:16.870 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:16.870 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.870 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.870 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 79/31 ECSubWrite(tid=117, reqid=client.4604.0:593, at_version=79'92, trim_to=0'0, roll_forward_to=79'88))", 2024-04-19T01:05:16.870 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:49.674833+0000", 2024-04-19T01:05:16.870 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 27.114934862999998, 2024-04-19T01:05:16.871 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.028247395000000002, 2024-04-19T01:05:16.871 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:16.871 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:16.871 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:16.871 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:16.871 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.871 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:16.872 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.674833+0000", 2024-04-19T01:05:16.872 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:16.872 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.872 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.872 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:16.872 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.674833+0000", 2024-04-19T01:05:16.872 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:16.872 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.873 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.873 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:16.873 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.674834+0000", 2024-04-19T01:05:16.873 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3680000000000001e-06 2024-04-19T01:05:16.873 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.873 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.873 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:16.874 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.675074+0000", 2024-04-19T01:05:16.874 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00023945699999999999 2024-04-19T01:05:16.874 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.874 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.874 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:16.874 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.675108+0000", 2024-04-19T01:05:16.874 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4424000000000001e-05 2024-04-19T01:05:16.874 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.874 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.874 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:16.874 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.675127+0000", 2024-04-19T01:05:16.874 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9038999999999998e-05 2024-04-19T01:05:16.874 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.874 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.875 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:16.875 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.690509+0000", 2024-04-19T01:05:16.875 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015381234000000001 2024-04-19T01:05:16.875 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.875 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.875 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:16.875 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.690527+0000", 2024-04-19T01:05:16.875 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8155000000000002e-05 2024-04-19T01:05:16.875 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.875 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.875 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:16.875 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.702999+0000", 2024-04-19T01:05:16.875 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.01247173 2024-04-19T01:05:16.875 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.876 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.876 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:16.876 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.703081+0000", 2024-04-19T01:05:16.876 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1988000000000002e-05 2024-04-19T01:05:16.876 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:16.876 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:16.876 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:16.876 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.876 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.876 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 79/31 ECSubWrite(tid=118, reqid=client.4604.0:594, at_version=79'93, trim_to=0'0, roll_forward_to=79'88))", 2024-04-19T01:05:16.876 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:49.679572+0000", 2024-04-19T01:05:16.876 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 27.110195824000002, 2024-04-19T01:05:16.876 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024026347999999999, 2024-04-19T01:05:16.876 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:16.876 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:16.876 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:16.877 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:16.877 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.878 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:16.878 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.679572+0000", 2024-04-19T01:05:16.878 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:16.878 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.878 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.878 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:16.878 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.679572+0000", 2024-04-19T01:05:16.878 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:16.879 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.879 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.879 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:16.879 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.679574+0000", 2024-04-19T01:05:16.879 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9700000000000002e-06 2024-04-19T01:05:16.879 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.880 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.880 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:16.880 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.679637+0000", 2024-04-19T01:05:16.880 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2646999999999998e-05 2024-04-19T01:05:16.880 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.880 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.880 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:16.880 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.679677+0000", 2024-04-19T01:05:16.880 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0540000000000001e-05 2024-04-19T01:05:16.880 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.880 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.880 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:16.880 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.679697+0000", 2024-04-19T01:05:16.881 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0108000000000001e-05 2024-04-19T01:05:16.881 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.881 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.881 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:16.881 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.700314+0000", 2024-04-19T01:05:16.881 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020616520999999999 2024-04-19T01:05:16.881 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.881 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.881 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:16.882 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.700332+0000", 2024-04-19T01:05:16.882 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7665000000000001e-05 2024-04-19T01:05:16.882 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.882 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.882 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:16.882 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.703547+0000", 2024-04-19T01:05:16.882 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0032152729999999998 2024-04-19T01:05:16.882 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.882 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.882 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:16.882 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.703598+0000", 2024-04-19T01:05:16.882 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1623999999999999e-05 2024-04-19T01:05:16.882 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:16.883 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:16.883 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:16.883 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.883 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.883 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 79/31 ECSubWrite(tid=119, reqid=client.4604.0:595, at_version=79'94, trim_to=0'0, roll_forward_to=79'88))", 2024-04-19T01:05:16.883 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:49.682205+0000", 2024-04-19T01:05:16.883 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 27.107563395, 2024-04-19T01:05:16.883 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022707974999999998, 2024-04-19T01:05:16.883 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:16.883 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:16.883 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:16.883 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:16.883 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.883 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:16.883 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.682205+0000", 2024-04-19T01:05:16.884 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:16.884 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.884 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.884 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:16.884 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.682205+0000", 2024-04-19T01:05:16.884 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:16.884 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.884 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.884 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:16.884 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.682206+0000", 2024-04-19T01:05:16.884 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5149999999999999e-06 2024-04-19T01:05:16.884 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.884 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.884 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:16.885 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.682263+0000", 2024-04-19T01:05:16.885 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6456000000000003e-05 2024-04-19T01:05:16.885 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.885 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.885 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:16.885 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.682303+0000", 2024-04-19T01:05:16.885 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0101000000000001e-05 2024-04-19T01:05:16.885 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.885 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.885 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:16.885 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.682322+0000", 2024-04-19T01:05:16.885 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9266000000000002e-05 2024-04-19T01:05:16.885 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.885 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.885 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:16.886 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.703175+0000", 2024-04-19T01:05:16.886 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020853097000000001 2024-04-19T01:05:16.886 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.886 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.886 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:16.886 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.703191+0000", 2024-04-19T01:05:16.886 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6234000000000001e-05 2024-04-19T01:05:16.886 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.886 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.886 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:16.886 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.704842+0000", 2024-04-19T01:05:16.886 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0016502699999999999 2024-04-19T01:05:16.886 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.886 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.887 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:16.887 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.704913+0000", 2024-04-19T01:05:16.887 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1036000000000002e-05 2024-04-19T01:05:16.887 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:16.887 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:16.887 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:16.887 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.887 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.887 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.es4 86/31 ECSubWrite(tid=195, reqid=client.4604.0:653, at_version=86'39, trim_to=0'0, roll_forward_to=84'38))", 2024-04-19T01:05:16.887 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:56.572490+0000", 2024-04-19T01:05:16.887 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.217277698, 2024-04-19T01:05:16.887 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022945307000000002, 2024-04-19T01:05:16.887 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:16.887 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:16.888 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:16.888 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:16.888 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.888 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:16.888 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.572490+0000", 2024-04-19T01:05:16.888 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:16.888 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.888 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.888 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:16.888 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.572490+0000", 2024-04-19T01:05:16.888 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:16.888 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.888 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.888 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:16.888 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.572491+0000", 2024-04-19T01:05:16.889 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1800000000000005e-07 2024-04-19T01:05:16.889 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.889 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.889 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:16.889 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.574244+0000", 2024-04-19T01:05:16.889 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017527179999999999 2024-04-19T01:05:16.889 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.889 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.889 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:16.889 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.574268+0000", 2024-04-19T01:05:16.889 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3689999999999998e-05 2024-04-19T01:05:16.889 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.889 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.897 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:16.897 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.574282+0000", 2024-04-19T01:05:16.898 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4011e-05 2024-04-19T01:05:16.898 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.898 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.898 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:16.898 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.574403+0000", 2024-04-19T01:05:16.898 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012181 2024-04-19T01:05:16.898 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.898 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.898 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:16.898 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.574420+0000", 2024-04-19T01:05:16.899 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6390000000000001e-05 2024-04-19T01:05:16.899 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.899 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.899 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:16.899 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.595364+0000", 2024-04-19T01:05:16.899 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020944312999999999 2024-04-19T01:05:16.899 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.899 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.899 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:16.899 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.595436+0000", 2024-04-19T01:05:16.899 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1557000000000006e-05 2024-04-19T01:05:16.899 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:16.899 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:16.899 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:16.899 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.900 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.900 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds5 102/98 ECSubWrite(tid=209, reqid=client.4604.0:759, at_version=102'79, trim_to=0'0, roll_forward_to=0'0))", 2024-04-19T01:05:16.900 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:05:12.701960+0000", 2024-04-19T01:05:16.900 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.0878082649999996, 2024-04-19T01:05:16.900 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.039216858, 2024-04-19T01:05:16.900 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:16.900 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:16.900 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:16.901 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:16.901 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.901 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:16.901 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.701960+0000", 2024-04-19T01:05:16.901 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:16.901 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.901 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.901 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:16.901 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.701960+0000", 2024-04-19T01:05:16.901 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:16.901 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.901 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.901 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:16.901 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.701962+0000", 2024-04-19T01:05:16.902 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.103e-06 2024-04-19T01:05:16.902 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.902 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.902 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:16.902 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.702295+0000", 2024-04-19T01:05:16.902 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00033357999999999999 2024-04-19T01:05:16.902 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.902 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.903 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:16.903 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.702341+0000", 2024-04-19T01:05:16.903 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5207999999999998e-05 2024-04-19T01:05:16.903 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.903 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.903 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:16.903 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.702367+0000", 2024-04-19T01:05:16.903 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6876000000000002e-05 2024-04-19T01:05:16.903 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.903 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.903 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:16.903 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.707690+0000", 2024-04-19T01:05:16.903 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0053220890000000003 2024-04-19T01:05:16.904 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.904 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.904 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:16.904 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.707719+0000", 2024-04-19T01:05:16.904 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9899e-05 2024-04-19T01:05:16.904 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.904 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.904 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:16.904 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.741009+0000", 2024-04-19T01:05:16.904 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.033289524000000001 2024-04-19T01:05:16.904 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.904 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.905 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:16.905 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.741177+0000", 2024-04-19T01:05:16.905 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000167579 2024-04-19T01:05:16.905 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:16.906 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:16.906 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:16.906 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.906 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.906 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds5 102/98 ECSubWrite(tid=210, reqid=client.4604.0:760, at_version=102'80, trim_to=0'0, roll_forward_to=0'0))", 2024-04-19T01:05:16.906 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:05:12.712751+0000", 2024-04-19T01:05:16.906 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.0770168739999999, 2024-04-19T01:05:16.906 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.028519213000000002, 2024-04-19T01:05:16.906 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:16.906 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:16.906 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:16.907 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:16.907 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.907 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:16.907 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.712751+0000", 2024-04-19T01:05:16.907 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:16.907 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.907 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.907 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:16.907 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.712751+0000", 2024-04-19T01:05:16.907 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:16.907 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.907 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.907 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:16.907 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.712753+0000", 2024-04-19T01:05:16.908 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0099999999999998e-06 2024-04-19T01:05:16.908 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.908 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.908 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:16.908 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.713133+0000", 2024-04-19T01:05:16.908 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000380274 2024-04-19T01:05:16.908 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.908 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.908 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:16.908 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.713172+0000", 2024-04-19T01:05:16.908 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9072000000000001e-05 2024-04-19T01:05:16.908 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.908 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.908 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:16.909 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.713196+0000", 2024-04-19T01:05:16.909 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3929000000000001e-05 2024-04-19T01:05:16.909 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.909 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.909 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:16.909 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.717092+0000", 2024-04-19T01:05:16.909 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003896039 2024-04-19T01:05:16.909 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.909 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.909 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:16.909 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.717108+0000", 2024-04-19T01:05:16.909 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5990000000000001e-05 2024-04-19T01:05:16.909 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.909 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.910 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:16.910 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.741186+0000", 2024-04-19T01:05:16.910 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024077352 2024-04-19T01:05:16.910 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.910 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.910 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:16.910 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.741270+0000", 2024-04-19T01:05:16.910 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4547000000000002e-05 2024-04-19T01:05:16.911 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:16.911 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:16.911 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:16.911 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.911 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.911 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds5 102/98 ECSubWrite(tid=211, reqid=client.4604.0:761, at_version=102'81, trim_to=0'0, roll_forward_to=0'0))", 2024-04-19T01:05:16.913 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:05:12.717642+0000", 2024-04-19T01:05:16.913 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.0721258909999998, 2024-04-19T01:05:16.913 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023817917000000001, 2024-04-19T01:05:16.913 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:16.913 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:16.913 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:16.913 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:16.913 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.913 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:16.913 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.717642+0000", 2024-04-19T01:05:16.913 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:16.914 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.914 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.914 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:16.914 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.717642+0000", 2024-04-19T01:05:16.914 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:16.914 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.914 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.914 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:16.914 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.717644+0000", 2024-04-19T01:05:16.914 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5230000000000001e-06 2024-04-19T01:05:16.914 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.914 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.914 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:16.914 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.717868+0000", 2024-04-19T01:05:16.914 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00022392899999999999 2024-04-19T01:05:16.915 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.915 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.915 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:16.915 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.717911+0000", 2024-04-19T01:05:16.915 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3256000000000001e-05 2024-04-19T01:05:16.915 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.915 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.915 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:16.915 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.717926+0000", 2024-04-19T01:05:16.916 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5642e-05 2024-04-19T01:05:16.916 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.916 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.916 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:16.916 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.731101+0000", 2024-04-19T01:05:16.916 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013174079999999999 2024-04-19T01:05:16.916 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.916 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.916 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:16.916 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.731126+0000", 2024-04-19T01:05:16.916 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5986000000000002e-05 2024-04-19T01:05:16.916 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.916 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.916 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:16.916 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.741364+0000", 2024-04-19T01:05:16.917 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010237227999999999 2024-04-19T01:05:16.917 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:16.917 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:16.917 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:16.917 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.741460+0000", 2024-04-19T01:05:16.917 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6273000000000005e-05 2024-04-19T01:05:16.917 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:16.917 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:16.917 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:16.917 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:16.917 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:16.917 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:05:16.918 INFO:teuthology.orchestra.run.smithi033.stdout:moved 0 / 130 (0%) 2024-04-19T01:05:16.919 INFO:teuthology.orchestra.run.smithi033.stdout:avg 8.66667 2024-04-19T01:05:16.919 INFO:teuthology.orchestra.run.smithi033.stdout:stddev 2.52102 -> 2.52102 (expected baseline 2.8441) 2024-04-19T01:05:16.919 INFO:teuthology.orchestra.run.smithi033.stdout:min osd.12 with 4 -> 4 pgs (0.461538 -> 0.461538 * mean) 2024-04-19T01:05:16.919 INFO:teuthology.orchestra.run.smithi033.stdout:max osd.11 with 15 -> 15 pgs (1.73077 -> 1.73077 * mean) 2024-04-19T01:05:16.919 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:05:16.919 INFO:teuthology.orchestra.run.smithi033.stdout:oload 1000 2024-04-19T01:05:16.919 INFO:teuthology.orchestra.run.smithi033.stdout:max_change 0.05 2024-04-19T01:05:16.919 INFO:teuthology.orchestra.run.smithi033.stdout:max_change_osds 4 2024-04-19T01:05:16.919 INFO:teuthology.orchestra.run.smithi033.stdout:average_utilization 0.0004 2024-04-19T01:05:16.919 INFO:teuthology.orchestra.run.smithi033.stdout:overload_utilization 0.0045 2024-04-19T01:05:16.923 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-19T01:05:16.924 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:05:16.918+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:05:16.924 INFO:teuthology.orchestra.run.smithi033.stderr:no change 2024-04-19T01:05:16.924 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-04-19T01:05:16.924 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 110 1.0 2024-04-19T01:05:17.020 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:05:17.018+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:05:17.098 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:05:17.098 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-04-19T01:05:17.098 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-04-19T01:05:17.098 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:05:17.112 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-19T01:05:17.155 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:05:17.154+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:05:17.223 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:05:17.220+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:05:17.231 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-04-19T01:05:17.232+0000 7f61f7aa9640 -1 mgr.server reply reply (2) No such file or directory no change 2024-04-19T01:05:17.232 INFO:teuthology.orchestra.run.smithi033.stdout:moved 2 / 130 (1.53846%) 2024-04-19T01:05:17.232 INFO:teuthology.orchestra.run.smithi033.stdout:avg 8.66667 2024-04-19T01:05:17.232 INFO:teuthology.orchestra.run.smithi033.stdout:stddev 2.52102 -> 2.24103 (expected baseline 2.8441) 2024-04-19T01:05:17.232 INFO:teuthology.orchestra.run.smithi033.stdout:min osd.12 with 4 -> 5 pgs (0.461538 -> 0.576923 * mean) 2024-04-19T01:05:17.232 INFO:teuthology.orchestra.run.smithi033.stdout:max osd.11 with 15 -> 14 pgs (1.73077 -> 1.61538 * mean) 2024-04-19T01:05:17.232 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:05:17.232 INFO:teuthology.orchestra.run.smithi033.stdout:oload 110 2024-04-19T01:05:17.232 INFO:teuthology.orchestra.run.smithi033.stdout:max_change 1 2024-04-19T01:05:17.233 INFO:teuthology.orchestra.run.smithi033.stdout:max_change_osds 4 2024-04-19T01:05:17.233 INFO:teuthology.orchestra.run.smithi033.stdout:average_utilization 0.0004 2024-04-19T01:05:17.233 INFO:teuthology.orchestra.run.smithi033.stdout:overload_utilization 0.0005 2024-04-19T01:05:17.233 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14 weight 1.0000 -> 0.8961 2024-04-19T01:05:17.233 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11 weight 1.0000 -> 0.9028 2024-04-19T01:05:17.233 INFO:teuthology.orchestra.run.smithi033.stderr:no change 2024-04-19T01:05:17.252 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-04-19T01:05:17.252 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 1000 1.0 2024-04-19T01:05:17.271 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:05:17.271 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-04-19T01:05:17.271 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-04-19T01:05:17.272 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-04-19T01:05:17.272 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:05:17.285 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-19T01:05:17.324 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:05:17.324+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:05:17.413 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:05:17.413 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-04-19T01:05:17.413 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-04-19T01:05:17.413 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-04-19T01:05:17.414 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.414 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s1 43/31 ECSubWrite(tid=43, reqid=client.4604.0:251, at_version=43'29, trim_to=0'0, roll_forward_to=43'27))", 2024-04-19T01:05:17.414 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.985924+0000", 2024-04-19T01:05:17.414 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.427020716999998, 2024-04-19T01:05:17.414 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.029591623000000001, 2024-04-19T01:05:17.414 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:17.414 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:17.414 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:17.414 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:17.414 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.415 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:17.415 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.985924+0000", 2024-04-19T01:05:17.415 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:17.415 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.415 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.415 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:17.415 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.985924+0000", 2024-04-19T01:05:17.415 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:17.415 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.415 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.415 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:17.415 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.985927+0000", 2024-04-19T01:05:17.415 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3460000000000001e-06 2024-04-19T01:05:17.416 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.416 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.416 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:17.416 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986452+0000", 2024-04-19T01:05:17.416 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00052555199999999999 2024-04-19T01:05:17.416 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.416 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.416 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:17.416 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986541+0000", 2024-04-19T01:05:17.416 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8352999999999999e-05 2024-04-19T01:05:17.416 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.416 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.416 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:17.416 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986597+0000", 2024-04-19T01:05:17.417 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6144999999999999e-05 2024-04-19T01:05:17.417 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.417 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.417 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:17.417 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986776+0000", 2024-04-19T01:05:17.417 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017914999999999999 2024-04-19T01:05:17.417 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.417 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.417 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:17.417 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986811+0000", 2024-04-19T01:05:17.417 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5233e-05 2024-04-19T01:05:17.417 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.417 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.418 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:17.418 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.015444+0000", 2024-04-19T01:05:17.418 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.028632676999999999 2024-04-19T01:05:17.418 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.418 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.419 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:17.419 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.015516+0000", 2024-04-19T01:05:17.419 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2167000000000002e-05 2024-04-19T01:05:17.419 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:17.419 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:17.419 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:17.419 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.419 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.419 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:289 3.1s0 3:8ea800b0:::smithi15528762-8:head [stat,delete] snapc 7=[7,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-19T01:05:17.419 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.872650+0000", 2024-04-19T01:05:17.419 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 62.540294949, 2024-04-19T01:05:17.420 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.043399042999999998, 2024-04-19T01:05:17.420 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:17.420 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:17.420 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:17.420 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:05:17.420 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:05:17.420 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:17.420 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 289 2024-04-19T01:05:17.420 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.420 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:17.420 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.420 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:17.420 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.872650+0000", 2024-04-19T01:05:17.421 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:17.421 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.421 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.421 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:17.421 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.872650+0000", 2024-04-19T01:05:17.421 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:17.421 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.421 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.421 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:17.421 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.872653+0000", 2024-04-19T01:05:17.421 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.23e-06 2024-04-19T01:05:17.421 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.421 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.421 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:17.422 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.872669+0000", 2024-04-19T01:05:17.422 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5293000000000001e-05 2024-04-19T01:05:17.422 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.422 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.422 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:17.422 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.872701+0000", 2024-04-19T01:05:17.422 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2391999999999998e-05 2024-04-19T01:05:17.422 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.422 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.422 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:17.422 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.872721+0000", 2024-04-19T01:05:17.422 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0400000000000001e-05 2024-04-19T01:05:17.422 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.422 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.423 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:17.423 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.872889+0000", 2024-04-19T01:05:17.423 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016723000000000001 2024-04-19T01:05:17.423 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.423 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.423 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:05:17.423 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.873172+0000", 2024-04-19T01:05:17.423 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000283338 2024-04-19T01:05:17.423 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.423 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.423 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:17.423 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.875406+0000", 2024-04-19T01:05:17.423 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022335839999999998 2024-04-19T01:05:17.423 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.424 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.424 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:17.424 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.914580+0000", 2024-04-19T01:05:17.424 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.039174253999999999 2024-04-19T01:05:17.424 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.424 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.424 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-04-19T01:05:17.424 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.914900+0000", 2024-04-19T01:05:17.424 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000320663 2024-04-19T01:05:17.424 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.424 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.424 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:17.424 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.916049+0000", 2024-04-19T01:05:17.424 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001148659 2024-04-19T01:05:17.425 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:17.425 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:17.425 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:17.425 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.425 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.425 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s6 44/31 ECSubWrite(tid=82, reqid=client.4604.0:280, at_version=44'50, trim_to=0'0, roll_forward_to=43'48))", 2024-04-19T01:05:17.425 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.882093+0000", 2024-04-19T01:05:17.425 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 62.530851654999999, 2024-04-19T01:05:17.425 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.042938993000000002, 2024-04-19T01:05:17.426 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:17.426 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:17.426 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:17.426 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:17.426 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.426 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:17.426 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882093+0000", 2024-04-19T01:05:17.426 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:17.426 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.426 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.426 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:17.427 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882093+0000", 2024-04-19T01:05:17.427 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:17.427 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.427 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.427 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:17.427 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882094+0000", 2024-04-19T01:05:17.427 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0130000000000001e-06 2024-04-19T01:05:17.427 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.427 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.427 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:17.427 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882590+0000", 2024-04-19T01:05:17.427 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00049558999999999996 2024-04-19T01:05:17.427 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.427 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.427 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:17.428 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882631+0000", 2024-04-19T01:05:17.428 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1399000000000001e-05 2024-04-19T01:05:17.428 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.428 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.428 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:17.428 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882654+0000", 2024-04-19T01:05:17.428 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2535999999999999e-05 2024-04-19T01:05:17.428 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.428 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.428 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:17.428 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882871+0000", 2024-04-19T01:05:17.428 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021730999999999999 2024-04-19T01:05:17.429 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.429 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.429 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:17.429 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882896+0000", 2024-04-19T01:05:17.429 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5111999999999999e-05 2024-04-19T01:05:17.429 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.430 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.430 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:17.430 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.924961+0000", 2024-04-19T01:05:17.430 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.042065022000000001 2024-04-19T01:05:17.430 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.430 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.430 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:17.430 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.925032+0000", 2024-04-19T01:05:17.430 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1011000000000002e-05 2024-04-19T01:05:17.430 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:17.431 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:17.431 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:17.431 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.431 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.431 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.0s6 44/31 ECSubWrite(tid=25, reqid=client.4604.0:284, at_version=44'22, trim_to=0'0, roll_forward_to=44'20))", 2024-04-19T01:05:17.431 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.921980+0000", 2024-04-19T01:05:17.431 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 62.490965381999999, 2024-04-19T01:05:17.431 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.036689795999999997, 2024-04-19T01:05:17.431 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:17.431 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:17.431 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:17.431 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:17.431 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.431 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:17.432 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.921980+0000", 2024-04-19T01:05:17.432 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:17.432 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.432 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.432 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:17.432 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.921980+0000", 2024-04-19T01:05:17.432 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:17.432 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.432 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.432 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:17.432 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.921981+0000", 2024-04-19T01:05:17.432 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.378e-06 2024-04-19T01:05:17.432 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.432 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.432 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:17.433 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.923355+0000", 2024-04-19T01:05:17.433 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0013742820000000001 2024-04-19T01:05:17.433 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.433 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.433 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:17.433 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.923387+0000", 2024-04-19T01:05:17.433 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1702999999999998e-05 2024-04-19T01:05:17.433 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.433 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.433 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:17.433 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.923405+0000", 2024-04-19T01:05:17.433 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8071999999999999e-05 2024-04-19T01:05:17.433 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.433 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.434 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:17.434 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.923699+0000", 2024-04-19T01:05:17.434 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029384700000000002 2024-04-19T01:05:17.434 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.434 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.434 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:17.434 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.923727+0000", 2024-04-19T01:05:17.434 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8085999999999998e-05 2024-04-19T01:05:17.434 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.434 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.434 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:17.434 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.958618+0000", 2024-04-19T01:05:17.434 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.034891349000000002 2024-04-19T01:05:17.435 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.435 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.435 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:17.435 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.958669+0000", 2024-04-19T01:05:17.435 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1078999999999997e-05 2024-04-19T01:05:17.435 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:17.435 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:17.435 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:17.435 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.435 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.435 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds6 47/31 ECSubWrite(tid=61, reqid=client.4604.0:330, at_version=47'32, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:05:17.435 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.677697+0000", 2024-04-19T01:05:17.435 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 59.735248437999999, 2024-04-19T01:05:17.435 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.034267463999999997, 2024-04-19T01:05:17.435 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:17.436 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:17.436 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:17.436 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:17.436 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.436 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:17.436 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.677697+0000", 2024-04-19T01:05:17.436 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:17.436 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.436 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.436 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:17.436 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.677697+0000", 2024-04-19T01:05:17.436 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:17.436 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.436 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.436 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:17.437 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.677699+0000", 2024-04-19T01:05:17.437 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3929999999999998e-06 2024-04-19T01:05:17.437 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.437 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.437 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:17.437 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678182+0000", 2024-04-19T01:05:17.437 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00048326199999999998 2024-04-19T01:05:17.437 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.437 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.437 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:17.437 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678227+0000", 2024-04-19T01:05:17.437 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4301999999999997e-05 2024-04-19T01:05:17.438 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.438 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.438 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:17.438 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678254+0000", 2024-04-19T01:05:17.438 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7396e-05 2024-04-19T01:05:17.438 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.438 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.438 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:17.438 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678409+0000", 2024-04-19T01:05:17.438 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000154759 2024-04-19T01:05:17.440 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.441 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.441 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:17.441 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678438+0000", 2024-04-19T01:05:17.442 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8926000000000001e-05 2024-04-19T01:05:17.442 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.442 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.442 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:17.442 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711880+0000", 2024-04-19T01:05:17.442 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.033442209000000001 2024-04-19T01:05:17.442 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.442 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.442 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:17.442 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711964+0000", 2024-04-19T01:05:17.442 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4216999999999997e-05 2024-04-19T01:05:17.442 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:17.442 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:17.442 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:17.443 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.443 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.443 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds6 47/31 ECSubWrite(tid=62, reqid=client.4604.0:331, at_version=47'33, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:05:17.443 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.682679+0000", 2024-04-19T01:05:17.443 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 59.730266047999997, 2024-04-19T01:05:17.443 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.029354201999999999, 2024-04-19T01:05:17.443 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:17.443 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:17.443 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:17.443 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:17.443 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.443 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:17.443 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682679+0000", 2024-04-19T01:05:17.443 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:17.443 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.444 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.444 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:17.444 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682679+0000", 2024-04-19T01:05:17.444 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:17.444 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.444 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.444 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:17.444 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682681+0000", 2024-04-19T01:05:17.444 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7799999999999999e-06 2024-04-19T01:05:17.444 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.444 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.444 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:17.444 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682945+0000", 2024-04-19T01:05:17.444 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000263875 2024-04-19T01:05:17.444 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.444 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.445 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:17.445 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682975+0000", 2024-04-19T01:05:17.445 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0216e-05 2024-04-19T01:05:17.445 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.445 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.445 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:17.445 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682990+0000", 2024-04-19T01:05:17.445 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5333e-05 2024-04-19T01:05:17.445 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.445 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.445 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:17.445 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692463+0000", 2024-04-19T01:05:17.445 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0094724660000000006 2024-04-19T01:05:17.445 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.446 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.446 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:17.446 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.692479+0000", 2024-04-19T01:05:17.446 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5892e-05 2024-04-19T01:05:17.447 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.447 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.447 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:17.447 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711976+0000", 2024-04-19T01:05:17.447 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019497012000000001 2024-04-19T01:05:17.447 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.447 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.447 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:17.447 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.712033+0000", 2024-04-19T01:05:17.447 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7627999999999998e-05 2024-04-19T01:05:17.447 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:17.448 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:17.448 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:17.448 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.448 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.448 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s1 59/31 ECSubWrite(tid=66, reqid=client.4604.0:423, at_version=59'51, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:05:17.448 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.541655+0000", 2024-04-19T01:05:17.448 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 47.871290002000002, 2024-04-19T01:05:17.448 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.038825746000000001, 2024-04-19T01:05:17.448 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:17.449 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:17.449 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:17.449 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:17.449 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.449 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:17.449 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.541655+0000", 2024-04-19T01:05:17.449 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:17.449 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.449 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.449 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:17.449 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.541655+0000", 2024-04-19T01:05:17.449 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:17.450 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.450 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.450 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:17.450 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.541656+0000", 2024-04-19T01:05:17.450 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1699999999999998e-07 2024-04-19T01:05:17.450 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.450 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.450 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:17.450 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.544278+0000", 2024-04-19T01:05:17.450 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002622728 2024-04-19T01:05:17.450 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.450 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.451 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:17.451 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.544314+0000", 2024-04-19T01:05:17.451 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5751999999999999e-05 2024-04-19T01:05:17.451 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.451 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.451 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:17.451 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.544338+0000", 2024-04-19T01:05:17.451 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4003e-05 2024-04-19T01:05:17.451 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.452 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.452 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:17.452 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.544577+0000", 2024-04-19T01:05:17.452 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00023931500000000001 2024-04-19T01:05:17.452 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.453 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.453 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:17.453 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.544603+0000", 2024-04-19T01:05:17.453 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5854999999999999e-05 2024-04-19T01:05:17.453 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.453 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.453 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:17.453 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.580392+0000", 2024-04-19T01:05:17.453 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.035788477999999999 2024-04-19T01:05:17.453 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.453 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.453 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:17.453 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.580481+0000", 2024-04-19T01:05:17.453 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8998000000000004e-05 2024-04-19T01:05:17.453 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:17.454 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:17.454 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:17.455 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.455 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.455 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s1 59/31 ECSubWrite(tid=67, reqid=client.4604.0:424, at_version=59'52, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:05:17.455 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.549050+0000", 2024-04-19T01:05:17.456 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 47.863894629999997, 2024-04-19T01:05:17.456 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.031505582999999997, 2024-04-19T01:05:17.456 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:17.456 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:17.456 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:17.456 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:17.456 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.456 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:17.456 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549050+0000", 2024-04-19T01:05:17.456 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:17.456 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.456 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.456 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:17.456 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549050+0000", 2024-04-19T01:05:17.456 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:17.457 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.457 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.457 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:17.457 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549052+0000", 2024-04-19T01:05:17.457 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0669999999999999e-06 2024-04-19T01:05:17.457 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.457 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.457 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:17.457 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.550013+0000", 2024-04-19T01:05:17.457 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00096065699999999996 2024-04-19T01:05:17.457 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.457 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.457 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:17.457 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.550048+0000", 2024-04-19T01:05:17.457 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4535000000000001e-05 2024-04-19T01:05:17.458 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.458 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.458 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:17.458 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.550071+0000", 2024-04-19T01:05:17.458 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3057e-05 2024-04-19T01:05:17.458 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.458 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.458 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:17.458 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.561847+0000", 2024-04-19T01:05:17.458 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.01177603 2024-04-19T01:05:17.458 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.458 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.458 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:17.458 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.561873+0000", 2024-04-19T01:05:17.459 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6679999999999999e-05 2024-04-19T01:05:17.459 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.459 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.459 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:17.459 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.580493+0000", 2024-04-19T01:05:17.459 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018619125 2024-04-19T01:05:17.459 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.459 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.459 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:17.459 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.580556+0000", 2024-04-19T01:05:17.459 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3431999999999998e-05 2024-04-19T01:05:17.460 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:17.460 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:17.460 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:17.460 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.460 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.460 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s1 79/31 ECSubWrite(tid=115, reqid=client.4604.0:591, at_version=79'90, trim_to=0'0, roll_forward_to=79'86))", 2024-04-19T01:05:17.460 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:49.658363+0000", 2024-04-19T01:05:17.460 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 27.754582197000001, 2024-04-19T01:05:17.460 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.046590765999999999, 2024-04-19T01:05:17.460 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:17.460 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:17.460 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:17.460 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:17.460 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.460 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:17.461 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658363+0000", 2024-04-19T01:05:17.461 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:17.461 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.461 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.461 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:17.461 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658363+0000", 2024-04-19T01:05:17.461 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:17.461 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.461 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.461 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:17.462 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658364+0000", 2024-04-19T01:05:17.462 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6759999999999999e-06 2024-04-19T01:05:17.462 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.462 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.463 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:17.463 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658923+0000", 2024-04-19T01:05:17.463 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00055818399999999998 2024-04-19T01:05:17.463 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.463 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.463 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:17.464 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658954+0000", 2024-04-19T01:05:17.464 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1554999999999999e-05 2024-04-19T01:05:17.464 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.464 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.464 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:17.464 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658977+0000", 2024-04-19T01:05:17.464 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3005000000000001e-05 2024-04-19T01:05:17.465 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.465 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.465 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:17.465 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.659163+0000", 2024-04-19T01:05:17.465 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018526400000000001 2024-04-19T01:05:17.465 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.465 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.465 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:17.466 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.659193+0000", 2024-04-19T01:05:17.466 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0348000000000001e-05 2024-04-19T01:05:17.466 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.466 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.466 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:17.466 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.704869+0000", 2024-04-19T01:05:17.466 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.045676481999999997 2024-04-19T01:05:17.466 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.466 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.466 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:17.466 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.704954+0000", 2024-04-19T01:05:17.466 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4252000000000006e-05 2024-04-19T01:05:17.466 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:17.466 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:17.467 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:17.467 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.467 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.468 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s1 79/31 ECSubWrite(tid=116, reqid=client.4604.0:592, at_version=79'91, trim_to=0'0, roll_forward_to=79'88))", 2024-04-19T01:05:17.468 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:49.666488+0000", 2024-04-19T01:05:17.468 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 27.746456881, 2024-04-19T01:05:17.468 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.040748924999999998, 2024-04-19T01:05:17.468 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:17.468 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:17.468 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:17.468 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:17.468 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.468 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:17.468 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.666488+0000", 2024-04-19T01:05:17.468 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:17.468 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.469 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.469 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:17.469 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.666488+0000", 2024-04-19T01:05:17.469 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:17.469 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.469 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.469 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:17.469 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.666490+0000", 2024-04-19T01:05:17.469 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0030000000000001e-06 2024-04-19T01:05:17.469 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.469 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.469 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:17.469 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.666740+0000", 2024-04-19T01:05:17.469 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00024945000000000002 2024-04-19T01:05:17.469 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.470 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.470 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:17.470 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.666785+0000", 2024-04-19T01:05:17.470 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5145000000000002e-05 2024-04-19T01:05:17.472 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.472 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.472 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:17.472 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.666810+0000", 2024-04-19T01:05:17.472 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4915000000000001e-05 2024-04-19T01:05:17.472 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.472 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.473 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:17.473 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.677635+0000", 2024-04-19T01:05:17.473 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010825293999999999 2024-04-19T01:05:17.473 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.473 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.473 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:17.473 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.677659+0000", 2024-04-19T01:05:17.473 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3896e-05 2024-04-19T01:05:17.473 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.473 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.473 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:17.473 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.707167+0000", 2024-04-19T01:05:17.473 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.029507681000000001 2024-04-19T01:05:17.474 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.474 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.474 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:17.474 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.707237+0000", 2024-04-19T01:05:17.474 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0541000000000001e-05 2024-04-19T01:05:17.474 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:17.474 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:17.474 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:17.474 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.474 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.474 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s1 79/31 ECSubWrite(tid=117, reqid=client.4604.0:593, at_version=79'92, trim_to=0'0, roll_forward_to=79'88))", 2024-04-19T01:05:17.474 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:49.674022+0000", 2024-04-19T01:05:17.474 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 27.738922716000001, 2024-04-19T01:05:17.475 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.033276779999999999, 2024-04-19T01:05:17.475 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:17.475 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:17.475 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:17.475 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:17.475 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.475 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:17.475 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.674022+0000", 2024-04-19T01:05:17.476 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:17.476 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.476 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.476 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:17.476 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.674022+0000", 2024-04-19T01:05:17.476 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:17.477 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.477 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.477 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:17.477 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.674025+0000", 2024-04-19T01:05:17.477 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.384e-06 2024-04-19T01:05:17.477 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.477 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.477 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:17.477 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.674363+0000", 2024-04-19T01:05:17.477 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00033799500000000001 2024-04-19T01:05:17.477 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.477 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.477 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:17.477 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.674412+0000", 2024-04-19T01:05:17.478 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8844000000000002e-05 2024-04-19T01:05:17.478 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.478 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.478 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:17.478 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.674436+0000", 2024-04-19T01:05:17.478 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4409e-05 2024-04-19T01:05:17.478 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.478 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.478 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:17.478 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.693200+0000", 2024-04-19T01:05:17.478 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018764268000000001 2024-04-19T01:05:17.478 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.478 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.478 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:17.479 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.693224+0000", 2024-04-19T01:05:17.479 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3646999999999999e-05 2024-04-19T01:05:17.479 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.479 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.479 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:17.479 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.707249+0000", 2024-04-19T01:05:17.479 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014025055 2024-04-19T01:05:17.479 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.479 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.479 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:17.479 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.707299+0000", 2024-04-19T01:05:17.479 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0178e-05 2024-04-19T01:05:17.479 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:17.479 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:17.479 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:17.480 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.480 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.480 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s1 81/31 ECSubWrite(tid=122, reqid=client.4604.0:626, at_version=81'96, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:05:17.480 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:51.594420+0000", 2024-04-19T01:05:17.480 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 25.818524765999999, 2024-04-19T01:05:17.480 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.041892138000000002, 2024-04-19T01:05:17.480 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:17.480 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:17.480 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:17.480 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:17.480 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.480 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:17.480 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.594420+0000", 2024-04-19T01:05:17.480 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:17.480 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.481 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.481 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:17.481 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.594420+0000", 2024-04-19T01:05:17.481 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:17.481 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.481 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.481 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:17.481 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.594421+0000", 2024-04-19T01:05:17.481 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9299999999999997e-07 2024-04-19T01:05:17.481 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.481 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.481 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:17.481 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.595515+0000", 2024-04-19T01:05:17.482 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0010942829999999999 2024-04-19T01:05:17.482 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.482 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.482 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:17.482 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.595545+0000", 2024-04-19T01:05:17.482 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.957e-05 2024-04-19T01:05:17.482 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.482 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.482 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:17.482 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.595570+0000", 2024-04-19T01:05:17.482 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5486999999999998e-05 2024-04-19T01:05:17.483 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.483 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.483 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:17.483 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.595717+0000", 2024-04-19T01:05:17.483 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000146487 2024-04-19T01:05:17.483 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.483 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.483 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:17.484 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.595739+0000", 2024-04-19T01:05:17.484 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2279000000000001e-05 2024-04-19T01:05:17.484 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.484 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.484 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:17.484 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.636238+0000", 2024-04-19T01:05:17.484 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.040499304999999999 2024-04-19T01:05:17.484 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.484 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.484 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:17.484 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.636312+0000", 2024-04-19T01:05:17.484 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3934000000000001e-05 2024-04-19T01:05:17.484 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:17.484 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:17.485 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:17.485 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.485 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.485 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s1 81/31 ECSubWrite(tid=123, reqid=client.4604.0:627, at_version=81'97, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:05:17.485 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:51.600837+0000", 2024-04-19T01:05:17.485 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 25.812108295000002, 2024-04-19T01:05:17.485 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.035530869, 2024-04-19T01:05:17.485 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:17.485 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:17.485 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:17.485 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:17.485 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.485 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:17.485 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.600837+0000", 2024-04-19T01:05:17.485 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:17.486 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.486 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.486 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:17.486 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.600837+0000", 2024-04-19T01:05:17.486 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:17.486 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.486 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.486 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:17.486 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.600839+0000", 2024-04-19T01:05:17.486 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9130000000000001e-06 2024-04-19T01:05:17.486 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.486 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.486 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:17.487 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.601189+0000", 2024-04-19T01:05:17.487 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035037499999999999 2024-04-19T01:05:17.487 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.487 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.487 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:17.487 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.601223+0000", 2024-04-19T01:05:17.487 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4215999999999999e-05 2024-04-19T01:05:17.487 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.487 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.487 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:17.487 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.601246+0000", 2024-04-19T01:05:17.487 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2775000000000001e-05 2024-04-19T01:05:17.487 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.487 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.488 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:17.488 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.613649+0000", 2024-04-19T01:05:17.488 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012402826 2024-04-19T01:05:17.488 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.488 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.488 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:17.488 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.613672+0000", 2024-04-19T01:05:17.488 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2719000000000001e-05 2024-04-19T01:05:17.488 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.488 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.488 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:17.488 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.636328+0000", 2024-04-19T01:05:17.488 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022656859000000001 2024-04-19T01:05:17.488 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.488 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.489 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:17.489 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.636368+0000", 2024-04-19T01:05:17.489 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9186000000000001e-05 2024-04-19T01:05:17.489 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:17.489 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:17.489 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:17.489 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.489 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.489 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s1 81/31 ECSubWrite(tid=124, reqid=client.4604.0:628, at_version=81'98, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:05:17.489 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:51.606688+0000", 2024-04-19T01:05:17.489 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 25.806256805, 2024-04-19T01:05:17.489 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.045868434999999999, 2024-04-19T01:05:17.489 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:17.489 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:17.490 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:17.490 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:17.490 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.490 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:17.490 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.606688+0000", 2024-04-19T01:05:17.490 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:17.490 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.490 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.490 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:17.490 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.606688+0000", 2024-04-19T01:05:17.490 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:17.490 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.490 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.490 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:17.491 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.606690+0000", 2024-04-19T01:05:17.491 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9130000000000001e-06 2024-04-19T01:05:17.491 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.491 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.491 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:17.491 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.607071+0000", 2024-04-19T01:05:17.491 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00038100699999999999 2024-04-19T01:05:17.491 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.491 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.491 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:17.491 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.607111+0000", 2024-04-19T01:05:17.491 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9915e-05 2024-04-19T01:05:17.491 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.491 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.492 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:17.492 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.607133+0000", 2024-04-19T01:05:17.492 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2252999999999999e-05 2024-04-19T01:05:17.492 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.492 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.492 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:17.492 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.626657+0000", 2024-04-19T01:05:17.492 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019523934999999999 2024-04-19T01:05:17.492 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.492 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.492 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:17.492 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.626714+0000", 2024-04-19T01:05:17.492 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6804e-05 2024-04-19T01:05:17.492 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.492 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.493 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:17.493 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.652491+0000", 2024-04-19T01:05:17.493 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025776838999999999 2024-04-19T01:05:17.493 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.493 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.493 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:17.493 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.652557+0000", 2024-04-19T01:05:17.493 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5769e-05 2024-04-19T01:05:17.493 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:17.493 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:17.493 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:17.493 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.493 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.493 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s1 81/31 ECSubWrite(tid=125, reqid=client.4604.0:629, at_version=81'99, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:05:17.494 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:51.610687+0000", 2024-04-19T01:05:17.494 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 25.802258511000002, 2024-04-19T01:05:17.494 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.041959031000000001, 2024-04-19T01:05:17.494 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:17.494 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:17.494 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:17.494 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:17.494 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.494 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:17.494 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.610687+0000", 2024-04-19T01:05:17.494 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:17.494 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.494 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.494 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:17.495 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.610687+0000", 2024-04-19T01:05:17.495 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:17.495 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.495 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.495 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:17.495 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.610688+0000", 2024-04-19T01:05:17.495 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.99e-06 2024-04-19T01:05:17.495 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.495 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.495 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:17.495 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.610734+0000", 2024-04-19T01:05:17.495 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5674999999999999e-05 2024-04-19T01:05:17.495 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.496 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.496 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:17.496 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.610779+0000", 2024-04-19T01:05:17.496 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5277e-05 2024-04-19T01:05:17.496 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.496 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.496 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:17.496 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.610803+0000", 2024-04-19T01:05:17.496 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3742000000000001e-05 2024-04-19T01:05:17.496 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.496 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.496 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:17.496 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.635814+0000", 2024-04-19T01:05:17.496 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025010674 2024-04-19T01:05:17.496 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.497 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.497 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:17.497 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.635834+0000", 2024-04-19T01:05:17.497 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9755999999999999e-05 2024-04-19T01:05:17.497 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.497 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.497 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:17.497 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.652600+0000", 2024-04-19T01:05:17.497 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016766476999999998 2024-04-19T01:05:17.497 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.497 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.497 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:17.497 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.652646+0000", 2024-04-19T01:05:17.497 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5439999999999999e-05 2024-04-19T01:05:17.498 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:17.498 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:17.498 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:17.498 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.498 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.498 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s1 81/31 ECSubWrite(tid=126, reqid=client.4604.0:631, at_version=81'101, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:05:17.498 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:51.619941+0000", 2024-04-19T01:05:17.498 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 25.793003868, 2024-04-19T01:05:17.498 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.03518276, 2024-04-19T01:05:17.498 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:17.498 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:17.498 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:17.499 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:17.499 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.499 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:17.499 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.619941+0000", 2024-04-19T01:05:17.499 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:17.499 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.499 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.499 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:17.499 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.619941+0000", 2024-04-19T01:05:17.499 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:17.499 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.499 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.499 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:17.500 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.619942+0000", 2024-04-19T01:05:17.500 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.31e-06 2024-04-19T01:05:17.500 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.500 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.500 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:17.500 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.620655+0000", 2024-04-19T01:05:17.500 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00071261299999999996 2024-04-19T01:05:17.500 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.500 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.500 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:17.500 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.620689+0000", 2024-04-19T01:05:17.500 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3809000000000001e-05 2024-04-19T01:05:17.500 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.500 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.500 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:17.501 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.620708+0000", 2024-04-19T01:05:17.501 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8929000000000001e-05 2024-04-19T01:05:17.501 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.501 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.501 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:17.501 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.636467+0000", 2024-04-19T01:05:17.501 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015758892 2024-04-19T01:05:17.501 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.501 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.501 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:17.501 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.636484+0000", 2024-04-19T01:05:17.501 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6892999999999999e-05 2024-04-19T01:05:17.501 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.502 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.502 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:17.502 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.655090+0000", 2024-04-19T01:05:17.502 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018606304000000001 2024-04-19T01:05:17.502 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.502 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.502 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:17.502 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.655124+0000", 2024-04-19T01:05:17.502 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4010000000000001e-05 2024-04-19T01:05:17.502 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:17.502 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:17.502 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:17.502 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.502 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.503 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s6 92/31 ECSubWrite(tid=202, reqid=client.4604.0:691, at_version=92'124, trim_to=0'0, roll_forward_to=89'122))", 2024-04-19T01:05:17.503 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:05:02.632596+0000", 2024-04-19T01:05:17.503 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 14.780349446000001, 2024-04-19T01:05:17.503 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.054214710999999999, 2024-04-19T01:05:17.503 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:17.503 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:17.503 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:17.503 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:17.503 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.503 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:17.503 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.632596+0000", 2024-04-19T01:05:17.503 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:17.503 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.503 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.503 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:17.504 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.632596+0000", 2024-04-19T01:05:17.504 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:17.504 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.504 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.504 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:17.504 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.632596+0000", 2024-04-19T01:05:17.504 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.13e-07 2024-04-19T01:05:17.504 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.504 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.504 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:17.504 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.633061+0000", 2024-04-19T01:05:17.504 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00046472700000000001 2024-04-19T01:05:17.504 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.504 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.504 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:17.505 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.633115+0000", 2024-04-19T01:05:17.505 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4070999999999998e-05 2024-04-19T01:05:17.505 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.505 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.505 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:17.505 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.633132+0000", 2024-04-19T01:05:17.505 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7266e-05 2024-04-19T01:05:17.505 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.505 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.505 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:17.505 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.633340+0000", 2024-04-19T01:05:17.505 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000207751 2024-04-19T01:05:17.505 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.505 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.505 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:17.506 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.633375+0000", 2024-04-19T01:05:17.506 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5311999999999997e-05 2024-04-19T01:05:17.506 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.506 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.506 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:17.506 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.686731+0000", 2024-04-19T01:05:17.506 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.053356119 2024-04-19T01:05:17.506 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.506 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.506 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:17.506 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.686810+0000", 2024-04-19T01:05:17.506 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8951999999999999e-05 2024-04-19T01:05:17.506 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:17.506 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:17.507 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:17.507 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.507 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.507 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s6 92/31 ECSubWrite(tid=203, reqid=client.4604.0:692, at_version=92'125, trim_to=0'0, roll_forward_to=89'122))", 2024-04-19T01:05:17.507 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:05:02.646905+0000", 2024-04-19T01:05:17.507 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 14.766040221000001, 2024-04-19T01:05:17.507 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.040359465999999997, 2024-04-19T01:05:17.507 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:17.507 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:17.507 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:17.507 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:17.507 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.507 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:17.507 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.646905+0000", 2024-04-19T01:05:17.507 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:17.508 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.508 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.508 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:17.508 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.646905+0000", 2024-04-19T01:05:17.508 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:17.508 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.508 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.508 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:17.508 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.646907+0000", 2024-04-19T01:05:17.508 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4159999999999998e-06 2024-04-19T01:05:17.508 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.508 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.508 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:17.508 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.647559+0000", 2024-04-19T01:05:17.509 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00065197599999999997 2024-04-19T01:05:17.509 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.509 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.509 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:17.509 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.647612+0000", 2024-04-19T01:05:17.509 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2917000000000002e-05 2024-04-19T01:05:17.509 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.509 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.509 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:17.509 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.647636+0000", 2024-04-19T01:05:17.509 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3453000000000001e-05 2024-04-19T01:05:17.509 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.509 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.509 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:17.509 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.647826+0000", 2024-04-19T01:05:17.510 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000190022 2024-04-19T01:05:17.510 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.510 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.510 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:17.510 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.647860+0000", 2024-04-19T01:05:17.510 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4038000000000002e-05 2024-04-19T01:05:17.510 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.510 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.510 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:17.510 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.687200+0000", 2024-04-19T01:05:17.510 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.039340819999999999 2024-04-19T01:05:17.510 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.510 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.510 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:17.510 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.687264+0000", 2024-04-19T01:05:17.511 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3824000000000004e-05 2024-04-19T01:05:17.511 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:17.511 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:17.511 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:17.511 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.511 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.511 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds6 102/98 ECSubWrite(tid=208, reqid=client.4604.0:758, at_version=102'78, trim_to=0'0, roll_forward_to=0'0))", 2024-04-19T01:05:17.511 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:05:12.695421+0000", 2024-04-19T01:05:17.511 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.717523967, 2024-04-19T01:05:17.511 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.033302854, 2024-04-19T01:05:17.511 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:17.511 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:17.511 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:17.511 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:17.511 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.512 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:17.512 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.695421+0000", 2024-04-19T01:05:17.512 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:17.512 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.512 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.512 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:17.512 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.695421+0000", 2024-04-19T01:05:17.512 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:17.513 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.513 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.513 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:17.513 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.695422+0000", 2024-04-19T01:05:17.513 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.093e-06 2024-04-19T01:05:17.513 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.513 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.513 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:17.513 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.695609+0000", 2024-04-19T01:05:17.513 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018649699999999999 2024-04-19T01:05:17.513 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.513 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.513 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:17.513 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.695645+0000", 2024-04-19T01:05:17.514 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6301999999999999e-05 2024-04-19T01:05:17.514 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.514 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.514 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:17.514 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.695663+0000", 2024-04-19T01:05:17.514 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7779999999999999e-05 2024-04-19T01:05:17.514 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.514 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.514 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:17.514 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.695825+0000", 2024-04-19T01:05:17.514 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000162084 2024-04-19T01:05:17.514 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.514 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.514 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:17.515 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.695842+0000", 2024-04-19T01:05:17.515 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7533e-05 2024-04-19T01:05:17.515 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.515 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.515 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:17.515 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.728648+0000", 2024-04-19T01:05:17.515 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.032805444000000003 2024-04-19T01:05:17.515 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.515 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.515 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:17.515 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.728724+0000", 2024-04-19T01:05:17.515 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6120999999999998e-05 2024-04-19T01:05:17.515 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:17.515 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:17.515 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:17.516 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.516 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.516 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds6 102/98 ECSubWrite(tid=209, reqid=client.4604.0:759, at_version=102'79, trim_to=0'0, roll_forward_to=0'0))", 2024-04-19T01:05:17.516 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:05:12.701896+0000", 2024-04-19T01:05:17.516 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.7110489290000004, 2024-04-19T01:05:17.516 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.037575628, 2024-04-19T01:05:17.516 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:17.516 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:17.516 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:17.516 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:17.516 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.516 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:17.516 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.701896+0000", 2024-04-19T01:05:17.516 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:17.516 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.517 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.517 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:17.517 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.701896+0000", 2024-04-19T01:05:17.517 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:17.517 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.517 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.517 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:17.517 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.701898+0000", 2024-04-19T01:05:17.517 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5460000000000001e-06 2024-04-19T01:05:17.517 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.517 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.517 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:17.517 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.702313+0000", 2024-04-19T01:05:17.517 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00041570700000000002 2024-04-19T01:05:17.517 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.518 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.518 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:17.518 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.702344+0000", 2024-04-19T01:05:17.518 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0405000000000001e-05 2024-04-19T01:05:17.518 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.518 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.518 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:17.518 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.702365+0000", 2024-04-19T01:05:17.518 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1628999999999999e-05 2024-04-19T01:05:17.518 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.518 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.518 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:17.518 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.708509+0000", 2024-04-19T01:05:17.518 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0061432459999999998 2024-04-19T01:05:17.519 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.519 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.519 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:17.519 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.708531+0000", 2024-04-19T01:05:17.519 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2770000000000001e-05 2024-04-19T01:05:17.519 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.519 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.519 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:17.519 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.739412+0000", 2024-04-19T01:05:17.519 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.030880317000000001 2024-04-19T01:05:17.519 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:17.519 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:17.519 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:17.519 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.739472+0000", 2024-04-19T01:05:17.519 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0007999999999999e-05 2024-04-19T01:05:17.520 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:17.520 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:17.520 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:17.520 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:17.520 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:17.520 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:05:17.521 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2024-04-19T01:05:17.522 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-19T01:05:17.522 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:05:17.518+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:05:17.524 INFO:tasks.rados.rados.0.smithi155.stdout:326: done (2 left) 2024-04-19T01:05:17.525 INFO:tasks.rados.rados.0.smithi155.stdout:327: done (1 left) 2024-04-19T01:05:17.525 INFO:tasks.rados.rados.0.smithi155.stdout:329: done (0 left) 2024-04-19T01:05:17.525 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 47 current snap is 38 2024-04-19T01:05:17.525 INFO:tasks.rados.rados.0.smithi155.stdout:330: seq_num 132 ranges {1474560=671744,2146304=573440,2719744=753664,3473408=229376} 2024-04-19T01:05:17.550 INFO:tasks.rados.rados.0.smithi155.stdout:330: writing smithi15528762-47 from 1474560 to 2146304 tid 1 2024-04-19T01:05:17.557 INFO:tasks.rados.rados.0.smithi155.stdout:330: writing smithi15528762-47 from 2146304 to 2719744 tid 2 2024-04-19T01:05:17.565 INFO:teuthology.orchestra.run.smithi033.stdout:moved 0 / 130 (0%) 2024-04-19T01:05:17.565 INFO:teuthology.orchestra.run.smithi033.stdout:avg 8.66667 2024-04-19T01:05:17.565 INFO:teuthology.orchestra.run.smithi033.stdout:stddev 2.52102 -> 2.52102 (expected baseline 2.8441) 2024-04-19T01:05:17.565 INFO:teuthology.orchestra.run.smithi033.stdout:min osd.12 with 4 -> 4 pgs (0.461538 -> 0.461538 * mean) 2024-04-19T01:05:17.566 INFO:teuthology.orchestra.run.smithi033.stdout:max osd.11 with 15 -> 15 pgs (1.73077 -> 1.73077 * mean) 2024-04-19T01:05:17.566 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:05:17.566 INFO:teuthology.orchestra.run.smithi033.stdout:oload 1000 2024-04-19T01:05:17.566 INFO:teuthology.orchestra.run.smithi033.stdout:max_change 1 2024-04-19T01:05:17.566 INFO:teuthology.orchestra.run.smithi033.stdout:max_change_osds 4 2024-04-19T01:05:17.566 INFO:teuthology.orchestra.run.smithi033.stdout:average_utilization 0.0004 2024-04-19T01:05:17.566 INFO:teuthology.orchestra.run.smithi033.stdout:overload_utilization 0.0045 2024-04-19T01:05:17.566 INFO:teuthology.orchestra.run.smithi033.stderr:no change 2024-04-19T01:05:17.577 INFO:tasks.rados.rados.0.smithi155.stdout:330: writing smithi15528762-47 from 2719744 to 3473408 tid 3 2024-04-19T01:05:17.577 INFO:tasks.rados.rados.0.smithi155.stdout:330: writing smithi15528762-47 from 3473408 to 3702784 tid 4 2024-04-19T01:05:17.577 INFO:tasks.rados.rados.0.smithi155.stdout:331: setattr oid 35 current snap is 38 2024-04-19T01:05:17.579 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 37 current snap is 38 2024-04-19T01:05:17.579 INFO:tasks.rados.rados.0.smithi155.stdout:332: seq_num 133 ranges {0=688128,688128=720896,1409024=475136,1884160=360448} 2024-04-19T01:05:17.588 INFO:tasks.rados.rados.0.smithi155.stdout:332: writing smithi15528762-37 from 0 to 688128 tid 1 2024-04-19T01:05:17.595 INFO:tasks.rados.rados.0.smithi155.stdout:332: writing smithi15528762-37 from 688128 to 1409024 tid 2 2024-04-19T01:05:17.600 INFO:tasks.rados.rados.0.smithi155.stdout:332: writing smithi15528762-37 from 1409024 to 1884160 tid 3 2024-04-19T01:05:17.604 INFO:tasks.rados.rados.0.smithi155.stdout:332: writing smithi15528762-37 from 1884160 to 2244608 tid 4 2024-04-19T01:05:17.604 INFO:tasks.rados.rados.0.smithi155.stdout:333: copy_from oid 21 from oid 28 current snap is 38 2024-04-19T01:05:17.604 INFO:tasks.rados.rados.0.smithi155.stdout:330: finishing write tid 1 to smithi15528762-47 2024-04-19T01:05:17.605 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 35 v 23 (ObjNum 88 snap 15 seq_num 88) dirty exists 2024-04-19T01:05:17.605 INFO:tasks.rados.rados.0.smithi155.stdout:330: finishing write tid 2 to smithi15528762-47 2024-04-19T01:05:17.605 INFO:tasks.rados.rados.0.smithi155.stdout:330: finishing write tid 3 to smithi15528762-47 2024-04-19T01:05:17.605 INFO:tasks.rados.rados.0.smithi155.stdout:330: finishing write tid 4 to smithi15528762-47 2024-04-19T01:05:17.605 INFO:tasks.rados.rados.0.smithi155.stdout:330: finishing write tid 5 to smithi15528762-47 2024-04-19T01:05:17.605 INFO:tasks.rados.rados.0.smithi155.stdout:331: done (3 left) 2024-04-19T01:05:17.605 INFO:tasks.rados.rados.0.smithi155.stdout:334: copy_from oid 3 from oid 14 current snap is 38 2024-04-19T01:05:17.606 INFO:tasks.rados.rados.0.smithi155.stdout:335: read oid 16 snap -1 2024-04-19T01:05:17.606 INFO:tasks.rados.rados.0.smithi155.stdout:335: expect (ObjNum 109 snap 25 seq_num 109) 2024-04-19T01:05:17.606 INFO:tasks.rados.rados.0.smithi155.stdout:336: delete oid 19 current snap is 38 2024-04-19T01:05:17.606 INFO:tasks.rados.rados.0.smithi155.stdout:330: finishing write tid 6 to smithi15528762-47 2024-04-19T01:05:17.606 INFO:tasks.rados.rados.0.smithi155.stdout:330: oid 47 updating version 0 to 102 2024-04-19T01:05:17.606 INFO:tasks.rados.rados.0.smithi155.stdout:330: oid 47 updating version 102 to 105 2024-04-19T01:05:17.606 INFO:tasks.rados.rados.0.smithi155.stdout:330: oid 47 version 105 is already newer than 103 2024-04-19T01:05:17.606 INFO:tasks.rados.rados.0.smithi155.stdout:330: oid 47 version 105 is already newer than 104 2024-04-19T01:05:17.606 INFO:tasks.rados.rados.0.smithi155.stdout:330: oid 47 updating version 105 to 106 2024-04-19T01:05:17.606 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 47 v 106 (ObjNum 132 snap 38 seq_num 132) dirty exists 2024-04-19T01:05:17.606 INFO:tasks.rados.rados.0.smithi155.stdout:330: left oid 47 (ObjNum 132 snap 38 seq_num 132) 2024-04-19T01:05:17.607 INFO:tasks.rados.rados.0.smithi155.stdout:333: finishing copy_from to smithi15528762-21 2024-04-19T01:05:17.607 INFO:tasks.rados.rados.0.smithi155.stdout:333: got expected ENOENT (src dne) 2024-04-19T01:05:17.607 INFO:tasks.rados.rados.0.smithi155.stdout:333: finishing copy_from racing read to smithi15528762-21 2024-04-19T01:05:17.612 INFO:tasks.rados.rados.0.smithi155.stdout:330: done (5 left) 2024-04-19T01:05:17.612 INFO:tasks.rados.rados.0.smithi155.stdout:333: done (4 left) 2024-04-19T01:05:17.612 INFO:tasks.rados.rados.0.smithi155.stdout:336: done (3 left) 2024-04-19T01:05:17.612 INFO:tasks.rados.rados.0.smithi155.stdout:337: delete oid 8 current snap is 38 2024-04-19T01:05:17.622 INFO:tasks.rados.rados.0.smithi155.stdout:337: done (3 left) 2024-04-19T01:05:17.622 INFO:tasks.rados.rados.0.smithi155.stdout:338: rmattr oid 28 current snap is 38 2024-04-19T01:05:17.622 INFO:tasks.rados.rados.0.smithi155.stdout:338: done (3 left) 2024-04-19T01:05:17.622 INFO:tasks.rados.rados.0.smithi155.stdout:339: rmattr oid 40 current snap is 38 2024-04-19T01:05:17.622 INFO:tasks.rados.rados.0.smithi155.stdout:339: done (3 left) 2024-04-19T01:05:17.623 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 36 current snap is 38 2024-04-19T01:05:17.623 INFO:tasks.rados.rados.0.smithi155.stdout:340: seq_num 136 ranges {589824=737280,1327104=770048,2097152=540672,2637824=245760} 2024-04-19T01:05:17.623 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:05:17.622+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:05:17.630 INFO:tasks.rados.rados.0.smithi155.stdout:340: writing smithi15528762-36 from 589824 to 1327104 tid 1 2024-04-19T01:05:17.637 INFO:tasks.rados.rados.0.smithi155.stdout:340: writing smithi15528762-36 from 1327104 to 2097152 tid 2 2024-04-19T01:05:17.641 INFO:tasks.rados.rados.0.smithi155.stdout:340: writing smithi15528762-36 from 2097152 to 2637824 tid 3 2024-04-19T01:05:17.644 INFO:tasks.rados.rados.0.smithi155.stdout:340: writing smithi15528762-36 from 2637824 to 2883584 tid 4 2024-04-19T01:05:17.644 INFO:tasks.rados.rados.0.smithi155.stdout:341: read oid 31 snap -1 2024-04-19T01:05:17.644 INFO:tasks.rados.rados.0.smithi155.stdout:341: expect (ObjNum 125 snap 35 seq_num 125) 2024-04-19T01:05:17.644 INFO:tasks.rados.rados.0.smithi155.stdout:342: snap_create 2024-04-19T01:05:17.644 INFO:tasks.rados.rados.0.smithi155.stdout:335: expect (ObjNum 109 snap 25 seq_num 109) 2024-04-19T01:05:17.716 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:05:17.716 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-04-19T01:05:17.717 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-04-19T01:05:17.717 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:05:17.719 INFO:tasks.rados.rados.0.smithi155.stdout:332: finishing write tid 1 to smithi15528762-37 2024-04-19T01:05:17.719 INFO:tasks.rados.rados.0.smithi155.stdout:332: finishing write tid 2 to smithi15528762-37 2024-04-19T01:05:17.720 INFO:tasks.rados.rados.0.smithi155.stdout:332: finishing write tid 3 to smithi15528762-37 2024-04-19T01:05:17.720 INFO:tasks.rados.rados.0.smithi155.stdout:332: finishing write tid 4 to smithi15528762-37 2024-04-19T01:05:17.720 INFO:tasks.rados.rados.0.smithi155.stdout:334: finishing copy_from racing read to smithi15528762-3 2024-04-19T01:05:17.720 INFO:tasks.rados.rados.0.smithi155.stdout:332: finishing write tid 5 to smithi15528762-37 2024-04-19T01:05:17.720 INFO:tasks.rados.rados.0.smithi155.stdout:332: finishing write tid 6 to smithi15528762-37 2024-04-19T01:05:17.720 INFO:tasks.rados.rados.0.smithi155.stdout:332: oid 37 updating version 0 to 204 2024-04-19T01:05:17.720 INFO:tasks.rados.rados.0.smithi155.stdout:332: oid 37 version 204 is already newer than 201 2024-04-19T01:05:17.720 INFO:tasks.rados.rados.0.smithi155.stdout:332: oid 37 version 204 is already newer than 200 2024-04-19T01:05:17.720 INFO:tasks.rados.rados.0.smithi155.stdout:332: oid 37 version 204 is already newer than 202 2024-04-19T01:05:17.720 INFO:tasks.rados.rados.0.smithi155.stdout:332: oid 37 version 204 is already newer than 203 2024-04-19T01:05:17.720 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 37 v 204 (ObjNum 133 snap 38 seq_num 133) dirty exists 2024-04-19T01:05:17.720 INFO:tasks.rados.rados.0.smithi155.stdout:332: left oid 37 (ObjNum 133 snap 38 seq_num 133) 2024-04-19T01:05:17.720 INFO:tasks.rados.rados.0.smithi155.stdout:341: expect (ObjNum 125 snap 35 seq_num 125) 2024-04-19T01:05:17.723 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:05:17.721+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:05:17.730 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-19T01:05:17.776 INFO:tasks.rados.rados.0.smithi155.stdout:334: finishing copy_from to smithi15528762-3 2024-04-19T01:05:17.776 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 3 v 10 (ObjNum 65 snap 6 seq_num 65) dirty exists 2024-04-19T01:05:17.776 INFO:tasks.rados.rados.0.smithi155.stdout:340: finishing write tid 1 to smithi15528762-36 2024-04-19T01:05:17.776 INFO:tasks.rados.rados.0.smithi155.stdout:340: finishing write tid 2 to smithi15528762-36 2024-04-19T01:05:17.776 INFO:tasks.rados.rados.0.smithi155.stdout:340: finishing write tid 3 to smithi15528762-36 2024-04-19T01:05:17.776 INFO:tasks.rados.rados.0.smithi155.stdout:340: finishing write tid 4 to smithi15528762-36 2024-04-19T01:05:17.776 INFO:tasks.rados.rados.0.smithi155.stdout:340: finishing write tid 5 to smithi15528762-36 2024-04-19T01:05:17.776 INFO:tasks.rados.rados.0.smithi155.stdout:340: finishing write tid 6 to smithi15528762-36 2024-04-19T01:05:17.776 INFO:tasks.rados.rados.0.smithi155.stdout:340: oid 36 updating version 0 to 74 2024-04-19T01:05:17.776 INFO:tasks.rados.rados.0.smithi155.stdout:340: oid 36 updating version 74 to 78 2024-04-19T01:05:17.776 INFO:tasks.rados.rados.0.smithi155.stdout:340: oid 36 version 78 is already newer than 76 2024-04-19T01:05:17.777 INFO:tasks.rados.rados.0.smithi155.stdout:340: oid 36 version 78 is already newer than 75 2024-04-19T01:05:17.777 INFO:tasks.rados.rados.0.smithi155.stdout:340: oid 36 version 78 is already newer than 77 2024-04-19T01:05:17.777 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 36 v 78 (ObjNum 136 snap 38 seq_num 136) dirty exists 2024-04-19T01:05:17.777 INFO:tasks.rados.rados.0.smithi155.stdout:340: left oid 36 (ObjNum 136 snap 38 seq_num 136) 2024-04-19T01:05:17.824 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:05:17.822+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:05:17.892 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:05:17.893 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-04-19T01:05:17.893 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-04-19T01:05:17.893 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-04-19T01:05:17.893 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:05:17.908 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-19T01:05:17.910 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-19T01:05:17.958 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:05:17.957+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:05:18.026 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:05:18.024+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:05:18.068 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-04-19T01:05:18.068 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T01:05:18.079 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:05:18.079 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-04-19T01:05:18.079 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-04-19T01:05:18.079 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-04-19T01:05:18.079 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.079 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s5 44/31 ECSubWrite(tid=82, reqid=client.4604.0:280, at_version=44'50, trim_to=0'0, roll_forward_to=43'48))", 2024-04-19T01:05:18.079 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.886693+0000", 2024-04-19T01:05:18.079 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.191116159000003, 2024-04-19T01:05:18.079 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.044983205999999998, 2024-04-19T01:05:18.079 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:18.079 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:18.080 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:18.080 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:18.080 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.080 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:18.080 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.886693+0000", 2024-04-19T01:05:18.080 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:18.080 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.080 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.080 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:18.080 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.886693+0000", 2024-04-19T01:05:18.080 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:18.080 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.080 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.081 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:18.081 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.886694+0000", 2024-04-19T01:05:18.081 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2699999999999998e-07 2024-04-19T01:05:18.081 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.081 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.081 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:18.081 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.887101+0000", 2024-04-19T01:05:18.081 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00040740299999999998 2024-04-19T01:05:18.081 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.081 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.081 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:18.081 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.887130+0000", 2024-04-19T01:05:18.081 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9479e-05 2024-04-19T01:05:18.081 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.081 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.082 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:18.082 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.887152+0000", 2024-04-19T01:05:18.082 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1708999999999998e-05 2024-04-19T01:05:18.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.082 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.082 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:18.082 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.887281+0000", 2024-04-19T01:05:18.082 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012845599999999999 2024-04-19T01:05:18.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.082 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.082 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:18.082 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.887296+0000", 2024-04-19T01:05:18.082 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5642999999999999e-05 2024-04-19T01:05:18.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.083 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.083 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:18.083 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.931601+0000", 2024-04-19T01:05:18.083 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.044304705 2024-04-19T01:05:18.083 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.083 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.083 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:18.083 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.931676+0000", 2024-04-19T01:05:18.083 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4983999999999999e-05 2024-04-19T01:05:18.083 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:18.084 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:18.084 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:18.084 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.084 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.084 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 59/31 ECSubWrite(tid=66, reqid=client.4604.0:423, at_version=59'51, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:05:18.084 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.544994+0000", 2024-04-19T01:05:18.084 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 48.532815221, 2024-04-19T01:05:18.084 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.042329451999999997, 2024-04-19T01:05:18.084 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:18.084 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:18.084 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:18.084 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:18.084 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.084 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:18.085 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.544994+0000", 2024-04-19T01:05:18.085 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:18.085 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.085 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.085 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:18.085 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.544994+0000", 2024-04-19T01:05:18.085 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:18.085 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.085 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.085 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:18.085 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.544994+0000", 2024-04-19T01:05:18.085 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2399999999999997e-07 2024-04-19T01:05:18.085 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.085 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.085 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:18.086 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.545710+0000", 2024-04-19T01:05:18.086 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00071517300000000005 2024-04-19T01:05:18.086 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.086 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.086 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:18.086 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.545755+0000", 2024-04-19T01:05:18.086 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5105000000000003e-05 2024-04-19T01:05:18.086 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.086 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.086 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:18.086 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.545780+0000", 2024-04-19T01:05:18.086 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5052999999999999e-05 2024-04-19T01:05:18.086 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.086 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.087 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:18.087 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.546377+0000", 2024-04-19T01:05:18.087 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000597779 2024-04-19T01:05:18.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.087 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:18.087 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.546405+0000", 2024-04-19T01:05:18.087 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7809000000000001e-05 2024-04-19T01:05:18.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.087 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:18.087 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.587263+0000", 2024-04-19T01:05:18.087 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.040857841999999998 2024-04-19T01:05:18.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.088 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:18.088 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.587323+0000", 2024-04-19T01:05:18.088 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9966999999999997e-05 2024-04-19T01:05:18.088 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:18.088 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:18.088 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:18.088 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.088 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.088 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 59/31 ECSubWrite(tid=67, reqid=client.4604.0:424, at_version=59'52, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:05:18.088 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.549026+0000", 2024-04-19T01:05:18.088 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 48.528782540000002, 2024-04-19T01:05:18.088 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.038380015000000003, 2024-04-19T01:05:18.088 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:18.088 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:18.089 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:18.089 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:18.089 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.089 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:18.089 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549026+0000", 2024-04-19T01:05:18.089 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:18.089 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.089 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.089 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:18.089 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549026+0000", 2024-04-19T01:05:18.089 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:18.089 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.089 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.089 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:18.089 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549029+0000", 2024-04-19T01:05:18.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7269999999999999e-06 2024-04-19T01:05:18.090 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.090 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.090 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:18.090 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549687+0000", 2024-04-19T01:05:18.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00065801800000000001 2024-04-19T01:05:18.090 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.090 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.090 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:18.090 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549722+0000", 2024-04-19T01:05:18.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4882000000000002e-05 2024-04-19T01:05:18.090 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.090 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.090 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:18.090 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549745+0000", 2024-04-19T01:05:18.091 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2895999999999999e-05 2024-04-19T01:05:18.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.091 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.091 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:18.091 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.568051+0000", 2024-04-19T01:05:18.091 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018306368999999999 2024-04-19T01:05:18.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.091 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.091 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:18.091 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.568076+0000", 2024-04-19T01:05:18.091 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.493e-05 2024-04-19T01:05:18.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.091 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.091 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:18.092 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.587351+0000", 2024-04-19T01:05:18.092 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019274593999999999 2024-04-19T01:05:18.092 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.092 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.092 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:18.092 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.587406+0000", 2024-04-19T01:05:18.092 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5599000000000001e-05 2024-04-19T01:05:18.092 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:18.092 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:18.093 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:18.093 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.093 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.093 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:518 3.4s0 3:232ae717:::smithi15528762-15:head [copy-from ver 52] snapc 16=[16,15,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-04-19T01:05:18.093 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:41.513636+0000", 2024-04-19T01:05:18.093 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 36.564172677999998, 2024-04-19T01:05:18.093 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.099118577999999999, 2024-04-19T01:05:18.093 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:18.093 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:18.093 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:18.093 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:05:18.094 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:05:18.094 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:18.094 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 518 2024-04-19T01:05:18.094 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.094 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:18.094 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.094 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:18.094 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513636+0000", 2024-04-19T01:05:18.094 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:18.094 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.094 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.094 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:18.094 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513636+0000", 2024-04-19T01:05:18.095 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:18.095 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.095 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.095 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:18.095 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513639+0000", 2024-04-19T01:05:18.095 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1240000000000001e-06 2024-04-19T01:05:18.095 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.095 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.095 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:18.095 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513655+0000", 2024-04-19T01:05:18.095 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6143000000000001e-05 2024-04-19T01:05:18.095 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.095 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.095 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:18.095 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513688+0000", 2024-04-19T01:05:18.096 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2396e-05 2024-04-19T01:05:18.096 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.096 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.096 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:18.096 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513705+0000", 2024-04-19T01:05:18.096 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7295999999999998e-05 2024-04-19T01:05:18.096 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.096 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.096 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:18.096 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513854+0000", 2024-04-19T01:05:18.096 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014896700000000001 2024-04-19T01:05:18.096 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.096 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.096 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op must wait for map", 2024-04-19T01:05:18.097 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513868+0000", 2024-04-19T01:05:18.097 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3604000000000001e-05 2024-04-19T01:05:18.097 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.097 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.097 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:18.097 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.519112+0000", 2024-04-19T01:05:18.097 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005243808 2024-04-19T01:05:18.097 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.097 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.097 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:05:18.097 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.519314+0000", 2024-04-19T01:05:18.097 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020256300000000001 2024-04-19T01:05:18.097 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.097 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.097 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:18.098 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.543242+0000", 2024-04-19T01:05:18.098 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023928362000000002 2024-04-19T01:05:18.098 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.098 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.098 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:18.098 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.591560+0000", 2024-04-19T01:05:18.098 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.048317225999999998 2024-04-19T01:05:18.098 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.098 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.099 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-04-19T01:05:18.100 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.612629+0000", 2024-04-19T01:05:18.100 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.021069623999999999 2024-04-19T01:05:18.100 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.100 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.100 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:18.100 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.612755+0000", 2024-04-19T01:05:18.100 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012546500000000001 2024-04-19T01:05:18.100 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:18.100 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:18.100 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:18.100 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.100 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.100 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:519 3.4s0 3:232ae717:::smithi15528762-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e71)", 2024-04-19T01:05:18.101 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:41.513738+0000", 2024-04-19T01:05:18.101 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 36.564070549, 2024-04-19T01:05:18.101 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.077425087000000004, 2024-04-19T01:05:18.101 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:18.101 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:18.101 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-04-19T01:05:18.101 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:05:18.101 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:05:18.101 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:18.101 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 519 2024-04-19T01:05:18.101 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.101 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:18.101 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.101 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:18.101 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513738+0000", 2024-04-19T01:05:18.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:18.102 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.102 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.102 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:18.102 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513738+0000", 2024-04-19T01:05:18.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:18.103 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.103 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:18.103 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513740+0000", 2024-04-19T01:05:18.103 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2300000000000001e-06 2024-04-19T01:05:18.103 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.103 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:18.103 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513746+0000", 2024-04-19T01:05:18.103 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5660000000000003e-06 2024-04-19T01:05:18.103 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.103 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:18.104 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513767+0000", 2024-04-19T01:05:18.104 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0990000000000001e-05 2024-04-19T01:05:18.104 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.104 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.104 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:18.104 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513779+0000", 2024-04-19T01:05:18.104 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2033e-05 2024-04-19T01:05:18.104 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.104 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.104 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:18.104 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513972+0000", 2024-04-19T01:05:18.104 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019274599999999999 2024-04-19T01:05:18.104 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.104 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.104 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:05:18.105 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.513984+0000", 2024-04-19T01:05:18.105 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.165e-05 2024-04-19T01:05:18.105 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.105 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.105 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:18.105 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.519641+0000", 2024-04-19T01:05:18.105 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056575640000000003 2024-04-19T01:05:18.105 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.105 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.105 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for blocked object", 2024-04-19T01:05:18.105 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.519731+0000", 2024-04-19T01:05:18.105 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9462000000000005e-05 2024-04-19T01:05:18.105 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.105 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.105 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:18.106 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.590802+0000", 2024-04-19T01:05:18.106 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.071071239999999994 2024-04-19T01:05:18.106 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.106 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.106 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:05:18.106 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.590982+0000", 2024-04-19T01:05:18.106 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017997 2024-04-19T01:05:18.106 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.106 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.106 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:18.106 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.591163+0000", 2024-04-19T01:05:18.106 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018163599999999999 2024-04-19T01:05:18.106 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:18.106 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:18.107 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:18.107 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.108 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.108 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 79/31 ECSubWrite(tid=115, reqid=client.4604.0:591, at_version=79'90, trim_to=0'0, roll_forward_to=79'86))", 2024-04-19T01:05:18.108 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:49.657669+0000", 2024-04-19T01:05:18.108 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 28.420139967000001, 2024-04-19T01:05:18.108 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.041645041000000001, 2024-04-19T01:05:18.108 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:18.108 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:18.108 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:18.108 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:18.108 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.109 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:18.109 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.657669+0000", 2024-04-19T01:05:18.109 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:18.109 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.109 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.109 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:18.109 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.657669+0000", 2024-04-19T01:05:18.109 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:18.109 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.109 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.109 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:18.109 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.657670+0000", 2024-04-19T01:05:18.109 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.424e-06 2024-04-19T01:05:18.109 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.109 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.110 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:18.110 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658272+0000", 2024-04-19T01:05:18.110 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00060176100000000003 2024-04-19T01:05:18.110 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.110 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.110 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:18.110 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658307+0000", 2024-04-19T01:05:18.110 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4820999999999997e-05 2024-04-19T01:05:18.110 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.110 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.111 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:18.111 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658331+0000", 2024-04-19T01:05:18.111 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3898999999999999e-05 2024-04-19T01:05:18.111 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.111 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.111 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:18.111 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658490+0000", 2024-04-19T01:05:18.111 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015878499999999999 2024-04-19T01:05:18.111 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.111 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.111 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:18.111 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658519+0000", 2024-04-19T01:05:18.112 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9720999999999999e-05 2024-04-19T01:05:18.112 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.112 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.113 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:18.113 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.699239+0000", 2024-04-19T01:05:18.113 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.040719568999999997 2024-04-19T01:05:18.113 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.113 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.113 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:18.113 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.699314+0000", 2024-04-19T01:05:18.113 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5061000000000005e-05 2024-04-19T01:05:18.113 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:18.113 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:18.113 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:18.113 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.113 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.113 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:605 3.4s0 3:219bff74:::smithi15528762-16:head [append 0~671744] snapc 1a=[19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-19T01:05:18.114 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:49.674586+0000", 2024-04-19T01:05:18.114 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 28.403223312000001, 2024-04-19T01:05:18.114 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.041485808999999998, 2024-04-19T01:05:18.114 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:18.114 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:18.114 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:18.114 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:05:18.114 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:05:18.114 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:18.114 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 605 2024-04-19T01:05:18.114 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.114 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:18.114 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.114 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:18.115 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.674586+0000", 2024-04-19T01:05:18.115 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:18.115 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.115 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.115 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:18.115 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.674586+0000", 2024-04-19T01:05:18.115 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:18.115 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.115 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.116 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:18.116 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.674588+0000", 2024-04-19T01:05:18.116 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7779999999999999e-06 2024-04-19T01:05:18.116 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.116 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.116 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:18.116 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.676436+0000", 2024-04-19T01:05:18.116 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0018479060000000001 2024-04-19T01:05:18.116 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.116 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.116 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:18.116 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.676485+0000", 2024-04-19T01:05:18.117 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8797000000000001e-05 2024-04-19T01:05:18.117 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.117 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.117 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:18.117 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.676509+0000", 2024-04-19T01:05:18.117 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3811999999999998e-05 2024-04-19T01:05:18.117 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.117 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.118 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:18.118 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.677380+0000", 2024-04-19T01:05:18.118 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00087160999999999998 2024-04-19T01:05:18.118 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.118 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.118 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:05:18.118 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.677579+0000", 2024-04-19T01:05:18.118 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000198402 2024-04-19T01:05:18.118 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.119 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.119 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:18.119 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.680409+0000", 2024-04-19T01:05:18.119 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0028303960000000002 2024-04-19T01:05:18.119 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.119 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.119 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:18.119 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.711082+0000", 2024-04-19T01:05:18.119 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.030672228999999999 2024-04-19T01:05:18.119 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.119 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.119 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-04-19T01:05:18.119 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.715856+0000", 2024-04-19T01:05:18.119 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0047740589999999998 2024-04-19T01:05:18.119 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.120 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.120 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:18.120 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.716071+0000", 2024-04-19T01:05:18.120 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021582 2024-04-19T01:05:18.120 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:18.120 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:18.120 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:18.120 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.120 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.120 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:606 3.4s0 3:219bff74:::smithi15528762-16:head [append 0~753664] snapc 1a=[19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-19T01:05:18.120 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:49.682434+0000", 2024-04-19T01:05:18.120 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 28.395375306999998, 2024-04-19T01:05:18.120 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.037255644999999997, 2024-04-19T01:05:18.120 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:18.120 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:18.121 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:18.121 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:05:18.121 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:05:18.121 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:18.121 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 606 2024-04-19T01:05:18.121 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.121 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:18.121 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.121 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:18.121 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.682434+0000", 2024-04-19T01:05:18.121 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:18.121 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.121 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.121 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:18.122 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.682434+0000", 2024-04-19T01:05:18.122 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:18.122 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.122 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.122 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:18.122 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.682441+0000", 2024-04-19T01:05:18.122 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5989999999999996e-06 2024-04-19T01:05:18.122 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.122 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.122 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:18.122 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.683247+0000", 2024-04-19T01:05:18.122 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00080545400000000004 2024-04-19T01:05:18.122 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.122 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.123 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:18.123 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.683274+0000", 2024-04-19T01:05:18.123 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7827999999999999e-05 2024-04-19T01:05:18.123 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.123 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.123 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:18.123 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.683292+0000", 2024-04-19T01:05:18.123 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7958999999999999e-05 2024-04-19T01:05:18.123 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.123 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.123 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:18.123 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.697174+0000", 2024-04-19T01:05:18.123 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013881881 2024-04-19T01:05:18.123 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.123 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.124 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:05:18.124 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.697346+0000", 2024-04-19T01:05:18.124 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000171898 2024-04-19T01:05:18.124 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.124 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.124 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:18.124 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.700021+0000", 2024-04-19T01:05:18.124 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0026752099999999999 2024-04-19T01:05:18.124 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.124 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.124 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:18.124 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.715555+0000", 2024-04-19T01:05:18.124 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.01553331 2024-04-19T01:05:18.124 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.124 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.125 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-04-19T01:05:18.125 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.719523+0000", 2024-04-19T01:05:18.125 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003968465 2024-04-19T01:05:18.125 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.125 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.125 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:18.126 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.719689+0000", 2024-04-19T01:05:18.126 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016604099999999999 2024-04-19T01:05:18.126 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:18.126 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:18.126 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:18.126 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.126 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.126 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:618 3.4s0 3:24fe125c:::smithi15528762-3:head [append 0~720896] snapc 1c=[1c,1b,19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-19T01:05:18.126 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:51.527932+0000", 2024-04-19T01:05:18.126 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 26.549876987000001, 2024-04-19T01:05:18.126 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.045645045000000002, 2024-04-19T01:05:18.126 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:18.127 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:18.127 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:18.127 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:05:18.127 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:05:18.127 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:18.127 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 618 2024-04-19T01:05:18.127 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.127 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:18.127 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.128 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:18.128 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.527932+0000", 2024-04-19T01:05:18.128 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:18.128 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.128 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.128 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:18.128 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.527932+0000", 2024-04-19T01:05:18.128 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:18.128 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.128 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.128 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:18.128 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.527935+0000", 2024-04-19T01:05:18.128 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.32e-06 2024-04-19T01:05:18.128 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.129 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.129 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:18.130 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.529172+0000", 2024-04-19T01:05:18.130 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0012371229999999999 2024-04-19T01:05:18.132 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.132 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.132 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:18.132 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.529205+0000", 2024-04-19T01:05:18.132 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3148999999999997e-05 2024-04-19T01:05:18.132 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.132 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.132 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:18.133 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.529229+0000", 2024-04-19T01:05:18.133 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3118000000000001e-05 2024-04-19T01:05:18.133 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.133 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.133 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:18.133 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.529364+0000", 2024-04-19T01:05:18.134 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000135833 2024-04-19T01:05:18.134 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.134 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.134 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op must wait for map", 2024-04-19T01:05:18.134 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.529378+0000", 2024-04-19T01:05:18.134 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3349999999999999e-05 2024-04-19T01:05:18.134 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.135 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.135 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:18.135 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.534487+0000", 2024-04-19T01:05:18.135 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0051091069999999999 2024-04-19T01:05:18.135 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.135 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.135 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:05:18.135 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.534645+0000", 2024-04-19T01:05:18.135 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000158579 2024-04-19T01:05:18.135 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.135 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.135 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:18.135 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.536865+0000", 2024-04-19T01:05:18.135 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022194939999999998 2024-04-19T01:05:18.136 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.136 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.136 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:18.136 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.572978+0000", 2024-04-19T01:05:18.136 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.036112559000000002 2024-04-19T01:05:18.136 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.136 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.136 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-04-19T01:05:18.136 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.573320+0000", 2024-04-19T01:05:18.136 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00034209900000000001 2024-04-19T01:05:18.136 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.136 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.136 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:18.136 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.573577+0000", 2024-04-19T01:05:18.136 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00025731400000000002 2024-04-19T01:05:18.137 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:18.137 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:18.137 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:18.137 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.137 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.137 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:620 3.4s0 3:24fe125c:::smithi15528762-3:head [append 0~360448] snapc 1c=[1c,1b,19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-19T01:05:18.137 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:51.547149+0000", 2024-04-19T01:05:18.137 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 26.530659584999999, 2024-04-19T01:05:18.137 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.040862640999999998, 2024-04-19T01:05:18.137 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:18.137 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:18.137 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:18.137 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:05:18.137 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:05:18.137 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:18.138 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 620 2024-04-19T01:05:18.138 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.138 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:18.138 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.138 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:18.138 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.547149+0000", 2024-04-19T01:05:18.138 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:18.138 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.138 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.139 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:18.139 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.547149+0000", 2024-04-19T01:05:18.139 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:18.139 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.139 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.139 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:18.139 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.547152+0000", 2024-04-19T01:05:18.139 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0790000000000001e-06 2024-04-19T01:05:18.139 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.139 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.139 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:18.140 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.547839+0000", 2024-04-19T01:05:18.140 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00068704599999999997 2024-04-19T01:05:18.140 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.140 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.140 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:18.140 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.547872+0000", 2024-04-19T01:05:18.140 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3045e-05 2024-04-19T01:05:18.140 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.140 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.140 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:18.140 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.547894+0000", 2024-04-19T01:05:18.140 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1515999999999999e-05 2024-04-19T01:05:18.140 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.140 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.141 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:18.141 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.575146+0000", 2024-04-19T01:05:18.141 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.027252327 2024-04-19T01:05:18.141 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.141 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.141 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:05:18.141 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.575446+0000", 2024-04-19T01:05:18.141 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029948799999999999 2024-04-19T01:05:18.141 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.141 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.141 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:18.141 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.576902+0000", 2024-04-19T01:05:18.141 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0014561730000000001 2024-04-19T01:05:18.141 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.141 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.142 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:18.142 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.585278+0000", 2024-04-19T01:05:18.142 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0083762349999999992 2024-04-19T01:05:18.142 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.142 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.142 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-04-19T01:05:18.142 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.587851+0000", 2024-04-19T01:05:18.142 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025729759999999998 2024-04-19T01:05:18.142 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.142 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.142 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:18.143 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.588012+0000", 2024-04-19T01:05:18.143 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016075599999999999 2024-04-19T01:05:18.143 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:18.143 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:18.143 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:18.143 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.143 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.143 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:621 3.4s0 3:24fe125c:::smithi15528762-3:head [setxattr _header (56) in=7b] snapc 1c=[1c,1b,19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-19T01:05:18.143 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:51.547930+0000", 2024-04-19T01:05:18.143 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 26.529878624999998, 2024-04-19T01:05:18.143 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.043519543000000001, 2024-04-19T01:05:18.143 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:18.143 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:18.143 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:18.145 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:05:18.146 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:05:18.146 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:18.146 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 621 2024-04-19T01:05:18.146 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.146 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:18.146 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.146 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:18.146 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.547930+0000", 2024-04-19T01:05:18.146 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:18.146 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.146 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.146 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:18.146 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.547930+0000", 2024-04-19T01:05:18.146 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:18.147 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.147 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.147 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:18.147 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.547932+0000", 2024-04-19T01:05:18.147 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.894e-06 2024-04-19T01:05:18.147 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.147 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.147 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:18.147 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.547942+0000", 2024-04-19T01:05:18.147 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0332e-05 2024-04-19T01:05:18.147 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.147 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.147 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:18.148 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.547964+0000", 2024-04-19T01:05:18.148 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1268999999999999e-05 2024-04-19T01:05:18.148 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.148 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.148 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:18.148 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.547979+0000", 2024-04-19T01:05:18.148 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5722e-05 2024-04-19T01:05:18.148 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.148 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.148 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:18.148 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.583319+0000", 2024-04-19T01:05:18.148 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.035339684000000003 2024-04-19T01:05:18.148 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.148 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.148 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:05:18.149 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.583593+0000", 2024-04-19T01:05:18.149 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00027369299999999998 2024-04-19T01:05:18.149 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.149 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.149 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:18.149 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.584632+0000", 2024-04-19T01:05:18.149 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001039531 2024-04-19T01:05:18.149 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.149 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.149 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:18.149 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.585936+0000", 2024-04-19T01:05:18.149 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001303294 2024-04-19T01:05:18.149 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.149 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.150 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-04-19T01:05:18.150 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.591233+0000", 2024-04-19T01:05:18.150 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005296935 2024-04-19T01:05:18.150 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.150 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.150 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:18.150 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.591450+0000", 2024-04-19T01:05:18.150 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021718900000000001 2024-04-19T01:05:18.150 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:18.150 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:18.150 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:18.150 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.150 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.151 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:622 3.4s0 3:24fe125c:::smithi15528762-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e81)", 2024-04-19T01:05:18.151 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:51.548002+0000", 2024-04-19T01:05:18.151 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 26.529806695000001, 2024-04-19T01:05:18.151 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.049427448999999998, 2024-04-19T01:05:18.151 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:18.151 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:18.151 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-04-19T01:05:18.151 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:05:18.151 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:05:18.151 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:18.151 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 622 2024-04-19T01:05:18.151 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.151 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:18.151 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.152 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:18.152 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.548002+0000", 2024-04-19T01:05:18.152 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:18.152 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.152 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.152 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:18.152 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.548002+0000", 2024-04-19T01:05:18.152 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:18.152 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.152 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.152 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:18.152 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.548004+0000", 2024-04-19T01:05:18.152 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.46e-06 2024-04-19T01:05:18.152 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.152 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.153 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:18.153 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.548010+0000", 2024-04-19T01:05:18.153 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7630000000000003e-06 2024-04-19T01:05:18.153 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.153 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.153 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:18.153 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.548037+0000", 2024-04-19T01:05:18.153 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6492e-05 2024-04-19T01:05:18.153 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.153 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.153 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:18.153 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.548053+0000", 2024-04-19T01:05:18.153 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6436e-05 2024-04-19T01:05:18.153 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.153 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.154 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:18.154 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.586552+0000", 2024-04-19T01:05:18.154 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.038498903000000001 2024-04-19T01:05:18.154 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.154 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.154 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-04-19T01:05:18.154 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.586701+0000", 2024-04-19T01:05:18.154 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014879300000000001 2024-04-19T01:05:18.155 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.155 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.155 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:18.155 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.592222+0000", 2024-04-19T01:05:18.155 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0055211629999999999 2024-04-19T01:05:18.155 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.155 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.155 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:05:18.155 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.592488+0000", 2024-04-19T01:05:18.155 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026554200000000001 2024-04-19T01:05:18.155 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.155 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.155 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:18.155 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.597430+0000", 2024-04-19T01:05:18.155 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0049418969999999998 2024-04-19T01:05:18.156 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:18.156 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:18.156 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:18.156 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.156 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.156 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 81/31 ECSubWrite(tid=122, reqid=client.4604.0:626, at_version=81'96, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:05:18.156 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:51.594389+0000", 2024-04-19T01:05:18.156 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 26.483420085999999, 2024-04-19T01:05:18.156 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.064667589999999997, 2024-04-19T01:05:18.156 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:18.156 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:18.156 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:18.156 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:18.156 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.157 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:18.157 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.594389+0000", 2024-04-19T01:05:18.157 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:18.157 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.157 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.157 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:18.157 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.594389+0000", 2024-04-19T01:05:18.157 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:18.157 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.158 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.158 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:18.158 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.594389+0000", 2024-04-19T01:05:18.159 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0900000000000002e-07 2024-04-19T01:05:18.159 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.159 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.159 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:18.159 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.595258+0000", 2024-04-19T01:05:18.159 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000868705 2024-04-19T01:05:18.159 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.159 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.159 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:18.160 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.595294+0000", 2024-04-19T01:05:18.160 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5685000000000001e-05 2024-04-19T01:05:18.160 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.160 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.160 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:18.160 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.595317+0000", 2024-04-19T01:05:18.160 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3527999999999999e-05 2024-04-19T01:05:18.160 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.160 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.160 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:18.160 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.595472+0000", 2024-04-19T01:05:18.160 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015491300000000001 2024-04-19T01:05:18.160 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.160 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.160 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:18.161 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.595497+0000", 2024-04-19T01:05:18.161 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4510000000000001e-05 2024-04-19T01:05:18.161 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.161 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.161 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:18.161 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.658996+0000", 2024-04-19T01:05:18.161 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.063499261000000001 2024-04-19T01:05:18.161 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.161 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.161 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:18.161 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.659056+0000", 2024-04-19T01:05:18.161 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0578999999999997e-05 2024-04-19T01:05:18.161 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:18.161 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:18.161 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:18.162 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.162 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.162 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 81/31 ECSubWrite(tid=123, reqid=client.4604.0:627, at_version=81'97, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:05:18.162 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:51.601189+0000", 2024-04-19T01:05:18.162 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 26.476620267000001, 2024-04-19T01:05:18.162 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.057910206999999998, 2024-04-19T01:05:18.163 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:18.163 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:18.163 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:18.163 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:18.163 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.163 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:18.163 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.601189+0000", 2024-04-19T01:05:18.163 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:18.163 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.163 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.163 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:18.163 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.601189+0000", 2024-04-19T01:05:18.163 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:18.163 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.164 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.164 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:18.164 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.601191+0000", 2024-04-19T01:05:18.164 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0200000000000001e-06 2024-04-19T01:05:18.164 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.164 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.164 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:18.164 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.601493+0000", 2024-04-19T01:05:18.164 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000302625 2024-04-19T01:05:18.164 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.164 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.164 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:18.164 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.601543+0000", 2024-04-19T01:05:18.165 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0016999999999999e-05 2024-04-19T01:05:18.165 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.165 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.165 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:18.165 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.601578+0000", 2024-04-19T01:05:18.165 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5079000000000001e-05 2024-04-19T01:05:18.165 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.165 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.165 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:18.165 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.609356+0000", 2024-04-19T01:05:18.165 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0077777990000000002 2024-04-19T01:05:18.166 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.166 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.166 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:18.166 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.609389+0000", 2024-04-19T01:05:18.166 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2662e-05 2024-04-19T01:05:18.166 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.166 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.166 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:18.166 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.659064+0000", 2024-04-19T01:05:18.166 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.049674836999999999 2024-04-19T01:05:18.166 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.166 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.166 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:18.166 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.659099+0000", 2024-04-19T01:05:18.167 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5167999999999999e-05 2024-04-19T01:05:18.167 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:18.167 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:18.167 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:18.167 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.167 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.167 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 81/31 ECSubWrite(tid=124, reqid=client.4604.0:628, at_version=81'98, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:05:18.167 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:51.606908+0000", 2024-04-19T01:05:18.167 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 26.470900653000001, 2024-04-19T01:05:18.167 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.052296042000000001, 2024-04-19T01:05:18.167 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:18.167 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:18.167 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:18.167 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:18.168 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.168 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:18.168 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.606908+0000", 2024-04-19T01:05:18.168 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:18.168 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.168 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.168 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:18.168 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.606908+0000", 2024-04-19T01:05:18.168 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:18.168 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.168 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.168 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:18.168 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.606910+0000", 2024-04-19T01:05:18.168 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1229999999999998e-06 2024-04-19T01:05:18.169 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.169 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.169 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:18.169 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.607105+0000", 2024-04-19T01:05:18.169 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000194293 2024-04-19T01:05:18.169 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.169 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.169 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:18.169 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.607144+0000", 2024-04-19T01:05:18.169 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9415000000000001e-05 2024-04-19T01:05:18.169 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.169 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.169 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:18.169 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.607170+0000", 2024-04-19T01:05:18.169 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5819000000000001e-05 2024-04-19T01:05:18.170 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.170 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.170 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:18.170 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.621613+0000", 2024-04-19T01:05:18.170 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014443123 2024-04-19T01:05:18.170 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.170 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.170 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:18.170 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.621631+0000", 2024-04-19T01:05:18.170 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8343e-05 2024-04-19T01:05:18.170 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.170 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.170 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:18.170 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.659115+0000", 2024-04-19T01:05:18.170 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.037483797999999999 2024-04-19T01:05:18.171 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.171 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.171 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:18.171 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.659204+0000", 2024-04-19T01:05:18.171 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9128000000000005e-05 2024-04-19T01:05:18.171 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:18.171 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:18.171 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:18.171 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.171 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.171 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 81/31 ECSubWrite(tid=125, reqid=client.4604.0:629, at_version=81'99, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:05:18.171 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:51.610690+0000", 2024-04-19T01:05:18.171 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 26.467118817999999, 2024-04-19T01:05:18.171 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.048566986999999999, 2024-04-19T01:05:18.171 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:18.172 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:18.172 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:18.172 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:18.172 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.172 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:18.172 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.610690+0000", 2024-04-19T01:05:18.172 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:18.172 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.172 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.172 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:18.172 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.610690+0000", 2024-04-19T01:05:18.172 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:18.172 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.172 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.172 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:18.173 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.610692+0000", 2024-04-19T01:05:18.173 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9640000000000002e-06 2024-04-19T01:05:18.173 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.173 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.173 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:18.173 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.610737+0000", 2024-04-19T01:05:18.173 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5290999999999997e-05 2024-04-19T01:05:18.173 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.173 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.173 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:18.173 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.610780+0000", 2024-04-19T01:05:18.173 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2633999999999999e-05 2024-04-19T01:05:18.173 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.173 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.173 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:18.174 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.610803+0000", 2024-04-19T01:05:18.174 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2792000000000001e-05 2024-04-19T01:05:18.174 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.174 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.174 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:18.174 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.632193+0000", 2024-04-19T01:05:18.175 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.021390273000000001 2024-04-19T01:05:18.175 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.176 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.176 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:18.177 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.632216+0000", 2024-04-19T01:05:18.178 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3011e-05 2024-04-19T01:05:18.178 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.178 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.179 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:18.179 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.659221+0000", 2024-04-19T01:05:18.180 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.027004759 2024-04-19T01:05:18.180 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.180 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.180 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:18.180 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.659257+0000", 2024-04-19T01:05:18.180 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6263000000000002e-05 2024-04-19T01:05:18.180 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:18.180 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:18.180 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:18.180 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.182 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.182 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s4 81/31 ECSubWrite(tid=126, reqid=client.4604.0:631, at_version=81'101, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:05:18.182 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:51.620071+0000", 2024-04-19T01:05:18.182 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 26.457738159000002, 2024-04-19T01:05:18.182 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.039719863000000001, 2024-04-19T01:05:18.182 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:18.182 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:18.182 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:18.182 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:18.182 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.182 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:18.182 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.620071+0000", 2024-04-19T01:05:18.182 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:18.183 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.183 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.183 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:18.183 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.620071+0000", 2024-04-19T01:05:18.183 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:18.183 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.183 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.183 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:18.183 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.620073+0000", 2024-04-19T01:05:18.183 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8419999999999999e-06 2024-04-19T01:05:18.183 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.184 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.184 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:18.184 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.620690+0000", 2024-04-19T01:05:18.184 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00061762799999999995 2024-04-19T01:05:18.184 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.184 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.184 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:18.184 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.620724+0000", 2024-04-19T01:05:18.184 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3422e-05 2024-04-19T01:05:18.184 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.184 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.184 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:18.185 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.620743+0000", 2024-04-19T01:05:18.185 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9405000000000001e-05 2024-04-19T01:05:18.185 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.185 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.185 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:18.185 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.632724+0000", 2024-04-19T01:05:18.185 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011981014999999999 2024-04-19T01:05:18.185 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.185 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.185 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:18.185 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.632750+0000", 2024-04-19T01:05:18.185 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6293000000000001e-05 2024-04-19T01:05:18.185 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.185 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.185 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:18.186 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.659730+0000", 2024-04-19T01:05:18.186 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.026979876999999999 2024-04-19T01:05:18.186 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.186 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.186 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:18.186 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.659791+0000", 2024-04-19T01:05:18.186 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0380999999999997e-05 2024-04-19T01:05:18.186 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:18.186 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:18.186 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:18.186 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.186 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.186 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s5 92/31 ECSubWrite(tid=202, reqid=client.4604.0:691, at_version=92'124, trim_to=0'0, roll_forward_to=89'122))", 2024-04-19T01:05:18.186 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:05:02.632382+0000", 2024-04-19T01:05:18.187 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.445426962999999, 2024-04-19T01:05:18.187 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.041632608000000002, 2024-04-19T01:05:18.187 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:18.187 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:18.187 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:18.187 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:18.187 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.187 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:18.187 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.632382+0000", 2024-04-19T01:05:18.187 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:18.187 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.187 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.187 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:18.187 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.632382+0000", 2024-04-19T01:05:18.188 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:18.188 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.188 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.188 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:18.188 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.632382+0000", 2024-04-19T01:05:18.188 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4999999999999998e-07 2024-04-19T01:05:18.188 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.188 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.188 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:18.188 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.632662+0000", 2024-04-19T01:05:18.188 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00028007100000000001 2024-04-19T01:05:18.188 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.188 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.188 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:18.188 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.632701+0000", 2024-04-19T01:05:18.189 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8958000000000002e-05 2024-04-19T01:05:18.189 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.189 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.189 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:18.189 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.632728+0000", 2024-04-19T01:05:18.189 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6336e-05 2024-04-19T01:05:18.189 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.189 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.189 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:18.189 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.632908+0000", 2024-04-19T01:05:18.189 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018037500000000001 2024-04-19T01:05:18.189 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.189 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.189 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:18.190 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.632942+0000", 2024-04-19T01:05:18.190 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3532999999999999e-05 2024-04-19T01:05:18.190 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.190 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.190 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:18.190 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.673822+0000", 2024-04-19T01:05:18.190 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.040880004999999997 2024-04-19T01:05:18.190 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.190 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.190 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:18.190 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.674015+0000", 2024-04-19T01:05:18.190 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019288000000000001 2024-04-19T01:05:18.190 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:18.190 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:18.190 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:18.191 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.191 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.191 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s5 92/31 ECSubWrite(tid=203, reqid=client.4604.0:692, at_version=92'125, trim_to=0'0, roll_forward_to=89'122))", 2024-04-19T01:05:18.191 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:05:02.646874+0000", 2024-04-19T01:05:18.191 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.430934616, 2024-04-19T01:05:18.191 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.045444988999999998, 2024-04-19T01:05:18.191 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:18.191 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:18.191 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:18.191 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:18.191 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.191 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:18.191 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.646874+0000", 2024-04-19T01:05:18.191 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:18.191 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.192 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.192 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:18.192 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.646874+0000", 2024-04-19T01:05:18.192 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:18.193 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.193 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.193 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:18.193 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.646877+0000", 2024-04-19T01:05:18.193 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4899999999999999e-06 2024-04-19T01:05:18.193 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.193 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.193 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:18.193 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.647225+0000", 2024-04-19T01:05:18.194 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00034817799999999999 2024-04-19T01:05:18.194 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.194 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.194 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:18.194 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.647259+0000", 2024-04-19T01:05:18.194 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3807999999999999e-05 2024-04-19T01:05:18.194 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.194 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.194 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:18.194 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.647284+0000", 2024-04-19T01:05:18.194 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5316e-05 2024-04-19T01:05:18.194 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.194 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.194 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:18.195 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.655476+0000", 2024-04-19T01:05:18.195 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0081919000000000002 2024-04-19T01:05:18.195 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.195 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.195 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:18.195 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.655503+0000", 2024-04-19T01:05:18.195 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6874999999999999e-05 2024-04-19T01:05:18.195 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.195 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.195 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:18.195 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.692230+0000", 2024-04-19T01:05:18.195 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.036727207999999997 2024-04-19T01:05:18.195 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.195 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.195 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:18.196 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.692319+0000", 2024-04-19T01:05:18.196 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9214000000000003e-05 2024-04-19T01:05:18.196 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:18.196 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:18.196 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:18.196 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.196 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.196 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4604.0:819 3.4s0 3:24fe125c:::smithi15528762-3:head [copy-from ver 10] snapc 27=[27,25,24,21] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-19T01:05:18.196 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:05:17.606746+0000", 2024-04-19T01:05:18.196 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.47106293700000001, 2024-04-19T01:05:18.196 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.079757058000000006, 2024-04-19T01:05:18.196 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:18.197 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:18.197 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:18.197 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-04-19T01:05:18.197 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4604", 2024-04-19T01:05:18.197 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:18.197 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 819 2024-04-19T01:05:18.197 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.197 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:18.197 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.197 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:18.197 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:17.606746+0000", 2024-04-19T01:05:18.197 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:18.197 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.197 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.197 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:18.198 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:17.606746+0000", 2024-04-19T01:05:18.198 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:18.198 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.198 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.198 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:18.198 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:17.606747+0000", 2024-04-19T01:05:18.198 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5899999999999996e-07 2024-04-19T01:05:18.198 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.198 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.198 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:18.198 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:17.606752+0000", 2024-04-19T01:05:18.198 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8200000000000002e-06 2024-04-19T01:05:18.198 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.198 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.199 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:18.199 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:17.606762+0000", 2024-04-19T01:05:18.199 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1009999999999998e-06 2024-04-19T01:05:18.199 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.199 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.199 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:18.199 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:17.606768+0000", 2024-04-19T01:05:18.199 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5830000000000003e-06 2024-04-19T01:05:18.200 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.200 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.200 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:18.200 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:17.607318+0000", 2024-04-19T01:05:18.200 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00055002499999999999 2024-04-19T01:05:18.200 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.200 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.200 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-04-19T01:05:18.200 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:17.607444+0000", 2024-04-19T01:05:18.200 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000125425 2024-04-19T01:05:18.200 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.200 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.200 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:18.200 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:17.628080+0000", 2024-04-19T01:05:18.200 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020636179000000001 2024-04-19T01:05:18.201 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.201 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.201 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:18.201 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:17.649022+0000", 2024-04-19T01:05:18.201 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020942539999999999 2024-04-19T01:05:18.201 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.201 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.201 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-04-19T01:05:18.201 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:17.686429+0000", 2024-04-19T01:05:18.201 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.037407101999999998 2024-04-19T01:05:18.201 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:18.201 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:18.201 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:18.201 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:17.686503+0000", 2024-04-19T01:05:18.201 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3623999999999998e-05 2024-04-19T01:05:18.202 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:18.202 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:18.202 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:18.202 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:18.202 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:18.202 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:05:18.203 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-04-19T01:05:18.203 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T01:05:18.204 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-04-19T01:05:18.204 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T01:05:18.205 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-04-19T01:05:18.205 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:05:18.202+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:05:18.205 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-04-19T01:05:18.206 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T01:05:18.206 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-04-19T01:05:18.206 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T01:05:18.207 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-04-19T01:05:18.207 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T01:05:18.207 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-04-19T01:05:18.207 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T01:05:18.221 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-04-19T01:05:18.221 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T01:05:18.234 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2024-04-19T01:05:18.234 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T01:05:18.257 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2024-04-19T01:05:18.258 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T01:05:18.280 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2024-04-19T01:05:18.281 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T01:05:18.304 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:05:18.304+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:05:18.305 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2024-04-19T01:05:18.305 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T01:05:18.326 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2024-04-19T01:05:18.327 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-19T01:05:18.351 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2024-04-19T01:05:18.351 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-19T01:05:18.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:05:18.369+0000 7fee10871640 0 -- 172.21.15.33:0/1528981681 >> [v2:172.21.15.118:6800/506416463,v1:172.21.15.118:6801/506416463] conn(0x7fee0c21e7d0 msgr2=0x7fee0c220bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-19T01:05:18.378 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:18.379 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:18.379 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2024-04-19T01:05:18.393 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-04-19T01:05:18.401 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2024-04-19T01:05:18.402 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-19T01:05:18.405 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:05:18.404+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:05:18.507 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:05:18.504+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:05:18.519 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:18.519 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:18.532 INFO:tasks.rados.rados.0.smithi155.stdout:332: done (5 left) 2024-04-19T01:05:18.533 INFO:tasks.rados.rados.0.smithi155.stdout:334: done (4 left) 2024-04-19T01:05:18.533 INFO:tasks.rados.rados.0.smithi155.stdout:335: done (3 left) 2024-04-19T01:05:18.533 INFO:tasks.rados.rados.0.smithi155.stdout:340: done (2 left) 2024-04-19T01:05:18.533 INFO:tasks.rados.rados.0.smithi155.stdout:341: done (1 left) 2024-04-19T01:05:18.533 INFO:tasks.rados.rados.0.smithi155.stdout:342: done (0 left) 2024-04-19T01:05:18.533 INFO:tasks.rados.rados.0.smithi155.stdout:343: rollback oid 49 current snap is 39 2024-04-19T01:05:18.533 INFO:tasks.rados.rados.0.smithi155.stdout:rollback oid 49 to 35 2024-04-19T01:05:18.533 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 32 current snap is 39 2024-04-19T01:05:18.533 INFO:tasks.rados.rados.0.smithi155.stdout:344: seq_num 137 ranges {835584=753664,1589248=786432,2375680=655360,3031040=114688} 2024-04-19T01:05:18.534 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-04-19T01:05:18.546 INFO:tasks.rados.rados.0.smithi155.stdout:344: writing smithi15528762-32 from 835584 to 1589248 tid 1 2024-04-19T01:05:18.554 INFO:tasks.rados.rados.0.smithi155.stdout:344: writing smithi15528762-32 from 1589248 to 2375680 tid 2 2024-04-19T01:05:18.559 INFO:tasks.rados.rados.0.smithi155.stdout:344: writing smithi15528762-32 from 2375680 to 3031040 tid 3 2024-04-19T01:05:18.560 INFO:tasks.rados.rados.0.smithi155.stdout:344: writing smithi15528762-32 from 3031040 to 3145728 tid 4 2024-04-19T01:05:18.560 INFO:tasks.rados.rados.0.smithi155.stdout:345: snap_remove snap 31 2024-04-19T01:05:18.608 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:05:18.608+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:05:18.664 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:18.664 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:18.679 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-19T01:05:18.710 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:05:18.705+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:05:18.811 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:05:18.809+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:05:18.811 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:18.811 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:18.825 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-19T01:05:18.912 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:05:18.912+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:05:18.948 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:18.949 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:18.962 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-19T01:05:19.052 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:05:19.052+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:05:19.116 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:05:19.112+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:05:19.125 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:19.125 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:19.139 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-19T01:05:19.216 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:05:19.216+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:05:19.308 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:19.308 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:19.316 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:05:19.316+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:05:19.322 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-19T01:05:19.417 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:05:19.414+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:05:19.488 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:19.488 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:19.503 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-19T01:05:19.518 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:05:19.517+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:05:19.539 INFO:tasks.rados.rados.0.smithi155.stdout:345: done (2 left) 2024-04-19T01:05:19.539 INFO:tasks.rados.rados.0.smithi155.stdout:346: read oid 30 snap -1 2024-04-19T01:05:19.539 INFO:tasks.rados.rados.0.smithi155.stdout:346: expect (ObjNum 127 snap 36 seq_num 127) 2024-04-19T01:05:19.539 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 14 current snap is 39 2024-04-19T01:05:19.539 INFO:tasks.rados.rados.0.smithi155.stdout:343: finishing rollback tid 0 to smithi15528762-49 2024-04-19T01:05:19.539 INFO:tasks.rados.rados.0.smithi155.stdout:347: seq_num 138 ranges {1327104=770048,2097152=540672,2637824=753664,3391488=262144} 2024-04-19T01:05:19.542 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-19T01:05:19.553 INFO:tasks.rados.rados.0.smithi155.stdout:347: writing smithi15528762-14 from 1327104 to 2097152 tid 1 2024-04-19T01:05:19.556 INFO:tasks.rados.rados.0.smithi155.stdout:347: writing smithi15528762-14 from 2097152 to 2637824 tid 2 2024-04-19T01:05:19.561 INFO:tasks.rados.rados.0.smithi155.stdout:347: writing smithi15528762-14 from 2637824 to 3391488 tid 3 2024-04-19T01:05:19.563 INFO:tasks.rados.rados.0.smithi155.stdout:347: writing smithi15528762-14 from 3391488 to 3653632 tid 4 2024-04-19T01:05:19.564 INFO:tasks.rados.rados.0.smithi155.stdout:348: read oid 43 snap 37 2024-04-19T01:05:19.564 INFO:tasks.rados.rados.0.smithi155.stdout:348: expect (ObjNum 12 snap 0 seq_num 12) 2024-04-19T01:05:19.564 INFO:tasks.rados.rados.0.smithi155.stdout:349: read oid 13 snap 37 2024-04-19T01:05:19.564 INFO:tasks.rados.rados.0.smithi155.stdout:349: expect deleted 2024-04-19T01:05:19.564 INFO:tasks.rados.rados.0.smithi155.stdout:350: rollback oid 33 current snap is 39 2024-04-19T01:05:19.564 INFO:tasks.rados.rados.0.smithi155.stdout:rollback oid 33 to 38 2024-04-19T01:05:19.564 INFO:tasks.rados.rados.0.smithi155.stdout:343: finishing rollback tid 1 to smithi15528762-49 2024-04-19T01:05:19.564 INFO:tasks.rados.rados.0.smithi155.stdout:343: finishing rollback tid 2 to smithi15528762-49 2024-04-19T01:05:19.564 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 49 v 208 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-19T01:05:19.564 INFO:tasks.rados.rados.0.smithi155.stdout:344: finishing write tid 1 to smithi15528762-32 2024-04-19T01:05:19.564 INFO:tasks.rados.rados.0.smithi155.stdout:344: finishing write tid 2 to smithi15528762-32 2024-04-19T01:05:19.564 INFO:tasks.rados.rados.0.smithi155.stdout:344: finishing write tid 3 to smithi15528762-32 2024-04-19T01:05:19.564 INFO:tasks.rados.rados.0.smithi155.stdout:343: done (6 left) 2024-04-19T01:05:19.564 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 11 current snap is 39 2024-04-19T01:05:19.564 INFO:tasks.rados.rados.0.smithi155.stdout:351: seq_num 139 ranges {0=786432,786432=688128,1474560=475136,1949696=393216} 2024-04-19T01:05:19.568 INFO:tasks.rados.rados.0.smithi155.stdout:351: writing smithi15528762-11 from 0 to 786432 tid 1 2024-04-19T01:05:19.574 INFO:tasks.rados.rados.0.smithi155.stdout:351: writing smithi15528762-11 from 786432 to 1474560 tid 2 2024-04-19T01:05:19.578 INFO:tasks.rados.rados.0.smithi155.stdout:351: writing smithi15528762-11 from 1474560 to 1949696 tid 3 2024-04-19T01:05:19.579 INFO:tasks.rados.rados.0.smithi155.stdout:351: writing smithi15528762-11 from 1949696 to 2342912 tid 4 2024-04-19T01:05:19.580 INFO:tasks.rados.rados.0.smithi155.stdout:352: read oid 48 snap -1 2024-04-19T01:05:19.580 INFO:tasks.rados.rados.0.smithi155.stdout:352: expect (ObjNum 47 snap 0 seq_num 47) 2024-04-19T01:05:19.580 INFO:tasks.rados.rados.0.smithi155.stdout:344: finishing write tid 4 to smithi15528762-32 2024-04-19T01:05:19.580 INFO:tasks.rados.rados.0.smithi155.stdout:344: finishing write tid 5 to smithi15528762-32 2024-04-19T01:05:19.580 INFO:tasks.rados.rados.0.smithi155.stdout:344: finishing write tid 6 to smithi15528762-32 2024-04-19T01:05:19.580 INFO:tasks.rados.rados.0.smithi155.stdout:344: oid 32 updating version 0 to 142 2024-04-19T01:05:19.580 INFO:tasks.rados.rados.0.smithi155.stdout:344: oid 32 version 142 is already newer than 138 2024-04-19T01:05:19.580 INFO:tasks.rados.rados.0.smithi155.stdout:344: oid 32 version 142 is already newer than 141 2024-04-19T01:05:19.580 INFO:tasks.rados.rados.0.smithi155.stdout:344: oid 32 version 142 is already newer than 139 2024-04-19T01:05:19.580 INFO:tasks.rados.rados.0.smithi155.stdout:344: oid 32 version 142 is already newer than 140 2024-04-19T01:05:19.580 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 32 v 142 (ObjNum 137 snap 39 seq_num 137) dirty exists 2024-04-19T01:05:19.580 INFO:tasks.rados.rados.0.smithi155.stdout:344: left oid 32 (ObjNum 137 snap 39 seq_num 137) 2024-04-19T01:05:19.580 INFO:tasks.rados.rados.0.smithi155.stdout:350: finishing rollback tid 0 to smithi15528762-33 2024-04-19T01:05:19.580 INFO:tasks.rados.rados.0.smithi155.stdout:344: done (7 left) 2024-04-19T01:05:19.581 INFO:tasks.rados.rados.0.smithi155.stdout:349: done (6 left) 2024-04-19T01:05:19.581 INFO:tasks.rados.rados.0.smithi155.stdout:353: snap_create 2024-04-19T01:05:19.595 INFO:tasks.rados.rados.0.smithi155.stdout:352: expect (ObjNum 47 snap 0 seq_num 47) 2024-04-19T01:05:19.619 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:05:19.616+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:05:19.622 INFO:tasks.rados.rados.0.smithi155.stdout:350: finishing rollback tid 1 to smithi15528762-33 2024-04-19T01:05:19.623 INFO:tasks.rados.rados.0.smithi155.stdout:350: finishing rollback tid 2 to smithi15528762-33 2024-04-19T01:05:19.623 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 33 v 65 (ObjNum 289 snap 0 seq_num 0) dirty exists 2024-04-19T01:05:19.623 INFO:tasks.rados.rados.0.smithi155.stdout:348: expect (ObjNum 12 snap 0 seq_num 12) 2024-04-19T01:05:19.627 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:05:19.624+0000 7fe0195d2640 -1 snap_mapper.get_next_objects_to_trim::New Clone-Objects were added to Snap 21 after trimming was started 2024-04-19T01:05:19.631 INFO:tasks.rados.rados.0.smithi155.stdout:351: finishing write tid 1 to smithi15528762-11 2024-04-19T01:05:19.631 INFO:tasks.rados.rados.0.smithi155.stdout:351: finishing write tid 2 to smithi15528762-11 2024-04-19T01:05:19.631 INFO:tasks.rados.rados.0.smithi155.stdout:351: finishing write tid 3 to smithi15528762-11 2024-04-19T01:05:19.631 INFO:tasks.rados.rados.0.smithi155.stdout:351: finishing write tid 4 to smithi15528762-11 2024-04-19T01:05:19.631 INFO:tasks.rados.rados.0.smithi155.stdout:351: finishing write tid 5 to smithi15528762-11 2024-04-19T01:05:19.631 INFO:tasks.rados.rados.0.smithi155.stdout:347: finishing write tid 1 to smithi15528762-14 2024-04-19T01:05:19.631 INFO:tasks.rados.rados.0.smithi155.stdout:351: finishing write tid 6 to smithi15528762-11 2024-04-19T01:05:19.631 INFO:tasks.rados.rados.0.smithi155.stdout:351: oid 11 updating version 0 to 51 2024-04-19T01:05:19.631 INFO:tasks.rados.rados.0.smithi155.stdout:351: oid 11 updating version 51 to 52 2024-04-19T01:05:19.632 INFO:tasks.rados.rados.0.smithi155.stdout:351: oid 11 updating version 52 to 53 2024-04-19T01:05:19.632 INFO:tasks.rados.rados.0.smithi155.stdout:351: oid 11 version 53 is already newer than 49 2024-04-19T01:05:19.632 INFO:tasks.rados.rados.0.smithi155.stdout:351: oid 11 version 53 is already newer than 50 2024-04-19T01:05:19.632 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 11 v 53 (ObjNum 139 snap 39 seq_num 139) dirty exists 2024-04-19T01:05:19.632 INFO:tasks.rados.rados.0.smithi155.stdout:351: left oid 11 (ObjNum 139 snap 39 seq_num 139) 2024-04-19T01:05:19.632 INFO:tasks.rados.rados.0.smithi155.stdout:347: finishing write tid 2 to smithi15528762-14 2024-04-19T01:05:19.632 INFO:tasks.rados.rados.0.smithi155.stdout:347: finishing write tid 3 to smithi15528762-14 2024-04-19T01:05:19.632 INFO:tasks.rados.rados.0.smithi155.stdout:347: finishing write tid 4 to smithi15528762-14 2024-04-19T01:05:19.632 INFO:tasks.rados.rados.0.smithi155.stdout:347: finishing write tid 5 to smithi15528762-14 2024-04-19T01:05:19.633 INFO:tasks.rados.rados.0.smithi155.stdout:347: finishing write tid 6 to smithi15528762-14 2024-04-19T01:05:19.633 INFO:tasks.rados.rados.0.smithi155.stdout:347: oid 14 updating version 0 to 83 2024-04-19T01:05:19.633 INFO:tasks.rados.rados.0.smithi155.stdout:347: oid 14 updating version 83 to 85 2024-04-19T01:05:19.633 INFO:tasks.rados.rados.0.smithi155.stdout:347: oid 14 updating version 85 to 86 2024-04-19T01:05:19.633 INFO:tasks.rados.rados.0.smithi155.stdout:347: oid 14 updating version 86 to 87 2024-04-19T01:05:19.633 INFO:tasks.rados.rados.0.smithi155.stdout:347: oid 14 version 87 is already newer than 84 2024-04-19T01:05:19.633 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 14 v 87 (ObjNum 138 snap 39 seq_num 138) dirty exists 2024-04-19T01:05:19.633 INFO:tasks.rados.rados.0.smithi155.stdout:347: left oid 14 (ObjNum 138 snap 39 seq_num 138) 2024-04-19T01:05:19.641 INFO:tasks.rados.rados.0.smithi155.stdout:346: expect (ObjNum 127 snap 36 seq_num 127) 2024-04-19T01:05:19.655 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:19.655 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:19.669 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-19T01:05:19.719 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:05:19.720+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:05:19.821 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:05:19.820+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:05:19.826 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:19.826 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:19.838 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-19T01:05:19.921 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:05:19.920+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:05:19.962 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:19.962 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:19.975 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-19T01:05:20.022 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:05:20.018+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:05:20.123 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:05:20.124+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:05:20.138 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:20.138 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:20.151 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-04-19T01:05:20.225 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:05:20.221+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:05:20.268 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:20.268 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:20.282 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-04-19T01:05:20.325 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:05:20.322+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:05:20.401 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:20.401 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:20.415 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-04-19T01:05:20.424 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-19T01:05:20.428 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:05:20.421+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:05:20.527 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:05:20.525+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:05:20.531 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2024-04-19T01:05:20.534 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:20.534 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:20.538 INFO:tasks.rados.rados.0.smithi155.stdout:346: done (6 left) 2024-04-19T01:05:20.538 INFO:tasks.rados.rados.0.smithi155.stdout:347: done (5 left) 2024-04-19T01:05:20.538 INFO:tasks.rados.rados.0.smithi155.stdout:348: done (4 left) 2024-04-19T01:05:20.538 INFO:tasks.rados.rados.0.smithi155.stdout:350: done (3 left) 2024-04-19T01:05:20.538 INFO:tasks.rados.rados.0.smithi155.stdout:351: done (2 left) 2024-04-19T01:05:20.539 INFO:tasks.rados.rados.0.smithi155.stdout:352: done (1 left) 2024-04-19T01:05:20.539 INFO:tasks.rados.rados.0.smithi155.stdout:353: done (0 left) 2024-04-19T01:05:20.539 INFO:tasks.rados.rados.0.smithi155.stdout:354: snap_create 2024-04-19T01:05:20.548 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-19T01:05:20.586 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-04-19T01:05:20.587 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T01:05:20.607 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-04-19T01:05:20.608 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T01:05:20.628 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:05:20.625+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:05:20.628 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-04-19T01:05:20.629 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T01:05:20.651 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-04-19T01:05:20.651 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T01:05:20.668 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:05:20.668 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:05:20.668 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2024-04-19T01:05:20.668 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:05:20.669 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-04-19T01:05:20.669 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T01:05:20.683 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-19T01:05:20.691 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-04-19T01:05:20.692 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T01:05:20.714 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-04-19T01:05:20.715 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T01:05:20.729 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:05:20.725+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:05:20.738 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-04-19T01:05:20.738 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T01:05:20.758 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2024-04-19T01:05:20.758 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T01:05:20.781 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2024-04-19T01:05:20.782 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T01:05:20.805 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2024-04-19T01:05:20.805 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T01:05:20.810 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:05:20.810 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:05:20.810 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2024-04-19T01:05:20.810 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2024-04-19T01:05:20.810 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:05:20.825 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-19T01:05:20.828 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2024-04-19T01:05:20.829 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T01:05:20.831 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:05:20.828+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:05:20.848 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2024-04-19T01:05:20.848 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-19T01:05:20.872 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2024-04-19T01:05:20.873 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-19T01:05:20.895 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2024-04-19T01:05:20.896 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-19T01:05:20.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:05:20.917+0000 7f3d56d9d640 0 -- 172.21.15.33:0/906989766 >> [v2:172.21.15.155:6807/741132602,v1:172.21.15.155:6810/741132602] conn(0x7f3d582269a0 msgr2=0x7f3d58228d90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-19T01:05:20.924 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2024-04-19T01:05:20.932 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:05:20.932+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:05:20.952 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:05:20.952 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2024-04-19T01:05:20.952 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2024-04-19T01:05:20.952 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-04-19T01:05:20.953 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.953 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:263 3.fs0 3:f8a60714:::smithi15528762-14:head [append 0~720896] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:05:20.953 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.990535+0000", 2024-04-19T01:05:20.953 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 66.959395096999998, 2024-04-19T01:05:20.953 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.043978092000000003, 2024-04-19T01:05:20.953 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:20.953 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:20.953 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:20.953 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:20.953 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:20.953 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:20.953 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 263 2024-04-19T01:05:20.953 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.953 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:20.953 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.954 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:20.954 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.990535+0000", 2024-04-19T01:05:20.954 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:20.954 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.954 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.954 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:20.954 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.990535+0000", 2024-04-19T01:05:20.954 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:20.954 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.954 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.954 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:20.954 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.990537+0000", 2024-04-19T01:05:20.954 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4160000000000001e-06 2024-04-19T01:05:20.955 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.955 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.955 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:20.955 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991425+0000", 2024-04-19T01:05:20.955 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00088775099999999999 2024-04-19T01:05:20.955 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.955 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.955 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:20.955 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991440+0000", 2024-04-19T01:05:20.955 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5305999999999999e-05 2024-04-19T01:05:20.955 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.955 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.955 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:20.955 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991447+0000", 2024-04-19T01:05:20.955 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.4760000000000001e-06 2024-04-19T01:05:20.956 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.956 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.956 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:20.956 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991516+0000", 2024-04-19T01:05:20.956 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.9041999999999994e-05 2024-04-19T01:05:20.956 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.956 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.957 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:20.957 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.991620+0000", 2024-04-19T01:05:20.957 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000103282 2024-04-19T01:05:20.957 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.957 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.957 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:20.957 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993018+0000", 2024-04-19T01:05:20.957 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001397976 2024-04-19T01:05:20.957 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.957 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.957 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:20.958 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.004440+0000", 2024-04-19T01:05:20.958 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011422261 2024-04-19T01:05:20.958 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.958 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.958 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:20.958 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.034454+0000", 2024-04-19T01:05:20.958 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.030013782999999999 2024-04-19T01:05:20.958 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.958 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.958 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:20.959 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.034513+0000", 2024-04-19T01:05:20.959 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.9799000000000001e-05 2024-04-19T01:05:20.959 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:20.959 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:20.959 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:20.959 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.959 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.959 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:264 3.fs0 3:f8a60714:::smithi15528762-14:head [append 0~376832] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:05:20.959 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.992666+0000", 2024-04-19T01:05:20.959 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 66.957264324999997, 2024-04-19T01:05:20.959 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.041981560000000001, 2024-04-19T01:05:20.959 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:20.959 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:20.959 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:20.960 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:20.960 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:20.960 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:20.960 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 264 2024-04-19T01:05:20.960 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.960 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:20.960 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.960 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:20.960 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.992666+0000", 2024-04-19T01:05:20.960 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:20.960 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.960 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.960 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:20.960 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.992666+0000", 2024-04-19T01:05:20.960 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:20.961 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.961 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.961 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:20.961 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.992668+0000", 2024-04-19T01:05:20.961 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.483e-06 2024-04-19T01:05:20.961 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.961 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.961 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:20.961 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993021+0000", 2024-04-19T01:05:20.961 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000352958 2024-04-19T01:05:20.961 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.961 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.961 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:20.962 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993031+0000", 2024-04-19T01:05:20.962 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0723000000000001e-05 2024-04-19T01:05:20.962 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.962 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.962 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:20.962 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993040+0000", 2024-04-19T01:05:20.962 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.507e-06 2024-04-19T01:05:20.963 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.963 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.963 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:20.963 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.000363+0000", 2024-04-19T01:05:20.963 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0073236819999999998 2024-04-19T01:05:20.963 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.963 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.963 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:20.963 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.000482+0000", 2024-04-19T01:05:20.963 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011817299999999999 2024-04-19T01:05:20.963 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.963 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.963 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:20.963 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.001332+0000", 2024-04-19T01:05:20.964 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000849886 2024-04-19T01:05:20.964 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.964 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.964 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:20.964 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.006292+0000", 2024-04-19T01:05:20.964 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0049605090000000001 2024-04-19T01:05:20.964 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.964 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.964 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:20.964 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.034602+0000", 2024-04-19T01:05:20.964 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.028310076999999999 2024-04-19T01:05:20.964 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.964 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.964 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:20.965 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.034648+0000", 2024-04-19T01:05:20.965 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5562000000000002e-05 2024-04-19T01:05:20.965 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:20.965 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:20.965 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:20.965 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.965 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.965 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:265 3.fs0 3:f8a60714:::smithi15528762-14:head [setxattr _header (56) in=7b] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:05:20.965 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.993067+0000", 2024-04-19T01:05:20.965 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 66.956863087000002, 2024-04-19T01:05:20.965 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.041773988999999997, 2024-04-19T01:05:20.965 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:20.965 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:20.965 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:20.966 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:20.966 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:20.966 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:20.966 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 265 2024-04-19T01:05:20.966 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.966 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:20.966 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.966 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:20.966 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993067+0000", 2024-04-19T01:05:20.966 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:20.966 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.966 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.966 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:20.966 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993067+0000", 2024-04-19T01:05:20.967 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:20.967 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.967 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.967 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:20.967 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993068+0000", 2024-04-19T01:05:20.967 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2699999999999999e-07 2024-04-19T01:05:20.967 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.967 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.967 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:20.967 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993072+0000", 2024-04-19T01:05:20.967 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0539999999999996e-06 2024-04-19T01:05:20.967 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.967 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.968 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:20.968 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993080+0000", 2024-04-19T01:05:20.968 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6140000000000003e-06 2024-04-19T01:05:20.968 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.968 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.968 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:20.968 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993119+0000", 2024-04-19T01:05:20.968 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8754000000000002e-05 2024-04-19T01:05:20.968 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.968 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.968 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:20.968 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.004460+0000", 2024-04-19T01:05:20.968 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011341188 2024-04-19T01:05:20.969 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.969 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.969 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:20.969 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.004518+0000", 2024-04-19T01:05:20.969 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8306e-05 2024-04-19T01:05:20.969 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.969 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.969 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:20.969 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.004774+0000", 2024-04-19T01:05:20.969 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00025565400000000002 2024-04-19T01:05:20.969 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.969 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.969 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:20.969 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.006318+0000", 2024-04-19T01:05:20.969 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0015444409999999999 2024-04-19T01:05:20.970 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.970 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.970 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:20.970 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.034787+0000", 2024-04-19T01:05:20.970 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.028468566000000001 2024-04-19T01:05:20.970 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.970 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.970 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:20.970 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.034841+0000", 2024-04-19T01:05:20.970 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4685000000000002e-05 2024-04-19T01:05:20.970 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:20.970 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:20.970 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:20.970 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.971 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.971 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:266 3.fs0 3:f8a60714:::smithi15528762-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:05:20.971 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.993144+0000", 2024-04-19T01:05:20.971 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 66.956786034000004, 2024-04-19T01:05:20.971 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.042235719999999997, 2024-04-19T01:05:20.971 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:20.971 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:20.971 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:05:20.971 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:20.971 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:20.971 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:20.971 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 266 2024-04-19T01:05:20.971 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.971 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:20.972 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.972 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:20.972 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993144+0000", 2024-04-19T01:05:20.972 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:20.972 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.972 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.972 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:20.973 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993144+0000", 2024-04-19T01:05:20.973 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:20.973 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.973 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.973 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:20.973 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993145+0000", 2024-04-19T01:05:20.973 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5099999999999999e-07 2024-04-19T01:05:20.973 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.973 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.973 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:20.973 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993149+0000", 2024-04-19T01:05:20.973 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4539999999999999e-06 2024-04-19T01:05:20.973 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.973 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.974 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:20.974 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993157+0000", 2024-04-19T01:05:20.974 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.6349999999999995e-06 2024-04-19T01:05:20.974 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.974 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.974 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:20.974 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.993166+0000", 2024-04-19T01:05:20.974 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2709999999999998e-06 2024-04-19T01:05:20.974 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.974 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.974 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:20.974 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.004924+0000", 2024-04-19T01:05:20.974 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011757834 2024-04-19T01:05:20.974 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.975 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.975 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:05:20.975 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.004971+0000", 2024-04-19T01:05:20.975 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6947000000000003e-05 2024-04-19T01:05:20.975 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.975 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.975 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:20.975 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.035366+0000", 2024-04-19T01:05:20.975 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.03039501 2024-04-19T01:05:20.975 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.975 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.975 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:20.975 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.035380+0000", 2024-04-19T01:05:20.975 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3918e-05 2024-04-19T01:05:20.975 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:20.976 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:20.976 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:20.976 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.976 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.976 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:317 3.fs0 3:f23900a8:::smithi15528762-6:head [append 0~475136] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:05:20.976 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.601883+0000", 2024-04-19T01:05:20.976 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 63.348047055999999, 2024-04-19T01:05:20.976 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.029134103000000001, 2024-04-19T01:05:20.976 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:20.976 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:20.976 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:20.976 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:20.976 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:20.976 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:20.976 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 317 2024-04-19T01:05:20.977 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.977 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:20.977 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.977 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:20.977 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.601883+0000", 2024-04-19T01:05:20.977 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:20.977 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.977 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.977 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:20.977 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.601883+0000", 2024-04-19T01:05:20.977 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:20.977 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.977 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.977 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:20.977 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.601886+0000", 2024-04-19T01:05:20.978 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.2709999999999999e-06 2024-04-19T01:05:20.978 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.978 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.978 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:20.978 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.602696+0000", 2024-04-19T01:05:20.978 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00080995700000000004 2024-04-19T01:05:20.978 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.978 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.978 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:20.978 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.602714+0000", 2024-04-19T01:05:20.978 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8168999999999998e-05 2024-04-19T01:05:20.978 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.978 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.978 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:20.978 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.602721+0000", 2024-04-19T01:05:20.979 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.554e-06 2024-04-19T01:05:20.979 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.979 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.979 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:20.979 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.602799+0000", 2024-04-19T01:05:20.979 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7804000000000002e-05 2024-04-19T01:05:20.979 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.979 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.979 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:20.979 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.602920+0000", 2024-04-19T01:05:20.979 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012105299999999999 2024-04-19T01:05:20.979 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.979 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.979 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:20.979 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.604215+0000", 2024-04-19T01:05:20.980 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0012944129999999999 2024-04-19T01:05:20.980 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.980 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.980 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:20.980 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.609630+0000", 2024-04-19T01:05:20.980 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0054153789999999997 2024-04-19T01:05:20.980 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.980 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.980 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:20.980 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.630924+0000", 2024-04-19T01:05:20.980 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021294306999999998 2024-04-19T01:05:20.980 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.980 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.980 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:20.980 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.631018+0000", 2024-04-19T01:05:20.981 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.3195999999999993e-05 2024-04-19T01:05:20.981 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:20.981 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:20.982 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:20.982 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.982 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.982 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:318 3.fs0 3:f23900a8:::smithi15528762-6:head [append 0~622592] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:05:20.982 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.609539+0000", 2024-04-19T01:05:20.982 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 63.340391461999999, 2024-04-19T01:05:20.982 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021603679000000001, 2024-04-19T01:05:20.982 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:20.982 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:20.982 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:20.982 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:20.983 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:20.983 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:20.983 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 318 2024-04-19T01:05:20.983 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.983 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:20.983 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.983 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:20.983 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.609539+0000", 2024-04-19T01:05:20.983 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:20.983 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.983 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.983 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:20.984 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.609539+0000", 2024-04-19T01:05:20.984 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:20.984 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.984 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.984 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:20.984 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.609541+0000", 2024-04-19T01:05:20.984 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.855e-06 2024-04-19T01:05:20.984 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.985 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.985 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:20.985 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610203+0000", 2024-04-19T01:05:20.985 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00066190200000000002 2024-04-19T01:05:20.985 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.985 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.985 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:20.985 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610215+0000", 2024-04-19T01:05:20.985 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2143e-05 2024-04-19T01:05:20.986 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.986 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.986 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:20.986 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610221+0000", 2024-04-19T01:05:20.986 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4590000000000003e-06 2024-04-19T01:05:20.986 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.986 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.986 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:20.987 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610283+0000", 2024-04-19T01:05:20.987 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1322999999999995e-05 2024-04-19T01:05:20.987 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.987 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.987 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:20.987 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.610394+0000", 2024-04-19T01:05:20.987 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000111044 2024-04-19T01:05:20.987 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.988 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.988 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:20.988 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611327+0000", 2024-04-19T01:05:20.988 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00093369799999999995 2024-04-19T01:05:20.988 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.988 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.988 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:20.988 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.623095+0000", 2024-04-19T01:05:20.988 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011767454 2024-04-19T01:05:20.989 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.989 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.989 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:20.989 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.631101+0000", 2024-04-19T01:05:20.989 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0080059889999999998 2024-04-19T01:05:20.989 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.989 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.989 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:20.990 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.631143+0000", 2024-04-19T01:05:20.990 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1811999999999999e-05 2024-04-19T01:05:20.990 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:20.990 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:20.990 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:20.990 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.990 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.991 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:319 3.fs0 3:f23900a8:::smithi15528762-6:head [append 0~147456] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:05:20.991 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.611402+0000", 2024-04-19T01:05:20.991 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 63.338528498999999, 2024-04-19T01:05:20.991 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.020301099, 2024-04-19T01:05:20.991 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:20.991 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:20.991 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:20.991 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:20.991 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:20.992 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:20.992 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 319 2024-04-19T01:05:20.992 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.992 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:20.992 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.992 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:20.992 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611402+0000", 2024-04-19T01:05:20.992 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:20.992 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.993 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.993 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:20.993 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611402+0000", 2024-04-19T01:05:20.993 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:20.993 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.993 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.993 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:20.993 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611404+0000", 2024-04-19T01:05:20.994 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.52e-06 2024-04-19T01:05:20.994 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.994 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.994 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:20.994 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611632+0000", 2024-04-19T01:05:20.994 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000227526 2024-04-19T01:05:20.994 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.994 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.995 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:20.995 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611660+0000", 2024-04-19T01:05:20.995 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7841999999999999e-05 2024-04-19T01:05:20.995 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.995 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.995 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:20.995 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611678+0000", 2024-04-19T01:05:20.995 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8479e-05 2024-04-19T01:05:20.995 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.996 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.996 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:20.996 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.620224+0000", 2024-04-19T01:05:20.996 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0085460999999999992 2024-04-19T01:05:20.996 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.996 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.996 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:20.996 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.620334+0000", 2024-04-19T01:05:20.997 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010964 2024-04-19T01:05:20.997 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.997 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.997 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:20.997 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.620982+0000", 2024-04-19T01:05:20.997 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00064788699999999999 2024-04-19T01:05:20.997 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.997 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.998 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:20.998 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.623230+0000", 2024-04-19T01:05:20.998 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0022481950000000001 2024-04-19T01:05:20.998 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.998 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.998 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:20.998 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.631664+0000", 2024-04-19T01:05:20.998 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0084339810000000001 2024-04-19T01:05:20.998 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:20.998 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:20.999 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:20.999 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.631703+0000", 2024-04-19T01:05:20.999 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8928999999999999e-05 2024-04-19T01:05:20.999 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:20.999 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:20.999 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:20.999 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.000 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.000 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:320 3.fs0 3:f23900a8:::smithi15528762-6:head [setxattr _header (55) in=7b] snapc 9=[9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:05:21.000 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.611709+0000", 2024-04-19T01:05:21.000 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 63.338221005000001, 2024-04-19T01:05:21.000 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.020427773, 2024-04-19T01:05:21.000 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:21.000 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:21.001 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:21.001 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:21.001 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:21.001 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:21.001 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 320 2024-04-19T01:05:21.001 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.001 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:21.002 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.002 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:21.002 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611709+0000", 2024-04-19T01:05:21.002 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.002 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.002 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.003 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:21.003 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611709+0000", 2024-04-19T01:05:21.003 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.003 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.003 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.004 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:21.004 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611711+0000", 2024-04-19T01:05:21.004 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.451e-06 2024-04-19T01:05:21.004 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.004 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.004 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:21.005 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611718+0000", 2024-04-19T01:05:21.005 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5800000000000003e-06 2024-04-19T01:05:21.005 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.005 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.005 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:21.005 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611736+0000", 2024-04-19T01:05:21.005 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7339000000000001e-05 2024-04-19T01:05:21.005 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.006 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.006 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:21.006 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611749+0000", 2024-04-19T01:05:21.006 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3557000000000001e-05 2024-04-19T01:05:21.006 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.006 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.006 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:21.007 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.622598+0000", 2024-04-19T01:05:21.007 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010848315000000001 2024-04-19T01:05:21.007 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.007 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.007 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:21.007 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.622654+0000", 2024-04-19T01:05:21.007 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6311999999999999e-05 2024-04-19T01:05:21.008 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.008 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.008 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:21.008 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.622895+0000", 2024-04-19T01:05:21.008 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00024134300000000001 2024-04-19T01:05:21.008 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.009 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.009 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:21.009 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.623782+0000", 2024-04-19T01:05:21.009 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00088635099999999996 2024-04-19T01:05:21.009 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.009 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.010 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:21.010 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.632093+0000", 2024-04-19T01:05:21.010 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.008311466 2024-04-19T01:05:21.010 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.010 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.010 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:21.010 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.632137+0000", 2024-04-19T01:05:21.010 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4058999999999999e-05 2024-04-19T01:05:21.011 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.011 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:21.011 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.011 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.011 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.011 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:321 3.fs0 3:f23900a8:::smithi15528762-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-19T01:05:21.011 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:17.611769+0000", 2024-04-19T01:05:21.011 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 63.338161468000003, 2024-04-19T01:05:21.011 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021684730999999999, 2024-04-19T01:05:21.011 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:21.011 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:21.012 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:05:21.012 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:21.012 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:21.012 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:21.012 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 321 2024-04-19T01:05:21.012 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.012 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:21.012 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.012 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:21.012 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611769+0000", 2024-04-19T01:05:21.012 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.012 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.012 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.012 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:21.013 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611769+0000", 2024-04-19T01:05:21.013 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.013 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.013 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.013 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:21.013 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611770+0000", 2024-04-19T01:05:21.013 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.04e-06 2024-04-19T01:05:21.013 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.013 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.013 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:21.013 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611775+0000", 2024-04-19T01:05:21.014 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3310000000000003e-06 2024-04-19T01:05:21.014 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.014 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.014 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:21.014 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611793+0000", 2024-04-19T01:05:21.014 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7408e-05 2024-04-19T01:05:21.014 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.014 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.014 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:21.014 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.611808+0000", 2024-04-19T01:05:21.015 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5259000000000001e-05 2024-04-19T01:05:21.015 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.015 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.015 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:21.015 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.623039+0000", 2024-04-19T01:05:21.015 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011230756 2024-04-19T01:05:21.015 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.015 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.016 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:05:21.016 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.623089+0000", 2024-04-19T01:05:21.016 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0078999999999999e-05 2024-04-19T01:05:21.016 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.016 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.016 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:21.017 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.632385+0000", 2024-04-19T01:05:21.017 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0092956649999999998 2024-04-19T01:05:21.017 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.017 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.017 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:21.017 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.632471+0000", 2024-04-19T01:05:21.018 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.6693999999999994e-05 2024-04-19T01:05:21.018 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.018 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.018 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:21.018 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:17.633454+0000", 2024-04-19T01:05:21.018 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00098249899999999996 2024-04-19T01:05:21.019 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.019 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:21.019 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.019 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.019 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.019 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:345 3.fs0 3:f23900a8:::smithi15528762-6:head [append 0~557056] snapc 9=[8,7] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:05:21.019 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.534632+0000", 2024-04-19T01:05:21.020 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 62.415298644000003, 2024-04-19T01:05:21.020 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.044819185999999997, 2024-04-19T01:05:21.020 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:21.021 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:21.021 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:21.021 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:21.021 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:21.021 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:21.021 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 345 2024-04-19T01:05:21.021 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.021 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:21.021 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.021 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:21.021 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.534632+0000", 2024-04-19T01:05:21.022 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.022 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.022 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.022 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:21.022 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.534632+0000", 2024-04-19T01:05:21.022 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.022 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.022 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.022 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:21.022 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.534634+0000", 2024-04-19T01:05:21.022 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4099999999999998e-06 2024-04-19T01:05:21.022 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.022 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.022 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:21.023 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.535690+0000", 2024-04-19T01:05:21.023 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010557260000000001 2024-04-19T01:05:21.023 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.023 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.023 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:21.023 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.535710+0000", 2024-04-19T01:05:21.023 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0339e-05 2024-04-19T01:05:21.023 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.023 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.023 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:21.023 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.535720+0000", 2024-04-19T01:05:21.023 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7599999999999997e-06 2024-04-19T01:05:21.023 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.023 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.024 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:21.024 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.535817+0000", 2024-04-19T01:05:21.024 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7218000000000003e-05 2024-04-19T01:05:21.024 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.024 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.024 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:21.024 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.536037+0000", 2024-04-19T01:05:21.024 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00021956699999999999 2024-04-19T01:05:21.024 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.024 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.024 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:21.025 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.537439+0000", 2024-04-19T01:05:21.025 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014022240000000001 2024-04-19T01:05:21.025 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.025 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.025 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:21.025 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.557535+0000", 2024-04-19T01:05:21.025 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.020095874999999999 2024-04-19T01:05:21.026 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.026 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.026 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:21.026 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.579388+0000", 2024-04-19T01:05:21.026 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021852772999999999 2024-04-19T01:05:21.026 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.026 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.027 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:21.027 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.579451+0000", 2024-04-19T01:05:21.027 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3293999999999994e-05 2024-04-19T01:05:21.027 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.028 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:21.028 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.028 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.028 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.028 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:346 3.fs0 3:f23900a8:::smithi15528762-6:head [append 0~671744] snapc 9=[8,7] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:05:21.028 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.540969+0000", 2024-04-19T01:05:21.029 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 62.408961116, 2024-04-19T01:05:21.029 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.039266644000000003, 2024-04-19T01:05:21.029 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:21.029 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:21.029 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:21.029 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:21.029 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:21.029 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:21.029 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 346 2024-04-19T01:05:21.029 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.029 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:21.029 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.029 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:21.030 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.540969+0000", 2024-04-19T01:05:21.030 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.030 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.030 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.030 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:21.030 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.540969+0000", 2024-04-19T01:05:21.030 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.030 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.031 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.031 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:21.031 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.540971+0000", 2024-04-19T01:05:21.031 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9039999999999999e-06 2024-04-19T01:05:21.031 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.031 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.031 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:21.031 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542173+0000", 2024-04-19T01:05:21.031 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00120192 2024-04-19T01:05:21.033 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.033 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.033 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:21.033 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542196+0000", 2024-04-19T01:05:21.033 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3159e-05 2024-04-19T01:05:21.033 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.033 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.033 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:21.034 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542210+0000", 2024-04-19T01:05:21.034 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3899e-05 2024-04-19T01:05:21.034 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.034 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.034 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:21.034 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.546462+0000", 2024-04-19T01:05:21.034 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0042514629999999996 2024-04-19T01:05:21.034 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.034 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.034 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:21.034 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.546596+0000", 2024-04-19T01:05:21.034 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013463900000000001 2024-04-19T01:05:21.034 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.034 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.035 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:21.035 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.547898+0000", 2024-04-19T01:05:21.035 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0013016239999999999 2024-04-19T01:05:21.035 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.035 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.035 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:21.035 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.558500+0000", 2024-04-19T01:05:21.035 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010601928 2024-04-19T01:05:21.035 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.035 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.035 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:21.035 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.580189+0000", 2024-04-19T01:05:21.035 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021688796 2024-04-19T01:05:21.035 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.035 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.036 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:21.036 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.580236+0000", 2024-04-19T01:05:21.036 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7311999999999997e-05 2024-04-19T01:05:21.036 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.036 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:21.036 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.036 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.036 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.036 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:347 3.fs0 3:f23900a8:::smithi15528762-6:head [append 0~98304] snapc 9=[8,7] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:05:21.036 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.542237+0000", 2024-04-19T01:05:21.036 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 62.407693559000002, 2024-04-19T01:05:21.036 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.038983274999999998, 2024-04-19T01:05:21.036 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:21.036 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:21.037 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:21.037 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:21.037 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:21.037 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:21.037 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 347 2024-04-19T01:05:21.037 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.037 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:21.037 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.037 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:21.037 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542237+0000", 2024-04-19T01:05:21.037 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.037 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.037 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.037 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:21.037 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542237+0000", 2024-04-19T01:05:21.038 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.038 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.038 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.038 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:21.038 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542238+0000", 2024-04-19T01:05:21.038 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3659999999999999e-06 2024-04-19T01:05:21.038 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.038 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.038 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:21.038 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542318+0000", 2024-04-19T01:05:21.038 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.0158000000000001e-05 2024-04-19T01:05:21.038 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.038 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.038 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:21.039 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542336+0000", 2024-04-19T01:05:21.039 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7836999999999999e-05 2024-04-19T01:05:21.039 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.039 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.039 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:21.039 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542348+0000", 2024-04-19T01:05:21.039 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1646e-05 2024-04-19T01:05:21.039 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.039 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.039 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:21.039 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.553869+0000", 2024-04-19T01:05:21.039 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011521164 2024-04-19T01:05:21.039 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.039 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.039 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:21.040 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.554020+0000", 2024-04-19T01:05:21.040 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015064200000000001 2024-04-19T01:05:21.040 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.040 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.040 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:21.040 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.554913+0000", 2024-04-19T01:05:21.040 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00089314099999999997 2024-04-19T01:05:21.040 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.040 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.040 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:21.040 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.558556+0000", 2024-04-19T01:05:21.040 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003642803 2024-04-19T01:05:21.040 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.040 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.040 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:21.041 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.581179+0000", 2024-04-19T01:05:21.041 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.022623306999999999 2024-04-19T01:05:21.041 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.041 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.041 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:21.041 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.581220+0000", 2024-04-19T01:05:21.041 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1211000000000002e-05 2024-04-19T01:05:21.041 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.041 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:21.041 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.041 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.041 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.041 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:348 3.fs0 3:f23900a8:::smithi15528762-6:head [setxattr _header (55) in=7b] snapc 9=[8,7] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:05:21.041 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.542369+0000", 2024-04-19T01:05:21.041 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 62.407561698999999, 2024-04-19T01:05:21.042 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.039646576000000003, 2024-04-19T01:05:21.042 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:21.042 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:21.042 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:21.042 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:21.042 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:21.042 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:21.042 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 348 2024-04-19T01:05:21.042 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.042 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:21.042 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.042 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:21.042 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542369+0000", 2024-04-19T01:05:21.042 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.043 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.043 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.043 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:21.043 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542369+0000", 2024-04-19T01:05:21.043 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.043 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.043 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.043 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:21.043 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542370+0000", 2024-04-19T01:05:21.043 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.384e-06 2024-04-19T01:05:21.043 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.043 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.043 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:21.043 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542376+0000", 2024-04-19T01:05:21.043 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.806e-06 2024-04-19T01:05:21.044 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.044 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.044 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:21.044 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542400+0000", 2024-04-19T01:05:21.044 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4218000000000001e-05 2024-04-19T01:05:21.044 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.044 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.044 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:21.044 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542411+0000", 2024-04-19T01:05:21.044 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0801999999999999e-05 2024-04-19T01:05:21.044 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.044 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.044 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:21.044 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.557611+0000", 2024-04-19T01:05:21.044 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015199724 2024-04-19T01:05:21.045 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.045 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.045 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:21.045 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.557731+0000", 2024-04-19T01:05:21.045 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012045900000000001 2024-04-19T01:05:21.045 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.045 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.045 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:21.045 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.558229+0000", 2024-04-19T01:05:21.045 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00049771099999999997 2024-04-19T01:05:21.045 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.045 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.045 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:21.045 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.559080+0000", 2024-04-19T01:05:21.046 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00085117400000000001 2024-04-19T01:05:21.046 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.046 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.046 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:21.046 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.581912+0000", 2024-04-19T01:05:21.047 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.022832086000000001 2024-04-19T01:05:21.047 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.047 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.047 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:21.047 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.582015+0000", 2024-04-19T01:05:21.047 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000103212 2024-04-19T01:05:21.047 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.048 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:21.048 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.048 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.048 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.048 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:349 3.fs0 3:f23900a8:::smithi15528762-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:05:21.048 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.542427+0000", 2024-04-19T01:05:21.048 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 62.407503527999999, 2024-04-19T01:05:21.048 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.041128310000000001, 2024-04-19T01:05:21.048 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:21.048 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:21.048 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:05:21.048 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:21.048 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:21.049 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:21.049 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 349 2024-04-19T01:05:21.049 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.049 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:21.049 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.049 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:21.049 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542427+0000", 2024-04-19T01:05:21.049 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.049 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.049 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.049 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:21.049 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542427+0000", 2024-04-19T01:05:21.049 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.049 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.050 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.050 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:21.050 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542428+0000", 2024-04-19T01:05:21.050 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.4e-07 2024-04-19T01:05:21.050 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.050 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.050 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:21.050 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542432+0000", 2024-04-19T01:05:21.050 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5499999999999996e-06 2024-04-19T01:05:21.050 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.050 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.050 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:21.050 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542446+0000", 2024-04-19T01:05:21.050 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3365e-05 2024-04-19T01:05:21.051 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.051 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.051 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:21.051 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.542456+0000", 2024-04-19T01:05:21.051 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.029e-05 2024-04-19T01:05:21.051 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.051 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.051 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:21.051 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.558587+0000", 2024-04-19T01:05:21.051 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.016131106999999999 2024-04-19T01:05:21.051 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.052 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.052 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:05:21.052 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.558676+0000", 2024-04-19T01:05:21.052 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.8823e-05 2024-04-19T01:05:21.052 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.052 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.052 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:21.052 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.582441+0000", 2024-04-19T01:05:21.052 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.023764864 2024-04-19T01:05:21.052 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.052 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.052 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:21.052 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.582493+0000", 2024-04-19T01:05:21.053 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2130999999999999e-05 2024-04-19T01:05:21.053 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.053 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.053 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:21.053 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.583555+0000", 2024-04-19T01:05:21.053 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010623399999999999 2024-04-19T01:05:21.053 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.053 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:21.053 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.053 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.053 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.053 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:660 3.fs0 3:f23900a8:::smithi15528762-6:head [append 0~786432] snapc 1d=[1d,1c,1b,14] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-04-19T01:05:21.053 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:56.630857+0000", 2024-04-19T01:05:21.054 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 24.319073197000002, 2024-04-19T01:05:21.054 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.038376939999999998, 2024-04-19T01:05:21.054 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:21.054 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:21.054 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:21.054 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:21.054 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:21.054 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:21.054 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 660 2024-04-19T01:05:21.054 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.054 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:21.054 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.054 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:21.054 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.630857+0000", 2024-04-19T01:05:21.055 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.055 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.055 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.055 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:21.055 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.630857+0000", 2024-04-19T01:05:21.055 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.055 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.055 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.055 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:21.055 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.630859+0000", 2024-04-19T01:05:21.055 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0719999999999998e-06 2024-04-19T01:05:21.055 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.055 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.055 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:21.056 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.631959+0000", 2024-04-19T01:05:21.056 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010997380000000001 2024-04-19T01:05:21.056 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.056 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.056 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:21.056 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.631981+0000", 2024-04-19T01:05:21.056 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.2354999999999999e-05 2024-04-19T01:05:21.056 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.056 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.056 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:21.056 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.631994+0000", 2024-04-19T01:05:21.056 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2541000000000001e-05 2024-04-19T01:05:21.056 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.057 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.057 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:21.057 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.632126+0000", 2024-04-19T01:05:21.057 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000132037 2024-04-19T01:05:21.057 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.057 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.057 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:21.058 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.632292+0000", 2024-04-19T01:05:21.058 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000165635 2024-04-19T01:05:21.058 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.058 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.058 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:21.058 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.634219+0000", 2024-04-19T01:05:21.058 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0019270800000000001 2024-04-19T01:05:21.058 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.058 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.058 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:21.059 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.658365+0000", 2024-04-19T01:05:21.059 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.024145811 2024-04-19T01:05:21.059 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.059 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.059 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:21.059 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.669104+0000", 2024-04-19T01:05:21.059 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010739395000000001 2024-04-19T01:05:21.059 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.059 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.059 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:21.059 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.669234+0000", 2024-04-19T01:05:21.059 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000130276 2024-04-19T01:05:21.060 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.060 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:21.060 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.060 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.060 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.060 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:661 3.fs0 3:f23900a8:::smithi15528762-6:head [append 0~458752] snapc 1d=[1d,1c,1b,14] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-04-19T01:05:21.060 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:56.634770+0000", 2024-04-19T01:05:21.060 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 24.315160104, 2024-04-19T01:05:21.060 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.037192913000000001, 2024-04-19T01:05:21.060 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:21.060 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:21.060 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:21.060 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:21.061 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:21.061 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:21.061 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 661 2024-04-19T01:05:21.061 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.061 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:21.061 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.061 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:21.061 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.634770+0000", 2024-04-19T01:05:21.061 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.061 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.061 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.061 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:21.061 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.634770+0000", 2024-04-19T01:05:21.062 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.062 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.062 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.062 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:21.062 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.634772+0000", 2024-04-19T01:05:21.062 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8360000000000001e-06 2024-04-19T01:05:21.062 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.062 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.062 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:21.062 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.635445+0000", 2024-04-19T01:05:21.062 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00067238900000000002 2024-04-19T01:05:21.062 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.062 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.063 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:21.063 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.635478+0000", 2024-04-19T01:05:21.063 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.3878000000000003e-05 2024-04-19T01:05:21.063 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.063 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.063 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:21.063 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.635488+0000", 2024-04-19T01:05:21.063 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.8020000000000003e-06 2024-04-19T01:05:21.063 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.063 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.063 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:21.063 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.648319+0000", 2024-04-19T01:05:21.064 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012830879 2024-04-19T01:05:21.064 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.064 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.064 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:21.064 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.648638+0000", 2024-04-19T01:05:21.064 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00031897300000000001 2024-04-19T01:05:21.064 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.064 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.064 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:21.064 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.650056+0000", 2024-04-19T01:05:21.064 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014178319999999999 2024-04-19T01:05:21.065 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.065 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.065 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:21.065 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.664923+0000", 2024-04-19T01:05:21.065 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.01486728 2024-04-19T01:05:21.065 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.065 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.065 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:21.065 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.671903+0000", 2024-04-19T01:05:21.065 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0069797399999999999 2024-04-19T01:05:21.065 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.065 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.065 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:21.065 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.671963+0000", 2024-04-19T01:05:21.066 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0303999999999997e-05 2024-04-19T01:05:21.066 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.066 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:21.066 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.066 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.066 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.066 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:662 3.fs0 3:f23900a8:::smithi15528762-6:head [append 0~442368] snapc 1d=[1d,1c,1b,14] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-04-19T01:05:21.066 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:56.637960+0000", 2024-04-19T01:05:21.066 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 24.311970862999999, 2024-04-19T01:05:21.066 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.040497497, 2024-04-19T01:05:21.066 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:21.066 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:21.066 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:21.067 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:21.067 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:21.067 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:21.067 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 662 2024-04-19T01:05:21.067 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.067 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:21.067 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.067 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:21.067 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.637960+0000", 2024-04-19T01:05:21.068 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.068 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.068 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.068 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:21.068 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.637960+0000", 2024-04-19T01:05:21.068 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.068 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.068 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.068 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:21.068 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.637961+0000", 2024-04-19T01:05:21.068 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.48e-06 2024-04-19T01:05:21.069 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.069 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.069 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:21.069 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.638402+0000", 2024-04-19T01:05:21.069 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000440698 2024-04-19T01:05:21.069 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.069 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.069 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:21.069 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.638421+0000", 2024-04-19T01:05:21.069 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9182999999999999e-05 2024-04-19T01:05:21.069 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.069 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.069 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:21.069 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.638430+0000", 2024-04-19T01:05:21.070 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.7749999999999998e-06 2024-04-19T01:05:21.070 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.070 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.070 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:21.070 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.658705+0000", 2024-04-19T01:05:21.070 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.020275472999999999 2024-04-19T01:05:21.070 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.070 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.070 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:21.070 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.658871+0000", 2024-04-19T01:05:21.070 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00016530900000000001 2024-04-19T01:05:21.070 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.070 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.070 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:21.071 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.660103+0000", 2024-04-19T01:05:21.071 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001232885 2024-04-19T01:05:21.071 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.071 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.071 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:21.071 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.669256+0000", 2024-04-19T01:05:21.071 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0091529620000000006 2024-04-19T01:05:21.071 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.071 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.071 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:21.071 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.678392+0000", 2024-04-19T01:05:21.071 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0091357639999999993 2024-04-19T01:05:21.071 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.071 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.072 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:21.072 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.678457+0000", 2024-04-19T01:05:21.072 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4968000000000005e-05 2024-04-19T01:05:21.072 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.072 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:21.072 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.072 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.072 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.072 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:663 3.fs0 3:f23900a8:::smithi15528762-6:head [append 0~262144] snapc 1d=[1d,1c,1b,14] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-04-19T01:05:21.072 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:56.639883+0000", 2024-04-19T01:05:21.072 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 24.310047702999999, 2024-04-19T01:05:21.072 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.038981265000000001, 2024-04-19T01:05:21.072 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:21.072 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:21.073 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:21.073 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:21.073 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:21.073 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:21.073 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 663 2024-04-19T01:05:21.073 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.073 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:21.073 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.073 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:21.073 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.639883+0000", 2024-04-19T01:05:21.073 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.073 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.073 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.073 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:21.073 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.639883+0000", 2024-04-19T01:05:21.074 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.074 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.074 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.074 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:21.074 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.639884+0000", 2024-04-19T01:05:21.074 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6020000000000001e-06 2024-04-19T01:05:21.074 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.074 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.074 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:21.074 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.640138+0000", 2024-04-19T01:05:21.074 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00025372399999999999 2024-04-19T01:05:21.074 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.074 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.075 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:21.075 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.640159+0000", 2024-04-19T01:05:21.075 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0879000000000001e-05 2024-04-19T01:05:21.075 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.075 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.075 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:21.075 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.640170+0000", 2024-04-19T01:05:21.075 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0702e-05 2024-04-19T01:05:21.075 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.075 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.075 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:21.075 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.664988+0000", 2024-04-19T01:05:21.076 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.024818468999999999 2024-04-19T01:05:21.076 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.076 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.076 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:21.076 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.665133+0000", 2024-04-19T01:05:21.076 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014525599999999999 2024-04-19T01:05:21.076 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.076 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.076 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:21.076 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.666115+0000", 2024-04-19T01:05:21.076 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00098171099999999991 2024-04-19T01:05:21.076 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.076 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.076 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:21.077 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.670349+0000", 2024-04-19T01:05:21.077 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0042341560000000002 2024-04-19T01:05:21.077 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.077 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.077 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:21.077 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.678680+0000", 2024-04-19T01:05:21.077 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0083308489999999995 2024-04-19T01:05:21.077 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.077 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.077 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:21.077 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.678864+0000", 2024-04-19T01:05:21.077 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00018391699999999999 2024-04-19T01:05:21.077 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.078 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:21.078 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.078 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.078 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.078 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:664 3.fs0 3:f23900a8:::smithi15528762-6:head [setxattr _header (56) in=7b] snapc 1d=[1d,1c,1b,14] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-04-19T01:05:21.078 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:56.640187+0000", 2024-04-19T01:05:21.078 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 24.309743096999998, 2024-04-19T01:05:21.079 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.038910425999999998, 2024-04-19T01:05:21.079 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:21.079 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:21.079 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:21.079 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:21.079 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:21.079 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:21.079 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 664 2024-04-19T01:05:21.079 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.079 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:21.079 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.079 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:21.079 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.640187+0000", 2024-04-19T01:05:21.080 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.080 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.080 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.080 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:21.080 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.640187+0000", 2024-04-19T01:05:21.080 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.080 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.080 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.080 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:21.080 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.640188+0000", 2024-04-19T01:05:21.080 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.2799999999999995e-07 2024-04-19T01:05:21.081 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.081 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.081 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:21.081 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.640193+0000", 2024-04-19T01:05:21.081 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3599999999999998e-06 2024-04-19T01:05:21.081 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.081 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.081 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:21.081 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.640200+0000", 2024-04-19T01:05:21.081 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7989999999999994e-06 2024-04-19T01:05:21.081 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.081 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.081 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:21.082 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.640206+0000", 2024-04-19T01:05:21.082 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6339999999999998e-06 2024-04-19T01:05:21.082 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.082 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.082 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:21.082 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.669495+0000", 2024-04-19T01:05:21.082 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.029289440999999999 2024-04-19T01:05:21.082 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.082 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.082 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:21.082 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.669587+0000", 2024-04-19T01:05:21.082 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.1848999999999994e-05 2024-04-19T01:05:21.082 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.082 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.083 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:21.083 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.670063+0000", 2024-04-19T01:05:21.083 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00047547900000000001 2024-04-19T01:05:21.083 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.083 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.083 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:21.083 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.672096+0000", 2024-04-19T01:05:21.083 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0020328740000000001 2024-04-19T01:05:21.083 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.083 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.083 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:21.083 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.679025+0000", 2024-04-19T01:05:21.083 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.006929742 2024-04-19T01:05:21.084 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.084 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.084 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:21.084 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.679098+0000", 2024-04-19T01:05:21.084 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2420000000000001e-05 2024-04-19T01:05:21.084 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.084 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:21.084 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.084 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.084 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.084 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:665 3.fs0 3:f23900a8:::smithi15528762-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e86)", 2024-04-19T01:05:21.084 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:56.640214+0000", 2024-04-19T01:05:21.084 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 24.309716636000001, 2024-04-19T01:05:21.085 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.042546254999999998, 2024-04-19T01:05:21.085 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:21.085 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:21.085 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:05:21.085 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:21.085 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:21.085 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:21.085 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 665 2024-04-19T01:05:21.085 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.085 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:21.085 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.085 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:21.086 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.640214+0000", 2024-04-19T01:05:21.086 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.086 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.086 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.086 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:21.086 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.640214+0000", 2024-04-19T01:05:21.086 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.086 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.086 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.086 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:21.087 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.640214+0000", 2024-04-19T01:05:21.087 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4799999999999999e-07 2024-04-19T01:05:21.087 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.087 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.087 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:21.087 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.640217+0000", 2024-04-19T01:05:21.087 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.334e-06 2024-04-19T01:05:21.087 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.087 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.087 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:21.087 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.640224+0000", 2024-04-19T01:05:21.087 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.0530000000000001e-06 2024-04-19T01:05:21.087 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.087 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.088 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:21.088 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.640229+0000", 2024-04-19T01:05:21.088 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.48e-06 2024-04-19T01:05:21.088 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.088 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.088 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:21.088 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.670412+0000", 2024-04-19T01:05:21.089 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.030183202999999999 2024-04-19T01:05:21.089 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.089 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.089 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:05:21.089 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.670542+0000", 2024-04-19T01:05:21.089 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012960099999999999 2024-04-19T01:05:21.089 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.089 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.089 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:21.089 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.679535+0000", 2024-04-19T01:05:21.089 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0089925720000000008 2024-04-19T01:05:21.089 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.089 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.090 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:21.090 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.679696+0000", 2024-04-19T01:05:21.090 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000161013 2024-04-19T01:05:21.090 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.090 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.090 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:21.090 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.682760+0000", 2024-04-19T01:05:21.090 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003064551 2024-04-19T01:05:21.090 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.090 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:21.090 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.090 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.090 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:21.090 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:05:21.092 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-19T01:05:21.093 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:05:21.094+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:05:21.193 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:05:21.189+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:05:21.267 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:05:21.267 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:05:21.267 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2024-04-19T01:05:21.267 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:05:21.282 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-19T01:05:21.294 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:05:21.296+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:05:21.395 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:05:21.396+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:05:21.455 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:05:21.455 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:05:21.455 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2024-04-19T01:05:21.455 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2024-04-19T01:05:21.455 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:05:21.470 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-19T01:05:21.496 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:05:21.494+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:05:21.524 INFO:tasks.rados.rados.0.smithi155.stdout:354: done (0 left) 2024-04-19T01:05:21.524 INFO:tasks.rados.rados.0.smithi155.stdout:355: snap_remove snap 37 2024-04-19T01:05:21.596 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:05:21.593+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:05:21.606 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:05:21.606 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2024-04-19T01:05:21.606 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2024-04-19T01:05:21.606 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-04-19T01:05:21.606 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.606 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.0s2 44/31 ECSubWrite(tid=25, reqid=client.4604.0:284, at_version=44'22, trim_to=0'0, roll_forward_to=44'20))", 2024-04-19T01:05:21.606 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:14.920303+0000", 2024-04-19T01:05:21.606 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 66.680283259999996, 2024-04-19T01:05:21.606 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011517777999999999, 2024-04-19T01:05:21.606 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:21.606 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:21.607 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:05:21.607 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:21.607 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.607 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:21.607 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.920303+0000", 2024-04-19T01:05:21.607 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.607 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.607 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.607 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:21.607 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.920303+0000", 2024-04-19T01:05:21.607 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.607 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.607 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.608 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:21.608 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.920304+0000", 2024-04-19T01:05:21.608 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2900000000000003e-07 2024-04-19T01:05:21.608 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.608 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.608 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:21.608 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.921382+0000", 2024-04-19T01:05:21.608 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010785510000000001 2024-04-19T01:05:21.608 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.608 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.608 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:21.609 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.921394+0000", 2024-04-19T01:05:21.610 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2183e-05 2024-04-19T01:05:21.610 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.610 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.610 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:21.610 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.921403+0000", 2024-04-19T01:05:21.610 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9330000000000007e-06 2024-04-19T01:05:21.610 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.611 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.611 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:21.611 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.921468+0000", 2024-04-19T01:05:21.611 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4191999999999998e-05 2024-04-19T01:05:21.611 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.611 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.611 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:21.611 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.921479+0000", 2024-04-19T01:05:21.611 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1763000000000001e-05 2024-04-19T01:05:21.611 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.611 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.611 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:21.611 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.931782+0000", 2024-04-19T01:05:21.611 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010302891999999999 2024-04-19T01:05:21.611 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.612 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.612 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:21.612 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.931821+0000", 2024-04-19T01:05:21.612 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8535000000000003e-05 2024-04-19T01:05:21.612 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.612 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:21.612 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.612 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.612 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.612 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.as1 48/31 ECSubWrite(tid=71, reqid=client.4604.0:354, at_version=48'10, trim_to=0'0, roll_forward_to=34'8))", 2024-04-19T01:05:21.612 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.556705+0000", 2024-04-19T01:05:21.612 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 63.043880965, 2024-04-19T01:05:21.612 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013008369000000001, 2024-04-19T01:05:21.612 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:21.612 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:21.613 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:05:21.613 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:21.613 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.613 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:21.613 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.556705+0000", 2024-04-19T01:05:21.613 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.613 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.613 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.613 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:21.613 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.556705+0000", 2024-04-19T01:05:21.613 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.613 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.613 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.613 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:21.614 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.556706+0000", 2024-04-19T01:05:21.614 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2800000000000005e-07 2024-04-19T01:05:21.614 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.614 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.614 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:21.614 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.557214+0000", 2024-04-19T01:05:21.614 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00050754200000000004 2024-04-19T01:05:21.614 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.614 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.614 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:21.614 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.557239+0000", 2024-04-19T01:05:21.614 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.567e-05 2024-04-19T01:05:21.614 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.614 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.614 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:21.615 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.557268+0000", 2024-04-19T01:05:21.615 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8496e-05 2024-04-19T01:05:21.615 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.615 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.615 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:21.615 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.557438+0000", 2024-04-19T01:05:21.615 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000169963 2024-04-19T01:05:21.616 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.616 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.616 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:21.616 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.557491+0000", 2024-04-19T01:05:21.616 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2689999999999999e-05 2024-04-19T01:05:21.616 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.616 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.616 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:21.616 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.569674+0000", 2024-04-19T01:05:21.616 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012183727 2024-04-19T01:05:21.616 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.616 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.616 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:21.616 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.569714+0000", 2024-04-19T01:05:21.616 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9353000000000001e-05 2024-04-19T01:05:21.617 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.617 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:21.617 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.617 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.617 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.617 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.3s5 49/31 ECSubWrite(tid=48, reqid=osd.15.0:48, at_version=49'38, trim_to=0'0, roll_forward_to=48'36))", 2024-04-19T01:05:21.617 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:19.533518+0000", 2024-04-19T01:05:21.617 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 62.067067979999997, 2024-04-19T01:05:21.617 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011056513, 2024-04-19T01:05:21.617 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:21.617 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:21.617 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:05:21.617 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:21.617 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.617 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:21.618 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.533518+0000", 2024-04-19T01:05:21.618 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.618 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.618 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.618 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:21.618 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.533518+0000", 2024-04-19T01:05:21.618 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.618 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.618 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.618 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:21.618 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.533519+0000", 2024-04-19T01:05:21.618 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4000000000000002e-07 2024-04-19T01:05:21.618 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.618 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.619 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:21.619 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.533543+0000", 2024-04-19T01:05:21.619 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4494999999999998e-05 2024-04-19T01:05:21.619 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.619 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.619 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:21.619 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.533556+0000", 2024-04-19T01:05:21.619 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3145e-05 2024-04-19T01:05:21.619 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.619 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.619 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:21.619 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.533565+0000", 2024-04-19T01:05:21.619 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.4330000000000004e-06 2024-04-19T01:05:21.619 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.619 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.620 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:21.620 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.533697+0000", 2024-04-19T01:05:21.620 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013253399999999999 2024-04-19T01:05:21.620 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.620 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.620 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:21.620 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.533714+0000", 2024-04-19T01:05:21.620 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7163000000000002e-05 2024-04-19T01:05:21.620 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.620 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.620 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:21.620 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.544518+0000", 2024-04-19T01:05:21.620 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010803063999999999 2024-04-19T01:05:21.620 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.620 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.621 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:21.621 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:19.544575+0000", 2024-04-19T01:05:21.621 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.7238999999999999e-05 2024-04-19T01:05:21.621 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.621 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:21.621 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.621 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.621 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.621 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.9s4 63/31 ECSubWrite(tid=152, reqid=client.4604.0:465, at_version=63'100, trim_to=0'0, roll_forward_to=62'98))", 2024-04-19T01:05:21.621 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:33.565180+0000", 2024-04-19T01:05:21.621 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 48.035406481999999, 2024-04-19T01:05:21.621 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018638968999999998, 2024-04-19T01:05:21.621 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:21.622 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:21.622 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:05:21.622 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:21.622 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.622 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:21.622 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:33.565180+0000", 2024-04-19T01:05:21.622 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.622 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.622 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.622 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:21.622 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:33.565180+0000", 2024-04-19T01:05:21.622 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.622 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.622 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.623 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:21.623 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:33.565181+0000", 2024-04-19T01:05:21.623 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7700000000000004e-07 2024-04-19T01:05:21.623 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.623 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.623 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:21.623 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:33.565938+0000", 2024-04-19T01:05:21.623 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00075768199999999997 2024-04-19T01:05:21.623 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.623 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.623 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:21.623 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:33.565973+0000", 2024-04-19T01:05:21.623 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5021999999999998e-05 2024-04-19T01:05:21.623 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.623 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.624 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:21.624 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:33.566001+0000", 2024-04-19T01:05:21.624 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8133e-05 2024-04-19T01:05:21.624 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.624 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.624 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:21.624 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:33.566147+0000", 2024-04-19T01:05:21.624 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000145539 2024-04-19T01:05:21.624 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.624 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.624 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:21.624 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:33.566165+0000", 2024-04-19T01:05:21.624 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7788999999999999e-05 2024-04-19T01:05:21.624 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.624 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.625 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:21.625 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:33.583741+0000", 2024-04-19T01:05:21.625 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017576789999999998 2024-04-19T01:05:21.625 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.625 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.625 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:21.625 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:33.583819+0000", 2024-04-19T01:05:21.625 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7237000000000006e-05 2024-04-19T01:05:21.625 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.625 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:21.625 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.625 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.626 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.626 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.as1 65/31 ECSubWrite(tid=107, reqid=client.4604.0:478, at_version=65'26, trim_to=0'0, roll_forward_to=64'24))", 2024-04-19T01:05:21.626 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:35.639127+0000", 2024-04-19T01:05:21.627 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 45.961459052999999, 2024-04-19T01:05:21.627 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011274793, 2024-04-19T01:05:21.627 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:21.627 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:21.627 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:05:21.627 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:21.627 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.627 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:21.627 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639127+0000", 2024-04-19T01:05:21.627 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.627 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.627 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.627 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:21.627 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639127+0000", 2024-04-19T01:05:21.627 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.628 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.628 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.628 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:21.628 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639128+0000", 2024-04-19T01:05:21.628 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6000000000000004e-07 2024-04-19T01:05:21.628 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.628 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.628 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:21.628 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639372+0000", 2024-04-19T01:05:21.628 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00024466199999999998 2024-04-19T01:05:21.628 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.628 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.628 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:21.628 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639401+0000", 2024-04-19T01:05:21.628 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8282000000000001e-05 2024-04-19T01:05:21.629 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.629 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.629 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:21.629 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639446+0000", 2024-04-19T01:05:21.629 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5513000000000003e-05 2024-04-19T01:05:21.629 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.629 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.629 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:21.629 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639612+0000", 2024-04-19T01:05:21.629 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000166069 2024-04-19T01:05:21.629 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.629 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.629 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:21.629 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639626+0000", 2024-04-19T01:05:21.629 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3811999999999999e-05 2024-04-19T01:05:21.630 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.630 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.630 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:21.630 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.650359+0000", 2024-04-19T01:05:21.630 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010732858 2024-04-19T01:05:21.630 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.630 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.630 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:21.630 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.650402+0000", 2024-04-19T01:05:21.630 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3037000000000001e-05 2024-04-19T01:05:21.630 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.630 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:21.630 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.630 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.631 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.631 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.2s3 79/31 ECSubWrite(tid=249, reqid=osd.11.0:249, at_version=79'21, trim_to=0'0, roll_forward_to=74'18))", 2024-04-19T01:05:21.631 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:49.638454+0000", 2024-04-19T01:05:21.631 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 31.962132681, 2024-04-19T01:05:21.631 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011186546, 2024-04-19T01:05:21.631 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:21.631 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:21.631 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:05:21.631 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:21.631 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.631 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:21.631 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.638454+0000", 2024-04-19T01:05:21.631 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.631 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.631 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.632 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:21.632 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.638454+0000", 2024-04-19T01:05:21.632 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.632 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.632 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.632 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:21.632 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.638454+0000", 2024-04-19T01:05:21.632 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.58e-07 2024-04-19T01:05:21.632 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.632 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.632 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:21.632 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.638536+0000", 2024-04-19T01:05:21.632 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.2070999999999994e-05 2024-04-19T01:05:21.632 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.632 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.633 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:21.633 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.638561+0000", 2024-04-19T01:05:21.633 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5026999999999999e-05 2024-04-19T01:05:21.633 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.633 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.633 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:21.633 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.638585+0000", 2024-04-19T01:05:21.633 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4324999999999998e-05 2024-04-19T01:05:21.633 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.633 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.633 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:21.633 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.638684+0000", 2024-04-19T01:05:21.633 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.8196999999999995e-05 2024-04-19T01:05:21.633 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.633 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.633 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:21.634 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.638702+0000", 2024-04-19T01:05:21.634 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7989999999999999e-05 2024-04-19T01:05:21.634 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.634 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.634 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:21.635 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.649593+0000", 2024-04-19T01:05:21.635 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010891813 2024-04-19T01:05:21.635 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.635 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.635 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:21.636 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.649640+0000", 2024-04-19T01:05:21.636 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6665000000000001e-05 2024-04-19T01:05:21.636 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.636 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:21.636 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.636 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.636 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.637 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.3s5 80/31 ECSubWrite(tid=63, reqid=client.4604.0:612, at_version=80'51, trim_to=0'0, roll_forward_to=57'49))", 2024-04-19T01:05:21.637 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:50.660352+0000", 2024-04-19T01:05:21.637 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 30.940234511, 2024-04-19T01:05:21.637 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010734285999999999, 2024-04-19T01:05:21.637 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:21.637 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:21.637 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:05:21.637 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:21.637 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.637 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:21.637 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:50.660352+0000", 2024-04-19T01:05:21.637 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.637 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.638 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.638 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:21.638 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:50.660352+0000", 2024-04-19T01:05:21.638 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.638 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.638 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.638 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:21.638 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:50.660352+0000", 2024-04-19T01:05:21.638 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2899999999999999e-07 2024-04-19T01:05:21.638 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.638 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.638 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:21.638 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:50.660870+0000", 2024-04-19T01:05:21.639 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00051747399999999997 2024-04-19T01:05:21.639 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.639 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.639 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:21.639 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:50.660884+0000", 2024-04-19T01:05:21.639 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4608e-05 2024-04-19T01:05:21.639 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.639 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.639 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:21.639 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:50.660892+0000", 2024-04-19T01:05:21.639 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.9740000000000002e-06 2024-04-19T01:05:21.639 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.639 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.639 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:21.640 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:50.660991+0000", 2024-04-19T01:05:21.640 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.8862000000000003e-05 2024-04-19T01:05:21.640 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.640 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.640 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:21.640 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:50.661003+0000", 2024-04-19T01:05:21.640 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1629e-05 2024-04-19T01:05:21.640 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.640 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.640 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:21.640 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:50.671021+0000", 2024-04-19T01:05:21.640 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010018575 2024-04-19T01:05:21.640 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.640 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.641 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:21.641 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:50.671086+0000", 2024-04-19T01:05:21.641 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4734999999999996e-05 2024-04-19T01:05:21.641 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.641 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:21.641 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.641 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.641 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.641 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.es3 86/31 ECSubWrite(tid=195, reqid=client.4604.0:653, at_version=86'39, trim_to=0'0, roll_forward_to=84'38))", 2024-04-19T01:05:21.641 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:56.571261+0000", 2024-04-19T01:05:21.642 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 25.029324973000001, 2024-04-19T01:05:21.642 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.016452642, 2024-04-19T01:05:21.642 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:21.642 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:21.642 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:05:21.642 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:21.642 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.642 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:21.642 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.571261+0000", 2024-04-19T01:05:21.642 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.642 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.642 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.642 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:21.643 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.571261+0000", 2024-04-19T01:05:21.643 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.643 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.643 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.643 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:21.643 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.571262+0000", 2024-04-19T01:05:21.643 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2600000000000002e-07 2024-04-19T01:05:21.643 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.643 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.643 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:21.643 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.571978+0000", 2024-04-19T01:05:21.643 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00071624799999999995 2024-04-19T01:05:21.643 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.643 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.644 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:21.644 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.571990+0000", 2024-04-19T01:05:21.644 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1751e-05 2024-04-19T01:05:21.644 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.644 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.644 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:21.644 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.571996+0000", 2024-04-19T01:05:21.644 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3149999999999997e-06 2024-04-19T01:05:21.644 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.644 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.644 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:21.645 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.572061+0000", 2024-04-19T01:05:21.645 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5264000000000004e-05 2024-04-19T01:05:21.645 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.645 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.645 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:21.645 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.572070+0000", 2024-04-19T01:05:21.645 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.6500000000000002e-06 2024-04-19T01:05:21.645 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.645 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.645 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:21.645 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.587663+0000", 2024-04-19T01:05:21.645 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015592824999999999 2024-04-19T01:05:21.645 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.645 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.645 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:21.646 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.587714+0000", 2024-04-19T01:05:21.646 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.1063000000000002e-05 2024-04-19T01:05:21.646 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.646 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:21.646 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.646 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.646 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.646 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.2s3 86/31 ECSubWrite(tid=267, reqid=client.4604.0:657, at_version=86'37, trim_to=0'0, roll_forward_to=86'33))", 2024-04-19T01:05:21.646 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:56.614825+0000", 2024-04-19T01:05:21.646 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 24.985760941999999, 2024-04-19T01:05:21.646 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011489025999999999, 2024-04-19T01:05:21.646 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:21.647 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:21.647 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:05:21.647 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:21.647 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.647 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:21.647 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.614825+0000", 2024-04-19T01:05:21.647 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.647 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.648 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.648 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:21.648 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.614825+0000", 2024-04-19T01:05:21.648 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.648 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.648 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.648 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:21.648 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.614827+0000", 2024-04-19T01:05:21.648 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8670000000000001e-06 2024-04-19T01:05:21.648 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.648 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.648 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:21.648 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.615039+0000", 2024-04-19T01:05:21.648 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000211614 2024-04-19T01:05:21.648 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.649 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.649 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:21.649 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.615052+0000", 2024-04-19T01:05:21.649 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3083e-05 2024-04-19T01:05:21.649 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.649 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.649 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:21.649 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.615058+0000", 2024-04-19T01:05:21.649 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2060000000000004e-06 2024-04-19T01:05:21.649 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.649 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.649 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:21.649 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.615224+0000", 2024-04-19T01:05:21.649 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000166078 2024-04-19T01:05:21.649 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.649 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.650 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:21.650 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.615269+0000", 2024-04-19T01:05:21.650 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4582000000000001e-05 2024-04-19T01:05:21.650 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.650 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.650 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:21.650 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.626232+0000", 2024-04-19T01:05:21.650 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010963007 2024-04-19T01:05:21.650 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.650 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.650 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:21.650 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:56.626314+0000", 2024-04-19T01:05:21.650 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.2589000000000005e-05 2024-04-19T01:05:21.650 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.651 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:21.651 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.651 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.651 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.651 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.0s2 91/31 ECSubWrite(tid=96, reqid=client.4604.0:683, at_version=91'89, trim_to=0'0, roll_forward_to=89'87))", 2024-04-19T01:05:21.651 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:05:01.593810+0000", 2024-04-19T01:05:21.651 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 20.006776204000001, 2024-04-19T01:05:21.651 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011362906000000001, 2024-04-19T01:05:21.651 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:21.651 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:21.651 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:05:21.651 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:21.651 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.651 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:21.651 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:01.593810+0000", 2024-04-19T01:05:21.652 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.652 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.652 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.652 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:21.652 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:01.593810+0000", 2024-04-19T01:05:21.652 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.652 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.652 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.652 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:21.652 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:01.593810+0000", 2024-04-19T01:05:21.652 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4999999999999998e-07 2024-04-19T01:05:21.652 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.652 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.652 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:21.652 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:01.594171+0000", 2024-04-19T01:05:21.653 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000360195 2024-04-19T01:05:21.653 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.653 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.653 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:21.653 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:01.594200+0000", 2024-04-19T01:05:21.653 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9699000000000001e-05 2024-04-19T01:05:21.653 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.653 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.653 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:21.653 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:01.594219+0000", 2024-04-19T01:05:21.653 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8243000000000001e-05 2024-04-19T01:05:21.653 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.653 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.653 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:21.654 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:01.594403+0000", 2024-04-19T01:05:21.654 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00018461199999999999 2024-04-19T01:05:21.654 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.654 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.654 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:21.654 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:01.594423+0000", 2024-04-19T01:05:21.654 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.02e-05 2024-04-19T01:05:21.654 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.654 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.654 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:21.654 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:01.605070+0000", 2024-04-19T01:05:21.654 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010646654 2024-04-19T01:05:21.654 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.654 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.654 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:21.655 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:01.605173+0000", 2024-04-19T01:05:21.655 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000102853 2024-04-19T01:05:21.655 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.655 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:21.655 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.655 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.655 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.655 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.0s2 91/31 ECSubWrite(tid=97, reqid=client.4604.0:684, at_version=91'90, trim_to=0'0, roll_forward_to=89'87))", 2024-04-19T01:05:21.655 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:05:01.605812+0000", 2024-04-19T01:05:21.655 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 19.994773857999999, 2024-04-19T01:05:21.655 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011711571, 2024-04-19T01:05:21.655 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:21.655 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:21.655 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:05:21.655 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:21.656 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.656 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:21.656 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:01.605812+0000", 2024-04-19T01:05:21.656 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.656 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.656 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.656 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:21.656 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:01.605812+0000", 2024-04-19T01:05:21.656 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.656 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.656 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.656 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:21.656 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:01.605814+0000", 2024-04-19T01:05:21.656 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.593e-06 2024-04-19T01:05:21.657 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.657 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.657 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:21.657 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:01.606064+0000", 2024-04-19T01:05:21.657 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00025004999999999998 2024-04-19T01:05:21.657 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.657 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.658 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:21.658 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:01.606095+0000", 2024-04-19T01:05:21.658 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0799000000000001e-05 2024-04-19T01:05:21.658 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.658 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.658 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:21.658 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:01.606111+0000", 2024-04-19T01:05:21.658 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6473e-05 2024-04-19T01:05:21.658 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.658 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.658 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:21.658 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:01.606225+0000", 2024-04-19T01:05:21.658 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000113911 2024-04-19T01:05:21.658 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.659 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.659 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:21.659 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:01.606245+0000", 2024-04-19T01:05:21.659 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0123e-05 2024-04-19T01:05:21.659 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.659 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.659 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:21.659 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:01.617446+0000", 2024-04-19T01:05:21.659 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011201116000000001 2024-04-19T01:05:21.659 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.659 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.659 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:21.659 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:01.617524+0000", 2024-04-19T01:05:21.659 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7506e-05 2024-04-19T01:05:21.659 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.660 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:21.660 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.660 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.660 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.660 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.as1 97/31 ECSubWrite(tid=192, reqid=client.4604.0:722, at_version=97'88, trim_to=0'0, roll_forward_to=89'86))", 2024-04-19T01:05:21.660 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:05:07.581233+0000", 2024-04-19T01:05:21.660 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 14.019353109000001, 2024-04-19T01:05:21.660 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013324548, 2024-04-19T01:05:21.660 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:21.660 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:21.660 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:05:21.660 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:21.660 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.660 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:21.661 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.581233+0000", 2024-04-19T01:05:21.661 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.661 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.661 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.661 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:21.661 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.581233+0000", 2024-04-19T01:05:21.661 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.661 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.661 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.661 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:21.661 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.581236+0000", 2024-04-19T01:05:21.661 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7499999999999999e-06 2024-04-19T01:05:21.661 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.661 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.661 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:21.662 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.581552+0000", 2024-04-19T01:05:21.662 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00031587899999999998 2024-04-19T01:05:21.662 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.662 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.662 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:21.662 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.581612+0000", 2024-04-19T01:05:21.662 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0384999999999999e-05 2024-04-19T01:05:21.662 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.662 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.662 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:21.662 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.581647+0000", 2024-04-19T01:05:21.662 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4442e-05 2024-04-19T01:05:21.662 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.662 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.662 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:21.663 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.581831+0000", 2024-04-19T01:05:21.663 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00018393000000000001 2024-04-19T01:05:21.663 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.663 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.663 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:21.663 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.581858+0000", 2024-04-19T01:05:21.663 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7012000000000002e-05 2024-04-19T01:05:21.663 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.663 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.663 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:21.663 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.594439+0000", 2024-04-19T01:05:21.663 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012581188 2024-04-19T01:05:21.663 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.663 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.663 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:21.664 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.594558+0000", 2024-04-19T01:05:21.664 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000118962 2024-04-19T01:05:21.664 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.664 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:21.664 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.664 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.664 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.664 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.as1 97/31 ECSubWrite(tid=194, reqid=client.4604.0:723, at_version=97'89, trim_to=0'0, roll_forward_to=89'86))", 2024-04-19T01:05:21.664 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:05:07.596630+0000", 2024-04-19T01:05:21.664 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 14.003956453000001, 2024-04-19T01:05:21.664 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013245585000000001, 2024-04-19T01:05:21.664 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:21.664 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:21.664 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:05:21.664 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:21.665 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.665 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:21.665 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.596630+0000", 2024-04-19T01:05:21.665 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.665 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.665 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.665 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:21.665 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.596630+0000", 2024-04-19T01:05:21.665 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.665 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.665 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.665 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:21.665 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.596632+0000", 2024-04-19T01:05:21.665 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0470000000000001e-06 2024-04-19T01:05:21.665 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.666 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.666 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:21.666 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.597015+0000", 2024-04-19T01:05:21.666 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00038361299999999999 2024-04-19T01:05:21.666 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.666 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.666 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:21.666 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.597052+0000", 2024-04-19T01:05:21.666 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.6106000000000003e-05 2024-04-19T01:05:21.666 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.666 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.666 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:21.666 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.597076+0000", 2024-04-19T01:05:21.666 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4672e-05 2024-04-19T01:05:21.667 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.667 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.667 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:21.667 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.597256+0000", 2024-04-19T01:05:21.667 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00017985399999999999 2024-04-19T01:05:21.667 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.667 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.667 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:21.667 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.597285+0000", 2024-04-19T01:05:21.667 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9275e-05 2024-04-19T01:05:21.667 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.667 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.667 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:21.667 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.609786+0000", 2024-04-19T01:05:21.667 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012500755000000001 2024-04-19T01:05:21.668 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.668 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.668 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:21.668 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.609875+0000", 2024-04-19T01:05:21.668 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9263000000000002e-05 2024-04-19T01:05:21.668 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.668 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:21.668 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.668 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.668 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.668 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.as1 97/31 ECSubWrite(tid=196, reqid=client.4604.0:724, at_version=97'90, trim_to=0'0, roll_forward_to=97'88))", 2024-04-19T01:05:21.668 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:05:07.612638+0000", 2024-04-19T01:05:21.668 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 13.987948343999999, 2024-04-19T01:05:21.668 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014827862000000001, 2024-04-19T01:05:21.669 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:21.669 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:21.669 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:05:21.669 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:21.669 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.669 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:21.669 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.612638+0000", 2024-04-19T01:05:21.669 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.669 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.669 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.669 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:21.669 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.612638+0000", 2024-04-19T01:05:21.669 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.669 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.669 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.670 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:21.670 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.612641+0000", 2024-04-19T01:05:21.670 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7599999999999998e-06 2024-04-19T01:05:21.670 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.670 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.670 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:21.670 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.613003+0000", 2024-04-19T01:05:21.670 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00036252700000000002 2024-04-19T01:05:21.670 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.670 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.670 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:21.670 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.613046+0000", 2024-04-19T01:05:21.670 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3145000000000001e-05 2024-04-19T01:05:21.670 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.671 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.671 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:21.671 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.613072+0000", 2024-04-19T01:05:21.671 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5366000000000001e-05 2024-04-19T01:05:21.671 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.671 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.671 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:21.671 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.613321+0000", 2024-04-19T01:05:21.671 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000249208 2024-04-19T01:05:21.671 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.671 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.671 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:21.671 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.613353+0000", 2024-04-19T01:05:21.671 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.1699000000000003e-05 2024-04-19T01:05:21.671 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.671 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.672 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:21.672 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.627375+0000", 2024-04-19T01:05:21.672 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014022101 2024-04-19T01:05:21.672 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.672 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.672 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:21.672 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.627466+0000", 2024-04-19T01:05:21.672 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.1056000000000004e-05 2024-04-19T01:05:21.672 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.672 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:21.672 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.672 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.672 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.672 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.3s5 102/98 ECSubWrite(tid=101, reqid=client.4604.0:747, at_version=102'81, trim_to=0'0, roll_forward_to=102'79))", 2024-04-19T01:05:21.672 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:05:12.629199+0000", 2024-04-19T01:05:21.673 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 8.9713876799999994, 2024-04-19T01:05:21.673 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015073545000000001, 2024-04-19T01:05:21.673 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:21.673 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:21.673 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:05:21.673 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:21.673 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.673 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:21.673 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.629199+0000", 2024-04-19T01:05:21.673 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.673 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.673 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.673 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:21.673 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.629199+0000", 2024-04-19T01:05:21.673 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.674 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.674 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.674 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:21.674 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.629200+0000", 2024-04-19T01:05:21.674 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1680000000000001e-06 2024-04-19T01:05:21.674 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.674 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.674 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:21.674 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.629334+0000", 2024-04-19T01:05:21.674 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000134408 2024-04-19T01:05:21.674 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.674 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.674 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:21.674 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.629354+0000", 2024-04-19T01:05:21.675 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0307999999999999e-05 2024-04-19T01:05:21.675 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.675 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.675 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:21.675 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.629366+0000", 2024-04-19T01:05:21.675 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1355000000000001e-05 2024-04-19T01:05:21.675 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.675 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.675 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:21.675 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.629510+0000", 2024-04-19T01:05:21.675 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014463 2024-04-19T01:05:21.675 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.675 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.675 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:21.675 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.629539+0000", 2024-04-19T01:05:21.676 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.83e-05 2024-04-19T01:05:21.676 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.676 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.676 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:21.676 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.644209+0000", 2024-04-19T01:05:21.676 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014669926 2024-04-19T01:05:21.676 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.676 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.676 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:21.676 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.644272+0000", 2024-04-19T01:05:21.676 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3449999999999997e-05 2024-04-19T01:05:21.676 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.676 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:21.676 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.677 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.677 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.677 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds1 102/98 ECSubWrite(tid=208, reqid=client.4604.0:758, at_version=102'78, trim_to=0'0, roll_forward_to=0'0))", 2024-04-19T01:05:21.677 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:05:12.693446+0000", 2024-04-19T01:05:21.677 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 8.9071398930000001, 2024-04-19T01:05:21.677 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010882826, 2024-04-19T01:05:21.677 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:21.677 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:21.677 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:05:21.677 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:21.677 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.677 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:21.677 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.693446+0000", 2024-04-19T01:05:21.677 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.677 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.678 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.678 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:21.678 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.693446+0000", 2024-04-19T01:05:21.678 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.678 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.678 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.678 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:21.678 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.693449+0000", 2024-04-19T01:05:21.678 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.407e-06 2024-04-19T01:05:21.678 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.678 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.678 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:21.678 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.693671+0000", 2024-04-19T01:05:21.678 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00022273599999999999 2024-04-19T01:05:21.678 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.679 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.679 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:21.679 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.693712+0000", 2024-04-19T01:05:21.679 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0985999999999997e-05 2024-04-19T01:05:21.679 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.679 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.679 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:21.679 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.693740+0000", 2024-04-19T01:05:21.679 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7562000000000001e-05 2024-04-19T01:05:21.679 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.679 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.679 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:21.679 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.693846+0000", 2024-04-19T01:05:21.679 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010564 2024-04-19T01:05:21.679 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.680 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.680 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:21.680 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.693860+0000", 2024-04-19T01:05:21.680 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4469e-05 2024-04-19T01:05:21.680 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.680 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.680 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:21.680 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.704275+0000", 2024-04-19T01:05:21.680 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.01041486 2024-04-19T01:05:21.680 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.680 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.680 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:21.680 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.704329+0000", 2024-04-19T01:05:21.680 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4166000000000003e-05 2024-04-19T01:05:21.681 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.681 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:21.681 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.681 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.681 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.681 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds1 102/98 ECSubWrite(tid=210, reqid=client.4604.0:760, at_version=102'80, trim_to=0'0, roll_forward_to=0'0))", 2024-04-19T01:05:21.681 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:05:12.711185+0000", 2024-04-19T01:05:21.681 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 8.8894016390000008, 2024-04-19T01:05:21.681 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015081842999999999, 2024-04-19T01:05:21.681 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:21.681 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:21.681 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:05:21.681 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:21.681 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.681 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:21.682 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.711185+0000", 2024-04-19T01:05:21.682 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.682 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.682 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.682 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:21.682 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.711185+0000", 2024-04-19T01:05:21.682 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.682 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.682 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.682 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:21.682 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.711186+0000", 2024-04-19T01:05:21.682 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.088e-06 2024-04-19T01:05:21.682 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.682 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.682 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:21.683 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.711285+0000", 2024-04-19T01:05:21.683 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.9470999999999997e-05 2024-04-19T01:05:21.683 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.683 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.683 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:21.683 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.711302+0000", 2024-04-19T01:05:21.683 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7342000000000001e-05 2024-04-19T01:05:21.683 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.683 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.683 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:21.683 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.711315+0000", 2024-04-19T01:05:21.683 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2034e-05 2024-04-19T01:05:21.683 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.683 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.684 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:21.684 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.711460+0000", 2024-04-19T01:05:21.684 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000145938 2024-04-19T01:05:21.684 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.684 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.684 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:21.684 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.711484+0000", 2024-04-19T01:05:21.684 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3779000000000001e-05 2024-04-19T01:05:21.684 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.684 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.684 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:21.684 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.726245+0000", 2024-04-19T01:05:21.684 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014760337 2024-04-19T01:05:21.684 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.684 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.685 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:21.685 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.726266+0000", 2024-04-19T01:05:21.685 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1854000000000001e-05 2024-04-19T01:05:21.685 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.685 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:21.685 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.685 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.685 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.685 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.3s5 105/98 ECSubWrite(tid=117, reqid=osd.15.0:117, at_version=105'95, trim_to=0'0, roll_forward_to=104'91))", 2024-04-19T01:05:21.685 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:05:15.668343+0000", 2024-04-19T01:05:21.685 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.932243637, 2024-04-19T01:05:21.685 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011487809, 2024-04-19T01:05:21.685 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:21.685 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:21.686 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:05:21.686 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:21.686 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.686 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:21.686 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:15.668343+0000", 2024-04-19T01:05:21.686 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.686 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.686 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.686 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:21.686 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:15.668343+0000", 2024-04-19T01:05:21.686 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.686 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.686 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.686 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:21.687 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:15.668343+0000", 2024-04-19T01:05:21.687 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8700000000000002e-07 2024-04-19T01:05:21.687 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.687 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.687 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:21.687 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:15.668358+0000", 2024-04-19T01:05:21.687 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4696e-05 2024-04-19T01:05:21.687 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.687 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.687 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:21.687 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:15.668364+0000", 2024-04-19T01:05:21.687 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0549999999999996e-06 2024-04-19T01:05:21.688 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.688 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.688 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:21.688 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:15.668367+0000", 2024-04-19T01:05:21.688 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7699999999999999e-06 2024-04-19T01:05:21.688 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.688 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.688 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:21.688 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:15.669049+0000", 2024-04-19T01:05:21.688 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00068194399999999997 2024-04-19T01:05:21.688 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.688 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.688 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:21.689 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:15.669060+0000", 2024-04-19T01:05:21.689 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0772e-05 2024-04-19T01:05:21.689 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.689 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.689 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:21.689 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:15.679771+0000", 2024-04-19T01:05:21.689 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010711238999999999 2024-04-19T01:05:21.689 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.689 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.689 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:21.689 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:15.679830+0000", 2024-04-19T01:05:21.689 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.9045999999999997e-05 2024-04-19T01:05:21.689 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.689 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:21.689 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.690 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.690 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.690 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.bs5 105/31 ECSubWrite(tid=186, reqid=osd.10.0:186, at_version=105'56, trim_to=0'0, roll_forward_to=102'54))", 2024-04-19T01:05:21.690 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:05:15.668892+0000", 2024-04-19T01:05:21.690 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.9316945649999999, 2024-04-19T01:05:21.690 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010987492, 2024-04-19T01:05:21.690 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:21.690 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:21.690 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:05:21.690 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:21.690 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.690 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:21.690 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:15.668892+0000", 2024-04-19T01:05:21.690 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.690 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.691 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.691 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:21.691 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:15.668892+0000", 2024-04-19T01:05:21.691 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.691 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.691 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.691 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:21.691 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:15.668892+0000", 2024-04-19T01:05:21.691 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3300000000000001e-07 2024-04-19T01:05:21.691 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.691 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.691 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:21.691 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:15.668912+0000", 2024-04-19T01:05:21.691 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9969000000000001e-05 2024-04-19T01:05:21.692 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.692 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.692 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:21.692 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:15.668924+0000", 2024-04-19T01:05:21.692 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1949e-05 2024-04-19T01:05:21.692 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.692 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.692 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:21.692 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:15.668933+0000", 2024-04-19T01:05:21.692 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.7099999999999996e-06 2024-04-19T01:05:21.692 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.692 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.692 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:21.692 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:15.669029+0000", 2024-04-19T01:05:21.692 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.6001000000000005e-05 2024-04-19T01:05:21.693 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.693 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.693 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:21.693 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:15.669040+0000", 2024-04-19T01:05:21.693 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1956999999999999e-05 2024-04-19T01:05:21.693 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.693 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.693 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:21.693 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:15.679837+0000", 2024-04-19T01:05:21.693 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010796182 2024-04-19T01:05:21.693 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.693 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.693 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:21.693 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:15.679879+0000", 2024-04-19T01:05:21.694 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2490999999999997e-05 2024-04-19T01:05:21.694 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.694 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:21.694 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.694 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.694 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.694 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.1s1 105/31 ECSubWrite(tid=65, reqid=client.4604.0:794, at_version=105'62, trim_to=0'0, roll_forward_to=105'60))", 2024-04-19T01:05:21.694 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:05:15.703541+0000", 2024-04-19T01:05:21.694 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.8970450000000003, 2024-04-19T01:05:21.694 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.032273578999999997, 2024-04-19T01:05:21.694 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:21.694 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:21.694 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2024-04-19T01:05:21.694 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:21.695 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.695 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:21.695 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:15.703541+0000", 2024-04-19T01:05:21.695 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.695 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.695 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.695 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:21.695 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:15.703541+0000", 2024-04-19T01:05:21.695 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:21.695 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.695 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.695 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:21.695 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:15.703542+0000", 2024-04-19T01:05:21.695 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.3099999999999996e-07 2024-04-19T01:05:21.695 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.696 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.696 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:21.696 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:15.706064+0000", 2024-04-19T01:05:21.696 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0025220910000000002 2024-04-19T01:05:21.696 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.696 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.696 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:21.696 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:15.706084+0000", 2024-04-19T01:05:21.696 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9840999999999999e-05 2024-04-19T01:05:21.696 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.696 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.696 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:21.696 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:15.706095+0000", 2024-04-19T01:05:21.696 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1092e-05 2024-04-19T01:05:21.696 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.697 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.697 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:21.697 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:15.706182+0000", 2024-04-19T01:05:21.697 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.7026000000000003e-05 2024-04-19T01:05:21.697 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.697 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.697 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:21.697 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:15.706199+0000", 2024-04-19T01:05:21.697 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6404000000000001e-05 2024-04-19T01:05:21.697 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.697 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.697 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:21.697 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:15.735766+0000", 2024-04-19T01:05:21.697 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.029567626 2024-04-19T01:05:21.697 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:21.697 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:21.698 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:21.698 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:15.735815+0000", 2024-04-19T01:05:21.698 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8568e-05 2024-04-19T01:05:21.698 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.698 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:21.698 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.698 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:21.698 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:21.698 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:05:21.700 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-19T01:05:21.700 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:05:21.697+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:05:21.805 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:05:21.800+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:05:21.875 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:05:21.876 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:05:21.876 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2024-04-19T01:05:21.876 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:05:21.890 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-19T01:05:21.902 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:05:21.900+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:05:22.002 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:05:22.001+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:05:22.024 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:05:22.024 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:05:22.024 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2024-04-19T01:05:22.025 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2024-04-19T01:05:22.025 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:05:22.039 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-19T01:05:22.126 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:05:22.125+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:05:22.204 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:05:22.206+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:05:22.208 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:05:22.209 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2024-04-19T01:05:22.209 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2024-04-19T01:05:22.209 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-04-19T01:05:22.209 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.209 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:423 3.5s0 3:a03c5c89:::smithi15528762-37:head [append 0~655360] snapc e=[d] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-19T01:05:22.209 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:29.537904+0000", 2024-04-19T01:05:22.209 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 52.669800143000003, 2024-04-19T01:05:22.209 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.049520382000000002, 2024-04-19T01:05:22.209 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:22.209 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:22.209 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:22.209 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:22.209 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:22.209 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:22.210 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 423 2024-04-19T01:05:22.210 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.210 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:22.210 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.210 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:22.210 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.537904+0000", 2024-04-19T01:05:22.210 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.210 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.210 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.210 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:22.210 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.537904+0000", 2024-04-19T01:05:22.210 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.210 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.211 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.211 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:22.212 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.537906+0000", 2024-04-19T01:05:22.212 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7859999999999999e-06 2024-04-19T01:05:22.212 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.212 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.212 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:22.212 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.538813+0000", 2024-04-19T01:05:22.212 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00090743099999999999 2024-04-19T01:05:22.212 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.212 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.212 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:22.212 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.538828+0000", 2024-04-19T01:05:22.213 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5206e-05 2024-04-19T01:05:22.213 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.213 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.213 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:22.213 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.538838+0000", 2024-04-19T01:05:22.213 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.3070000000000008e-06 2024-04-19T01:05:22.213 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.213 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.213 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.214 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.538910+0000", 2024-04-19T01:05:22.214 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2206000000000006e-05 2024-04-19T01:05:22.214 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.214 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.214 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:22.214 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.539016+0000", 2024-04-19T01:05:22.214 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010665199999999999 2024-04-19T01:05:22.214 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.214 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.214 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:22.214 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.540080+0000", 2024-04-19T01:05:22.214 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001063273 2024-04-19T01:05:22.215 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.215 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.215 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:22.215 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.545973+0000", 2024-04-19T01:05:22.215 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0058930570000000002 2024-04-19T01:05:22.215 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.215 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.216 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:22.216 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.587359+0000", 2024-04-19T01:05:22.218 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.041386128000000001 2024-04-19T01:05:22.218 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.218 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.218 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:22.218 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.587424+0000", 2024-04-19T01:05:22.219 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5335999999999999e-05 2024-04-19T01:05:22.219 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.219 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:22.219 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.219 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.219 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.219 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:424 3.5s0 3:a03c5c89:::smithi15528762-37:head [append 0~638976] snapc e=[d] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-19T01:05:22.219 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:29.545481+0000", 2024-04-19T01:05:22.219 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 52.662223155, 2024-04-19T01:05:22.219 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.042187482999999998, 2024-04-19T01:05:22.219 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:22.220 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:22.220 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:22.220 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:22.220 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:22.220 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:22.220 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 424 2024-04-19T01:05:22.220 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.220 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:22.220 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.220 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:22.220 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.545481+0000", 2024-04-19T01:05:22.220 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.220 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.220 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.220 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:22.221 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.545481+0000", 2024-04-19T01:05:22.221 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.221 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.221 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.221 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:22.221 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.545483+0000", 2024-04-19T01:05:22.221 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3879999999999998e-06 2024-04-19T01:05:22.221 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.221 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.221 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:22.221 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.546298+0000", 2024-04-19T01:05:22.221 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00081462099999999996 2024-04-19T01:05:22.221 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.221 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.221 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:22.222 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.546314+0000", 2024-04-19T01:05:22.222 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5722e-05 2024-04-19T01:05:22.222 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.222 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.222 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:22.222 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.546320+0000", 2024-04-19T01:05:22.222 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0939999999999996e-06 2024-04-19T01:05:22.222 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.222 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.222 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.222 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.546381+0000", 2024-04-19T01:05:22.222 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1696e-05 2024-04-19T01:05:22.222 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.222 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.223 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:22.223 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.546497+0000", 2024-04-19T01:05:22.223 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000115903 2024-04-19T01:05:22.223 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.223 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.223 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:22.223 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547522+0000", 2024-04-19T01:05:22.224 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010249289999999999 2024-04-19T01:05:22.224 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.224 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.224 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:22.224 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.556940+0000", 2024-04-19T01:05:22.224 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0094178899999999999 2024-04-19T01:05:22.224 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.224 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.224 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:22.224 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.587578+0000", 2024-04-19T01:05:22.224 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.030638021000000001 2024-04-19T01:05:22.224 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.224 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.224 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:22.224 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.587668+0000", 2024-04-19T01:05:22.225 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.0218999999999998e-05 2024-04-19T01:05:22.225 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.225 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:22.225 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.225 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.225 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.225 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:427 3.5s0 3:a03c5c89:::smithi15528762-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e59)", 2024-04-19T01:05:22.225 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:29.547214+0000", 2024-04-19T01:05:22.225 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 52.660490365000001, 2024-04-19T01:05:22.225 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.043029057000000002, 2024-04-19T01:05:22.226 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:22.226 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:22.226 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:05:22.226 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:22.226 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:22.226 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:22.226 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 427 2024-04-19T01:05:22.226 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.226 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:22.226 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.226 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:22.226 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547214+0000", 2024-04-19T01:05:22.226 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.226 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.226 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.227 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:22.227 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547214+0000", 2024-04-19T01:05:22.227 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.227 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.227 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.227 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:22.227 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547214+0000", 2024-04-19T01:05:22.227 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3799999999999998e-07 2024-04-19T01:05:22.229 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.229 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.229 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:22.229 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547216+0000", 2024-04-19T01:05:22.229 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.131e-06 2024-04-19T01:05:22.229 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.229 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.229 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:22.229 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547223+0000", 2024-04-19T01:05:22.229 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2790000000000004e-06 2024-04-19T01:05:22.229 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.230 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.230 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:22.230 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.547226+0000", 2024-04-19T01:05:22.230 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4529999999999998e-06 2024-04-19T01:05:22.230 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.230 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.230 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.230 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.556968+0000", 2024-04-19T01:05:22.230 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0097420590000000008 2024-04-19T01:05:22.231 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.231 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.231 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:05:22.231 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.557060+0000", 2024-04-19T01:05:22.231 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.1807999999999999e-05 2024-04-19T01:05:22.231 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.231 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.231 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.231 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.588480+0000", 2024-04-19T01:05:22.231 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.031420333000000002 2024-04-19T01:05:22.231 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.232 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.232 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:22.232 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.588538+0000", 2024-04-19T01:05:22.232 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.7652000000000003e-05 2024-04-19T01:05:22.232 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.232 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.232 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:22.232 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:29.590243+0000", 2024-04-19T01:05:22.232 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0017049039999999999 2024-04-19T01:05:22.232 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.232 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:22.232 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.232 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.233 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.233 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:543 3.5s0 3:a8ddfdee:::smithi15528762-22:head [append 0~524288] snapc 17=[17,15,14,12] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-04-19T01:05:22.233 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:44.548241+0000", 2024-04-19T01:05:22.233 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 37.659463056999996, 2024-04-19T01:05:22.233 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.049103814000000003, 2024-04-19T01:05:22.233 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:22.233 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:22.233 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:22.233 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:22.234 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:22.234 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:22.234 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 543 2024-04-19T01:05:22.234 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.234 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:22.234 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.235 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:22.235 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.548241+0000", 2024-04-19T01:05:22.235 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.235 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.235 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.235 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:22.235 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.548241+0000", 2024-04-19T01:05:22.235 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.235 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.235 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.236 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:22.236 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.548244+0000", 2024-04-19T01:05:22.236 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.1609999999999999e-06 2024-04-19T01:05:22.236 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.236 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.236 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:22.236 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.548890+0000", 2024-04-19T01:05:22.236 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00064546500000000001 2024-04-19T01:05:22.236 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.236 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.236 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:22.236 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.548916+0000", 2024-04-19T01:05:22.236 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6563999999999998e-05 2024-04-19T01:05:22.236 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.236 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.237 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:22.237 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.548934+0000", 2024-04-19T01:05:22.237 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7351e-05 2024-04-19T01:05:22.237 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.237 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.237 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.237 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.549031+0000", 2024-04-19T01:05:22.238 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7528000000000005e-05 2024-04-19T01:05:22.238 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.238 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.238 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op must wait for map", 2024-04-19T01:05:22.238 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.549037+0000", 2024-04-19T01:05:22.238 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.7590000000000003e-06 2024-04-19T01:05:22.238 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.238 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.238 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.238 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.551497+0000", 2024-04-19T01:05:22.238 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0024606580000000001 2024-04-19T01:05:22.238 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.238 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.242 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:22.242 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.551597+0000", 2024-04-19T01:05:22.242 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.9525999999999995e-05 2024-04-19T01:05:22.242 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.242 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.242 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:22.242 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.552924+0000", 2024-04-19T01:05:22.242 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001327356 2024-04-19T01:05:22.242 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.242 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.242 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:22.243 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.559310+0000", 2024-04-19T01:05:22.243 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0063859900000000002 2024-04-19T01:05:22.243 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.243 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.243 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:22.243 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.597268+0000", 2024-04-19T01:05:22.243 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.037957492000000002 2024-04-19T01:05:22.243 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.243 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.243 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:22.243 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:44.597345+0000", 2024-04-19T01:05:22.243 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6964000000000004e-05 2024-04-19T01:05:22.243 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.243 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:22.244 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.244 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.245 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.245 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:591 3.5s0 3:a626fd0d:::smithi15528762-17:head [append 0~655360] snapc 1a=[19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-19T01:05:22.245 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:49.652801+0000", 2024-04-19T01:05:22.245 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 32.554903357999997, 2024-04-19T01:05:22.245 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.051043682999999999, 2024-04-19T01:05:22.245 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:22.245 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:22.245 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:22.246 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:22.246 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:22.246 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:22.246 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 591 2024-04-19T01:05:22.246 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.246 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:22.246 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.246 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:22.246 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.652801+0000", 2024-04-19T01:05:22.246 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.246 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.246 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.247 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:22.247 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.652801+0000", 2024-04-19T01:05:22.247 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.247 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.247 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.247 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:22.247 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.652826+0000", 2024-04-19T01:05:22.247 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5695e-05 2024-04-19T01:05:22.247 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.247 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.247 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:22.248 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.654116+0000", 2024-04-19T01:05:22.248 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001289807 2024-04-19T01:05:22.248 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.248 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.248 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:22.248 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.654135+0000", 2024-04-19T01:05:22.248 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8734999999999999e-05 2024-04-19T01:05:22.248 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.248 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.248 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:22.248 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.654147+0000", 2024-04-19T01:05:22.248 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2140000000000001e-05 2024-04-19T01:05:22.248 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.248 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.249 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.249 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.654229+0000", 2024-04-19T01:05:22.249 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1798000000000005e-05 2024-04-19T01:05:22.249 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.249 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.249 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:22.249 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.654371+0000", 2024-04-19T01:05:22.249 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000141817 2024-04-19T01:05:22.249 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.249 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.249 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:22.249 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.656095+0000", 2024-04-19T01:05:22.249 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00172386 2024-04-19T01:05:22.249 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.250 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.250 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:22.250 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.679643+0000", 2024-04-19T01:05:22.250 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.023548158999999999 2024-04-19T01:05:22.250 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.250 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.250 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:22.250 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.703771+0000", 2024-04-19T01:05:22.250 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.024128437999999999 2024-04-19T01:05:22.250 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.250 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.250 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:22.250 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.703844+0000", 2024-04-19T01:05:22.251 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.3233999999999997e-05 2024-04-19T01:05:22.251 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.251 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:22.251 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.251 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.251 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.251 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:592 3.5s0 3:a626fd0d:::smithi15528762-17:head [append 0~606208] snapc 1a=[19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-19T01:05:22.251 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:49.658117+0000", 2024-04-19T01:05:22.251 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 32.549586890999997, 2024-04-19T01:05:22.251 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.047890332000000001, 2024-04-19T01:05:22.251 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:22.251 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:22.251 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:22.251 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:22.252 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:22.252 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:22.252 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 592 2024-04-19T01:05:22.252 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.252 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:22.252 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.252 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:22.252 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.658117+0000", 2024-04-19T01:05:22.252 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.252 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.252 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.252 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:22.252 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.658117+0000", 2024-04-19T01:05:22.252 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.253 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.253 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.253 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:22.253 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.658120+0000", 2024-04-19T01:05:22.253 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9129999999999999e-06 2024-04-19T01:05:22.253 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.253 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.253 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:22.253 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.658931+0000", 2024-04-19T01:05:22.253 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00081110999999999998 2024-04-19T01:05:22.253 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.253 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.253 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:22.253 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.658965+0000", 2024-04-19T01:05:22.253 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4001999999999997e-05 2024-04-19T01:05:22.254 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.254 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.254 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:22.254 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.658981+0000", 2024-04-19T01:05:22.254 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5443000000000001e-05 2024-04-19T01:05:22.254 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.254 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.255 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.255 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663331+0000", 2024-04-19T01:05:22.255 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0043501240000000004 2024-04-19T01:05:22.255 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.255 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.255 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:22.255 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663457+0000", 2024-04-19T01:05:22.255 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000125857 2024-04-19T01:05:22.255 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.255 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.255 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:22.255 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.664259+0000", 2024-04-19T01:05:22.255 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00080283700000000004 2024-04-19T01:05:22.256 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.256 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.256 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:22.256 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.679757+0000", 2024-04-19T01:05:22.256 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015497100999999999 2024-04-19T01:05:22.256 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.256 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.256 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:22.256 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.705944+0000", 2024-04-19T01:05:22.256 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.026187892000000001 2024-04-19T01:05:22.256 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.256 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.256 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:22.256 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.706007+0000", 2024-04-19T01:05:22.257 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3052999999999994e-05 2024-04-19T01:05:22.257 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.257 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:22.257 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.257 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.257 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.257 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:593 3.5s0 3:a626fd0d:::smithi15528762-17:head [append 0~475136] snapc 1a=[19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-19T01:05:22.257 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:49.662459+0000", 2024-04-19T01:05:22.257 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 32.545244857, 2024-04-19T01:05:22.257 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.043664481999999998, 2024-04-19T01:05:22.257 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:22.257 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:22.258 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:22.258 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:22.258 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:22.258 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:22.258 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 593 2024-04-19T01:05:22.258 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.258 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:22.258 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.258 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:22.258 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.662459+0000", 2024-04-19T01:05:22.258 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.258 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.258 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.259 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:22.259 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.662459+0000", 2024-04-19T01:05:22.259 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.259 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.259 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.259 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:22.259 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.662465+0000", 2024-04-19T01:05:22.259 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2029999999999997e-06 2024-04-19T01:05:22.259 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.259 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.259 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:22.259 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663095+0000", 2024-04-19T01:05:22.259 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00062950400000000002 2024-04-19T01:05:22.259 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.259 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.260 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:22.260 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663191+0000", 2024-04-19T01:05:22.260 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.5866999999999995e-05 2024-04-19T01:05:22.260 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.260 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.260 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:22.260 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663213+0000", 2024-04-19T01:05:22.260 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1832e-05 2024-04-19T01:05:22.260 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.260 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.261 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.261 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.671032+0000", 2024-04-19T01:05:22.261 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0078190310000000006 2024-04-19T01:05:22.261 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.261 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.261 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:22.261 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.671178+0000", 2024-04-19T01:05:22.261 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014599400000000001 2024-04-19T01:05:22.261 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.261 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.261 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:22.261 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.672264+0000", 2024-04-19T01:05:22.261 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001086652 2024-04-19T01:05:22.261 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.262 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.262 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:22.262 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.679758+0000", 2024-04-19T01:05:22.262 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0074936059999999999 2024-04-19T01:05:22.262 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.262 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.262 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:22.262 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.706080+0000", 2024-04-19T01:05:22.262 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.026322534000000002 2024-04-19T01:05:22.262 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.262 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.262 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:22.262 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.706124+0000", 2024-04-19T01:05:22.263 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3259e-05 2024-04-19T01:05:22.263 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.263 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:22.263 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.263 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.263 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.263 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:594 3.5s0 3:a626fd0d:::smithi15528762-17:head [append 0~81920] snapc 1a=[19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-19T01:05:22.263 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:49.663273+0000", 2024-04-19T01:05:22.263 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 32.544430841999997, 2024-04-19T01:05:22.263 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.043572570999999997, 2024-04-19T01:05:22.263 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:22.263 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:22.263 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:22.263 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:22.264 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:22.264 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:22.264 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 594 2024-04-19T01:05:22.264 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.264 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:22.264 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.264 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:22.264 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663273+0000", 2024-04-19T01:05:22.264 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.264 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.264 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.264 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:22.265 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663273+0000", 2024-04-19T01:05:22.265 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.265 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.265 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.265 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:22.265 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663275+0000", 2024-04-19T01:05:22.265 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.421e-06 2024-04-19T01:05:22.265 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.265 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.266 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:22.266 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663327+0000", 2024-04-19T01:05:22.266 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2669e-05 2024-04-19T01:05:22.266 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.266 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.266 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:22.266 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663342+0000", 2024-04-19T01:05:22.266 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5046e-05 2024-04-19T01:05:22.266 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.266 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.266 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:22.266 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663351+0000", 2024-04-19T01:05:22.266 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.0610000000000002e-06 2024-04-19T01:05:22.267 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.267 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.267 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.267 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.676918+0000", 2024-04-19T01:05:22.267 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013566147000000001 2024-04-19T01:05:22.267 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.267 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.267 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:22.267 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.677079+0000", 2024-04-19T01:05:22.267 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000161683 2024-04-19T01:05:22.267 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.267 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.267 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:22.267 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.678015+0000", 2024-04-19T01:05:22.268 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00093627399999999996 2024-04-19T01:05:22.268 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.268 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.268 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:22.268 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.681601+0000", 2024-04-19T01:05:22.268 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003585464 2024-04-19T01:05:22.268 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.268 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.268 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:22.268 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.706786+0000", 2024-04-19T01:05:22.268 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.025185553999999999 2024-04-19T01:05:22.268 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.269 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.269 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:22.269 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.706846+0000", 2024-04-19T01:05:22.269 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.9252000000000001e-05 2024-04-19T01:05:22.269 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.269 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:22.269 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.270 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.270 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.270 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:595 3.5s0 3:a626fd0d:::smithi15528762-17:head [setxattr _header (57) in=7b] snapc 1a=[19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-19T01:05:22.270 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:49.663370+0000", 2024-04-19T01:05:22.270 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 32.544334384000003, 2024-04-19T01:05:22.270 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.044174650000000003, 2024-04-19T01:05:22.270 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:22.270 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:22.270 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:22.270 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:22.271 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:22.271 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:22.271 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 595 2024-04-19T01:05:22.271 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.271 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:22.271 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.271 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:22.271 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663370+0000", 2024-04-19T01:05:22.271 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.271 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.271 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.271 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:22.271 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663370+0000", 2024-04-19T01:05:22.271 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.272 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.272 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.272 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:22.272 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663371+0000", 2024-04-19T01:05:22.272 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.3699999999999999e-07 2024-04-19T01:05:22.272 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.272 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.272 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:22.272 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663376+0000", 2024-04-19T01:05:22.272 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6879999999999996e-06 2024-04-19T01:05:22.272 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.272 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.272 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:22.272 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663386+0000", 2024-04-19T01:05:22.273 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7829999999999993e-06 2024-04-19T01:05:22.273 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.273 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.273 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:22.273 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663393+0000", 2024-04-19T01:05:22.273 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2540000000000003e-06 2024-04-19T01:05:22.273 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.273 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.273 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.273 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.679885+0000", 2024-04-19T01:05:22.273 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.016492168000000001 2024-04-19T01:05:22.273 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.273 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.273 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:22.273 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.679999+0000", 2024-04-19T01:05:22.274 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000113638 2024-04-19T01:05:22.274 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.274 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.274 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:22.274 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.680695+0000", 2024-04-19T01:05:22.274 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00069550999999999999 2024-04-19T01:05:22.274 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.274 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.274 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:22.274 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.681605+0000", 2024-04-19T01:05:22.274 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00091078000000000003 2024-04-19T01:05:22.274 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.274 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.274 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:22.275 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.707485+0000", 2024-04-19T01:05:22.275 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.025879491000000001 2024-04-19T01:05:22.275 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.275 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.275 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:22.276 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.707544+0000", 2024-04-19T01:05:22.276 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.9401000000000003e-05 2024-04-19T01:05:22.276 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.276 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:22.276 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.276 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.276 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.276 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:596 3.5s0 3:a626fd0d:::smithi15528762-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e79)", 2024-04-19T01:05:22.276 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:49.663401+0000", 2024-04-19T01:05:22.276 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 32.544302672999997, 2024-04-19T01:05:22.276 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.045794460000000002, 2024-04-19T01:05:22.276 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:22.277 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:22.277 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:05:22.277 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:22.277 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:22.277 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:22.277 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 596 2024-04-19T01:05:22.277 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.277 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:22.277 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.277 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:22.277 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663401+0000", 2024-04-19T01:05:22.278 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.278 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.278 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.278 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:22.278 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663401+0000", 2024-04-19T01:05:22.278 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.278 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.278 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.278 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:22.278 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663402+0000", 2024-04-19T01:05:22.278 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2800000000000002e-07 2024-04-19T01:05:22.278 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.278 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.278 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:22.279 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663406+0000", 2024-04-19T01:05:22.279 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2830000000000003e-06 2024-04-19T01:05:22.279 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.279 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.279 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:22.279 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663412+0000", 2024-04-19T01:05:22.279 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.784e-06 2024-04-19T01:05:22.279 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.280 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.280 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:22.280 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.663416+0000", 2024-04-19T01:05:22.280 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1339999999999997e-06 2024-04-19T01:05:22.280 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.280 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.280 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.280 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.681057+0000", 2024-04-19T01:05:22.280 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017640841000000001 2024-04-19T01:05:22.280 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.280 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.280 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:05:22.280 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.681119+0000", 2024-04-19T01:05:22.280 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2592e-05 2024-04-19T01:05:22.280 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.281 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.281 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.281 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.707942+0000", 2024-04-19T01:05:22.281 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.026822915999999999 2024-04-19T01:05:22.281 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.281 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.281 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:22.281 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.708008+0000", 2024-04-19T01:05:22.281 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.6088999999999997e-05 2024-04-19T01:05:22.282 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.282 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.282 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:22.282 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:49.709196+0000", 2024-04-19T01:05:22.282 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0011873929999999999 2024-04-19T01:05:22.282 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.282 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:22.282 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.282 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.282 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.282 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:626 3.5s0 3:ab059e23:::smithi15528762-38:head [append 0~737280] snapc 1c=[1c,1b,19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-19T01:05:22.282 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:51.587820+0000", 2024-04-19T01:05:22.282 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 30.619884465999998, 2024-04-19T01:05:22.283 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.070148642999999997, 2024-04-19T01:05:22.283 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:22.283 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:22.283 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:22.283 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:22.283 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:22.283 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:22.283 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 626 2024-04-19T01:05:22.283 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.283 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:22.283 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.283 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:22.283 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.587820+0000", 2024-04-19T01:05:22.283 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.284 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.284 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.284 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:22.284 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.587820+0000", 2024-04-19T01:05:22.284 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.284 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.284 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.284 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:22.284 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.587822+0000", 2024-04-19T01:05:22.284 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5150000000000001e-06 2024-04-19T01:05:22.284 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.284 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.284 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:22.284 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.588850+0000", 2024-04-19T01:05:22.285 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010283029999999999 2024-04-19T01:05:22.285 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.285 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.285 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:22.285 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.588877+0000", 2024-04-19T01:05:22.285 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6593000000000001e-05 2024-04-19T01:05:22.286 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.286 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.286 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:22.286 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.588891+0000", 2024-04-19T01:05:22.286 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4245e-05 2024-04-19T01:05:22.286 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.286 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.286 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.286 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.588997+0000", 2024-04-19T01:05:22.286 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000106252 2024-04-19T01:05:22.287 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.287 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.287 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op must wait for map", 2024-04-19T01:05:22.287 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.589004+0000", 2024-04-19T01:05:22.287 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.8079999999999999e-06 2024-04-19T01:05:22.287 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.287 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.287 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.287 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.591485+0000", 2024-04-19T01:05:22.287 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0024811970000000001 2024-04-19T01:05:22.287 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.287 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.287 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:22.287 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.591587+0000", 2024-04-19T01:05:22.288 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000101387 2024-04-19T01:05:22.288 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.288 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.288 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:22.288 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.592588+0000", 2024-04-19T01:05:22.288 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010014850000000001 2024-04-19T01:05:22.288 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.288 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.288 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:22.288 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.602598+0000", 2024-04-19T01:05:22.288 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010010023999999999 2024-04-19T01:05:22.288 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.288 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.288 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:22.289 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.657845+0000", 2024-04-19T01:05:22.289 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.055246157999999997 2024-04-19T01:05:22.289 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.289 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.289 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:22.289 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.657968+0000", 2024-04-19T01:05:22.289 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000123676 2024-04-19T01:05:22.289 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.289 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:22.289 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.289 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.289 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.289 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:627 3.5s0 3:ab059e23:::smithi15528762-38:head [append 0~557056] snapc 1c=[1c,1b,19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-19T01:05:22.289 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:51.594492+0000", 2024-04-19T01:05:22.290 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 30.613211853999999, 2024-04-19T01:05:22.290 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.06373906, 2024-04-19T01:05:22.290 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:22.290 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:22.290 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:22.290 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:22.290 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:22.290 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:22.290 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 627 2024-04-19T01:05:22.290 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.290 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:22.290 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.290 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:22.291 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.594492+0000", 2024-04-19T01:05:22.291 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.291 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.291 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.291 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:22.291 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.594492+0000", 2024-04-19T01:05:22.291 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.291 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.291 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.291 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:22.291 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.594494+0000", 2024-04-19T01:05:22.291 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8899999999999999e-06 2024-04-19T01:05:22.291 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.291 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.292 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:22.292 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.595085+0000", 2024-04-19T01:05:22.292 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00059048800000000002 2024-04-19T01:05:22.292 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.292 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.292 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:22.292 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.595102+0000", 2024-04-19T01:05:22.292 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7515000000000001e-05 2024-04-19T01:05:22.292 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.292 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.292 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:22.292 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.595114+0000", 2024-04-19T01:05:22.292 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1678000000000001e-05 2024-04-19T01:05:22.292 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.297 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.297 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.297 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.598103+0000", 2024-04-19T01:05:22.297 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0029891789999999998 2024-04-19T01:05:22.298 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.298 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.298 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:22.298 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.598183+0000", 2024-04-19T01:05:22.298 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.9659000000000004e-05 2024-04-19T01:05:22.298 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.298 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.298 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:22.298 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.598883+0000", 2024-04-19T01:05:22.298 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00069997600000000005 2024-04-19T01:05:22.298 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.299 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.299 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:22.299 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.608539+0000", 2024-04-19T01:05:22.299 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0096562739999999994 2024-04-19T01:05:22.299 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.299 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.299 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:22.299 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.658124+0000", 2024-04-19T01:05:22.299 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.049585021 2024-04-19T01:05:22.299 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.299 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.299 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:22.299 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.658231+0000", 2024-04-19T01:05:22.299 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010738000000000001 2024-04-19T01:05:22.300 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.300 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:22.300 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.300 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.300 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.300 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:626 3.5s0 3:ab059e23:::smithi15528762-38:head [append 0~737280] snapc 1c=[1c,1b,19,18,17,14,12] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e81)", 2024-04-19T01:05:22.300 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:51.601568+0000", 2024-04-19T01:05:22.300 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 30.606135966, 2024-04-19T01:05:22.300 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.056289715999999997, 2024-04-19T01:05:22.300 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:22.300 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:22.300 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "waiting for ondisk", 2024-04-19T01:05:22.300 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:22.300 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:22.301 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:22.301 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 626 2024-04-19T01:05:22.301 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.301 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:22.301 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.301 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:22.301 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.601568+0000", 2024-04-19T01:05:22.301 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.301 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.301 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.301 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:22.301 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.601568+0000", 2024-04-19T01:05:22.301 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.301 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.301 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.302 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:22.302 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.601569+0000", 2024-04-19T01:05:22.302 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.9999999999999997e-07 2024-04-19T01:05:22.302 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.302 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.302 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:22.302 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.602838+0000", 2024-04-19T01:05:22.302 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0012697349999999999 2024-04-19T01:05:22.302 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.302 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.302 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:22.302 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.602848+0000", 2024-04-19T01:05:22.302 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.6970000000000005e-06 2024-04-19T01:05:22.303 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.303 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.303 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:22.303 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.602858+0000", 2024-04-19T01:05:22.303 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.482e-06 2024-04-19T01:05:22.303 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.303 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.303 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.303 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.602901+0000", 2024-04-19T01:05:22.303 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3711999999999998e-05 2024-04-19T01:05:22.303 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.303 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.303 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for ondisk", 2024-04-19T01:05:22.303 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.602952+0000", 2024-04-19T01:05:22.303 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0481e-05 2024-04-19T01:05:22.304 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.304 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.304 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:22.304 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.657858+0000", 2024-04-19T01:05:22.304 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.054905909000000003 2024-04-19T01:05:22.304 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.304 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:22.304 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.304 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.304 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.304 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:627 3.5s0 3:ab059e23:::smithi15528762-38:head [append 0~557056] snapc 1c=[1c,1b,19,18,17,14,12] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e81)", 2024-04-19T01:05:22.304 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:51.602874+0000", 2024-04-19T01:05:22.304 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 30.604830143000001, 2024-04-19T01:05:22.304 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.055263801000000001, 2024-04-19T01:05:22.305 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:22.305 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:22.305 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "waiting for ondisk", 2024-04-19T01:05:22.305 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:22.305 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:22.305 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:22.305 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 627 2024-04-19T01:05:22.305 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.305 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:22.305 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.305 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:22.305 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.602874+0000", 2024-04-19T01:05:22.305 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.305 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.305 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.306 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:22.306 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.602874+0000", 2024-04-19T01:05:22.306 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.306 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.306 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.306 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:22.306 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.602875+0000", 2024-04-19T01:05:22.306 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.4300000000000002e-07 2024-04-19T01:05:22.306 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.306 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.306 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:22.306 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603321+0000", 2024-04-19T01:05:22.306 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000446468 2024-04-19T01:05:22.307 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.307 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.307 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:22.307 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603332+0000", 2024-04-19T01:05:22.308 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1079e-05 2024-04-19T01:05:22.308 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.308 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.308 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:22.308 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603337+0000", 2024-04-19T01:05:22.308 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0760000000000002e-06 2024-04-19T01:05:22.308 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.308 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.308 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.308 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603389+0000", 2024-04-19T01:05:22.309 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.1990000000000002e-05 2024-04-19T01:05:22.309 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.309 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.309 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for ondisk", 2024-04-19T01:05:22.309 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603501+0000", 2024-04-19T01:05:22.309 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000111539 2024-04-19T01:05:22.309 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.309 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.309 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:22.309 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.658138+0000", 2024-04-19T01:05:22.309 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.054636905999999999 2024-04-19T01:05:22.309 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.309 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:22.309 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.310 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.310 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.310 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:628 3.5s0 3:ab059e23:::smithi15528762-38:head [append 0~606208] snapc 1c=[1c,1b,19,18,17,14,12] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e81)", 2024-04-19T01:05:22.310 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:51.603350+0000", 2024-04-19T01:05:22.310 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 30.604354051000001, 2024-04-19T01:05:22.310 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.055106055000000001, 2024-04-19T01:05:22.310 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:22.310 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:22.310 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:22.311 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:22.311 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:22.311 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:22.311 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 628 2024-04-19T01:05:22.311 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.311 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:22.311 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.311 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:22.311 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603350+0000", 2024-04-19T01:05:22.311 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.312 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.312 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.312 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:22.312 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603350+0000", 2024-04-19T01:05:22.312 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.312 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.312 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.312 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:22.312 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603351+0000", 2024-04-19T01:05:22.312 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3499999999999996e-07 2024-04-19T01:05:22.312 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.312 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.313 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:22.313 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603847+0000", 2024-04-19T01:05:22.313 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00049669000000000004 2024-04-19T01:05:22.313 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.313 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.313 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:22.313 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603858+0000", 2024-04-19T01:05:22.313 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0729e-05 2024-04-19T01:05:22.313 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.313 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.313 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:22.313 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603863+0000", 2024-04-19T01:05:22.313 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0869999999999997e-06 2024-04-19T01:05:22.314 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.314 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.314 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.314 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603919+0000", 2024-04-19T01:05:22.314 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.5618000000000002e-05 2024-04-19T01:05:22.314 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.314 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.314 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:22.314 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.604028+0000", 2024-04-19T01:05:22.314 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000109187 2024-04-19T01:05:22.314 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.314 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.314 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:22.314 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.604655+0000", 2024-04-19T01:05:22.315 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00062722400000000001 2024-04-19T01:05:22.315 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.315 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.315 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:22.315 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.609598+0000", 2024-04-19T01:05:22.315 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.004943037 2024-04-19T01:05:22.315 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.315 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.315 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:22.315 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.658378+0000", 2024-04-19T01:05:22.315 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.048779623000000001 2024-04-19T01:05:22.315 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.315 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.315 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:22.316 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.658456+0000", 2024-04-19T01:05:22.316 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.8225000000000004e-05 2024-04-19T01:05:22.316 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.316 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:22.316 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.316 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.316 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.316 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:629 3.5s0 3:ab059e23:::smithi15528762-38:head [setxattr _header (57) in=7b] snapc 1c=[1c,1b,19,18,17,14,12] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e81)", 2024-04-19T01:05:22.316 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:51.603874+0000", 2024-04-19T01:05:22.316 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 30.603829937, 2024-04-19T01:05:22.316 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.054746045, 2024-04-19T01:05:22.316 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:22.316 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:22.316 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:22.316 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:22.317 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:22.317 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:22.317 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 629 2024-04-19T01:05:22.317 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.317 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:22.317 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.317 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:22.318 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603874+0000", 2024-04-19T01:05:22.318 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.318 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.318 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.318 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:22.318 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603874+0000", 2024-04-19T01:05:22.318 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.318 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.318 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.318 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:22.318 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603875+0000", 2024-04-19T01:05:22.318 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8800000000000002e-07 2024-04-19T01:05:22.318 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.320 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.320 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:22.320 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603878+0000", 2024-04-19T01:05:22.320 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.6399999999999999e-06 2024-04-19T01:05:22.321 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.321 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.321 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:22.321 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603885+0000", 2024-04-19T01:05:22.321 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.1640000000000004e-06 2024-04-19T01:05:22.321 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.321 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.321 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:22.321 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603892+0000", 2024-04-19T01:05:22.321 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1469999999999998e-06 2024-04-19T01:05:22.321 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.322 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.322 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.322 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.608675+0000", 2024-04-19T01:05:22.322 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0047830379999999999 2024-04-19T01:05:22.323 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.323 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.323 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:22.323 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.608741+0000", 2024-04-19T01:05:22.325 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.6212999999999998e-05 2024-04-19T01:05:22.325 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.325 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.325 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:22.325 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.609027+0000", 2024-04-19T01:05:22.325 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00028588600000000001 2024-04-19T01:05:22.325 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.325 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.325 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:22.325 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.609773+0000", 2024-04-19T01:05:22.325 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000746563 2024-04-19T01:05:22.325 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.325 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.326 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:22.326 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.658557+0000", 2024-04-19T01:05:22.326 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.048783343999999999 2024-04-19T01:05:22.326 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.326 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.326 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:22.327 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.658620+0000", 2024-04-19T01:05:22.327 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3461999999999996e-05 2024-04-19T01:05:22.327 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.327 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:22.327 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.327 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.327 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.327 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:630 3.5s0 3:ab059e23:::smithi15528762-38:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e81)", 2024-04-19T01:05:22.327 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:51.603912+0000", 2024-04-19T01:05:22.328 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 30.603792489, 2024-04-19T01:05:22.328 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.056857458, 2024-04-19T01:05:22.328 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:22.328 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:22.328 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:05:22.328 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:22.328 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:22.329 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:22.329 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 630 2024-04-19T01:05:22.329 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.329 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:22.329 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.329 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:22.329 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603912+0000", 2024-04-19T01:05:22.329 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.329 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.329 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.329 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:22.329 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603912+0000", 2024-04-19T01:05:22.329 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.329 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.329 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.331 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:22.331 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603912+0000", 2024-04-19T01:05:22.331 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6899999999999997e-07 2024-04-19T01:05:22.331 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.331 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.331 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:22.332 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603915+0000", 2024-04-19T01:05:22.332 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9730000000000002e-06 2024-04-19T01:05:22.332 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.332 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.332 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:22.332 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603923+0000", 2024-04-19T01:05:22.332 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1100000000000003e-06 2024-04-19T01:05:22.332 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.332 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.332 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:22.332 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603929+0000", 2024-04-19T01:05:22.332 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0170000000000002e-06 2024-04-19T01:05:22.333 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.333 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.333 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.333 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.609249+0000", 2024-04-19T01:05:22.333 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0053197380000000001 2024-04-19T01:05:22.333 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.333 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.333 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:05:22.333 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.609358+0000", 2024-04-19T01:05:22.333 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010952599999999999 2024-04-19T01:05:22.333 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.333 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.333 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.333 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.658676+0000", 2024-04-19T01:05:22.334 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.049317986000000001 2024-04-19T01:05:22.334 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.334 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.334 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:22.334 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.658789+0000", 2024-04-19T01:05:22.334 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011216400000000001 2024-04-19T01:05:22.334 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.334 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.334 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:22.334 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.660769+0000", 2024-04-19T01:05:22.334 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0019803749999999999 2024-04-19T01:05:22.334 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.334 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:22.334 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.335 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.335 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.335 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:631 3.5s0 3:abe8a6c4:::smithi15528762-18:head [copy-from ver 8] snapc 1c=[1c,1b,19,18,17,14,12] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e81)", 2024-04-19T01:05:22.335 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:51.603938+0000", 2024-04-19T01:05:22.335 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 30.603766015000001, 2024-04-19T01:05:22.335 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.055423066, 2024-04-19T01:05:22.335 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:22.335 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:22.336 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:22.336 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:22.336 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:22.336 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:22.336 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 631 2024-04-19T01:05:22.337 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.337 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:22.337 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.337 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:22.337 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603938+0000", 2024-04-19T01:05:22.337 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.337 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.337 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.337 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:22.337 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603938+0000", 2024-04-19T01:05:22.337 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.337 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.337 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.338 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:22.338 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603938+0000", 2024-04-19T01:05:22.338 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7199999999999999e-07 2024-04-19T01:05:22.338 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.338 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.338 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:22.338 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603942+0000", 2024-04-19T01:05:22.338 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0809999999999998e-06 2024-04-19T01:05:22.339 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.339 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.339 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:22.339 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603953+0000", 2024-04-19T01:05:22.339 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1477e-05 2024-04-19T01:05:22.339 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.339 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.339 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:22.339 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.603962+0000", 2024-04-19T01:05:22.339 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.5010000000000004e-06 2024-04-19T01:05:22.339 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.339 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.339 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.339 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.609402+0000", 2024-04-19T01:05:22.340 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0054401069999999996 2024-04-19T01:05:22.340 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.340 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.340 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:22.340 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.609457+0000", 2024-04-19T01:05:22.340 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.5016000000000003e-05 2024-04-19T01:05:22.340 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.340 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.340 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:22.340 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.617371+0000", 2024-04-19T01:05:22.340 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0079141149999999993 2024-04-19T01:05:22.340 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.340 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.340 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:22.341 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.630663+0000", 2024-04-19T01:05:22.341 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013291956000000001 2024-04-19T01:05:22.341 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.341 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.341 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:22.341 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.659260+0000", 2024-04-19T01:05:22.341 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.028597651000000002 2024-04-19T01:05:22.341 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.341 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.341 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:22.341 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:51.659361+0000", 2024-04-19T01:05:22.341 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010069 2024-04-19T01:05:22.341 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.341 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:22.341 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.342 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.342 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.342 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:811 3.5s0 3:a03c5c89:::smithi15528762-37:head [append 0~688128] snapc 27=[27,25,24,21] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-19T01:05:22.342 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:05:17.589310+0000", 2024-04-19T01:05:22.342 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 4.6183943349999996, 2024-04-19T01:05:22.342 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.051808408, 2024-04-19T01:05:22.342 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:22.342 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:22.342 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:22.342 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:22.342 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:22.342 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:22.342 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 811 2024-04-19T01:05:22.342 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.343 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:22.343 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.343 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:22.343 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:17.589310+0000", 2024-04-19T01:05:22.343 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.343 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.343 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.343 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:22.343 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:17.589310+0000", 2024-04-19T01:05:22.343 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.343 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.343 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.343 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:22.343 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:17.589312+0000", 2024-04-19T01:05:22.344 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3159999999999999e-06 2024-04-19T01:05:22.344 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.344 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.344 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:22.344 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:17.590508+0000", 2024-04-19T01:05:22.344 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0011961999999999999 2024-04-19T01:05:22.344 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.344 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.345 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:22.345 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:17.590525+0000", 2024-04-19T01:05:22.345 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7073000000000001e-05 2024-04-19T01:05:22.345 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.345 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.345 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:22.345 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:17.590533+0000", 2024-04-19T01:05:22.345 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5510000000000002e-06 2024-04-19T01:05:22.345 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.346 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.346 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.346 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:17.590609+0000", 2024-04-19T01:05:22.346 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6635000000000001e-05 2024-04-19T01:05:22.346 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.346 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.346 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:22.346 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:17.590720+0000", 2024-04-19T01:05:22.346 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000110439 2024-04-19T01:05:22.346 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.346 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.346 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:22.346 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:17.591812+0000", 2024-04-19T01:05:22.346 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001092268 2024-04-19T01:05:22.347 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.347 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.347 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:22.347 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:17.596782+0000", 2024-04-19T01:05:22.347 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0049693630000000001 2024-04-19T01:05:22.347 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.347 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.347 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:22.347 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:17.641061+0000", 2024-04-19T01:05:22.347 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.044279668000000001 2024-04-19T01:05:22.348 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.348 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.348 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:22.348 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:17.641118+0000", 2024-04-19T01:05:22.348 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6895000000000003e-05 2024-04-19T01:05:22.348 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.348 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:22.348 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.348 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.348 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.348 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:812 3.5s0 3:a03c5c89:::smithi15528762-37:head [append 0~720896] snapc 27=[27,25,24,21] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-19T01:05:22.349 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:05:17.597294+0000", 2024-04-19T01:05:22.350 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 4.610410506, 2024-04-19T01:05:22.351 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.044032065000000002, 2024-04-19T01:05:22.351 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:22.351 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:22.351 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:22.351 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:22.351 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:22.351 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:22.351 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 812 2024-04-19T01:05:22.351 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.351 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:22.351 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.351 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:22.351 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:17.597294+0000", 2024-04-19T01:05:22.351 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.352 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.352 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.352 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:22.352 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:17.597294+0000", 2024-04-19T01:05:22.352 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.352 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.352 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.352 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:22.352 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:17.597295+0000", 2024-04-19T01:05:22.352 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.057e-06 2024-04-19T01:05:22.352 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.352 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.352 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:22.352 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:17.598074+0000", 2024-04-19T01:05:22.352 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00077945900000000003 2024-04-19T01:05:22.353 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.353 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.353 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:22.353 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:17.598087+0000", 2024-04-19T01:05:22.353 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2694000000000001e-05 2024-04-19T01:05:22.353 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.353 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.353 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:22.353 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:17.598093+0000", 2024-04-19T01:05:22.353 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0839999999999997e-06 2024-04-19T01:05:22.353 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.353 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.353 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.353 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:17.598146+0000", 2024-04-19T01:05:22.353 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3319000000000002e-05 2024-04-19T01:05:22.354 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.354 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.354 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:22.354 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:17.598219+0000", 2024-04-19T01:05:22.354 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2799000000000006e-05 2024-04-19T01:05:22.354 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.354 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.354 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:22.355 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:17.598887+0000", 2024-04-19T01:05:22.355 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00066830899999999996 2024-04-19T01:05:22.355 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.355 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.355 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:22.355 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:17.612883+0000", 2024-04-19T01:05:22.355 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013995558999999999 2024-04-19T01:05:22.355 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.355 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.355 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:22.355 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:17.641245+0000", 2024-04-19T01:05:22.355 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.028361999999999998 2024-04-19T01:05:22.355 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.355 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.355 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:22.356 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:17.641326+0000", 2024-04-19T01:05:22.356 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.0785000000000007e-05 2024-04-19T01:05:22.356 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.356 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:22.356 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.356 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.356 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:22.356 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:05:22.358 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-04-19T01:05:22.358 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:05:22.357+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:05:22.458 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:05:22.456+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:05:22.527 INFO:tasks.rados.rados.0.smithi155.stdout:355: done (0 left) 2024-04-19T01:05:22.528 INFO:tasks.rados.rados.0.smithi155.stdout:356: read oid 17 snap -1 2024-04-19T01:05:22.528 INFO:tasks.rados.rados.0.smithi155.stdout:356: expect (ObjNum 130 snap 37 seq_num 130) 2024-04-19T01:05:22.528 INFO:tasks.rados.rados.0.smithi155.stdout:357: read oid 25 snap -1 2024-04-19T01:05:22.528 INFO:tasks.rados.rados.0.smithi155.stdout:357: expect (ObjNum 24 snap 0 seq_num 24) 2024-04-19T01:05:22.528 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 10 current snap is 41 2024-04-19T01:05:22.528 INFO:tasks.rados.rados.0.smithi155.stdout:358: seq_num 140 ranges {2260992=802816,3063808=425984,3489792=720896,4210688=409600} 2024-04-19T01:05:22.536 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:05:22.536 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:05:22.536 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2024-04-19T01:05:22.536 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:05:22.556 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-19T01:05:22.562 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:05:22.558+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:05:22.564 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-04-19T01:05:22.573 INFO:tasks.rados.rados.0.smithi155.stdout:358: writing smithi15528762-10 from 2260992 to 3063808 tid 1 2024-04-19T01:05:22.580 INFO:tasks.rados.rados.0.smithi155.stdout:358: writing smithi15528762-10 from 3063808 to 3489792 tid 2 2024-04-19T01:05:22.590 INFO:tasks.rados.rados.0.smithi155.stdout:358: writing smithi15528762-10 from 3489792 to 4210688 tid 3 2024-04-19T01:05:22.592 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [0] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-04-19T01:05:22.593 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-19T01:05:22.593 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-04-19T01:05:22.593 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T01:05:22.600 INFO:tasks.rados.rados.0.smithi155.stdout:358: writing smithi15528762-10 from 4210688 to 4620288 tid 4 2024-04-19T01:05:22.600 INFO:tasks.rados.rados.0.smithi155.stdout:359: snap_remove snap 39 2024-04-19T01:05:22.661 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:05:22.660+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:05:22.729 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:05:22.730 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2024-04-19T01:05:22.730 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2024-04-19T01:05:22.730 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2024-04-19T01:05:22.730 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:05:22.746 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-04-19T01:05:22.790 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:05:22.789+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:05:22.863 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:05:22.862+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:05:22.868 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:05:22.869 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2024-04-19T01:05:22.869 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2024-04-19T01:05:22.869 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2024-04-19T01:05:22.869 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.869 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:254 3.ds0 3:b9317ffa:::smithi15528762-1:head [append 0~376832] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:05:22.869 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.985096+0000", 2024-04-19T01:05:22.869 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 68.880447321999995, 2024-04-19T01:05:22.869 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.043912868000000001, 2024-04-19T01:05:22.869 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:22.869 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:22.869 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:22.869 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:22.869 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:22.870 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:22.870 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 254 2024-04-19T01:05:22.870 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.870 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:22.870 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.870 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:22.870 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985096+0000", 2024-04-19T01:05:22.870 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.870 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.870 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.870 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:22.870 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985096+0000", 2024-04-19T01:05:22.870 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.870 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.871 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.871 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:22.871 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985098+0000", 2024-04-19T01:05:22.871 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0540000000000002e-06 2024-04-19T01:05:22.871 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.871 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.871 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:22.871 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985632+0000", 2024-04-19T01:05:22.871 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00053471200000000001 2024-04-19T01:05:22.871 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.871 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.871 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:22.871 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985650+0000", 2024-04-19T01:05:22.871 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7217000000000001e-05 2024-04-19T01:05:22.871 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.872 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.872 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:22.872 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985658+0000", 2024-04-19T01:05:22.872 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.8689999999999993e-06 2024-04-19T01:05:22.872 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.872 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.872 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.872 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.995027+0000", 2024-04-19T01:05:22.872 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0093689159999999997 2024-04-19T01:05:22.872 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.872 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.872 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:22.872 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.995141+0000", 2024-04-19T01:05:22.872 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011387000000000001 2024-04-19T01:05:22.873 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.873 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.873 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:22.873 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.996005+0000", 2024-04-19T01:05:22.873 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00086397999999999998 2024-04-19T01:05:22.873 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.873 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.873 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:22.873 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.002138+0000", 2024-04-19T01:05:22.873 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.006133182 2024-04-19T01:05:22.873 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.874 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.874 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:22.874 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.028913+0000", 2024-04-19T01:05:22.874 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.026774686999999998 2024-04-19T01:05:22.874 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.874 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.874 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:22.874 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.029008+0000", 2024-04-19T01:05:22.874 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.5381e-05 2024-04-19T01:05:22.874 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.874 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:22.874 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.874 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.875 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.875 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:255 3.ds0 3:b9317ffa:::smithi15528762-1:head [setxattr _header (55) in=7b] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:05:22.875 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.985699+0000", 2024-04-19T01:05:22.875 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 68.879843784000002, 2024-04-19T01:05:22.875 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.043535886000000003, 2024-04-19T01:05:22.875 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:22.875 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:22.876 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:22.876 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:22.876 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:22.876 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:22.876 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 255 2024-04-19T01:05:22.876 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.876 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:22.876 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.876 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:22.876 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985699+0000", 2024-04-19T01:05:22.876 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.876 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.876 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.876 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:22.876 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985699+0000", 2024-04-19T01:05:22.877 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.877 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.877 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.877 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:22.877 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985700+0000", 2024-04-19T01:05:22.877 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4199999999999995e-07 2024-04-19T01:05:22.877 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.877 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.877 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:22.877 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985705+0000", 2024-04-19T01:05:22.877 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7960000000000003e-06 2024-04-19T01:05:22.877 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.877 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.877 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:22.877 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985715+0000", 2024-04-19T01:05:22.878 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0285999999999999e-05 2024-04-19T01:05:22.878 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.878 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.878 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:22.878 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985735+0000", 2024-04-19T01:05:22.878 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9666000000000001e-05 2024-04-19T01:05:22.878 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.878 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.878 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.878 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.001214+0000", 2024-04-19T01:05:22.878 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015479694 2024-04-19T01:05:22.878 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.878 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.878 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:22.878 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.001417+0000", 2024-04-19T01:05:22.879 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00020274099999999999 2024-04-19T01:05:22.879 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.879 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.879 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:22.879 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.001931+0000", 2024-04-19T01:05:22.879 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000513599 2024-04-19T01:05:22.879 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.879 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.879 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:22.879 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.002446+0000", 2024-04-19T01:05:22.879 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00051521100000000001 2024-04-19T01:05:22.879 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.879 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.879 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:22.880 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.029136+0000", 2024-04-19T01:05:22.880 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.026690523000000001 2024-04-19T01:05:22.880 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.880 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.880 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:22.880 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.029235+0000", 2024-04-19T01:05:22.880 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.8727999999999994e-05 2024-04-19T01:05:22.880 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.880 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:22.880 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.880 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.880 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.880 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:256 3.ds0 3:b9317ffa:::smithi15528762-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:05:22.880 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:13.985744+0000", 2024-04-19T01:05:22.880 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 68.879798997999998, 2024-04-19T01:05:22.881 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.046574693, 2024-04-19T01:05:22.881 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:22.881 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:22.881 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:05:22.881 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:22.881 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:22.881 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:22.881 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 256 2024-04-19T01:05:22.881 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.881 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:22.881 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.881 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:22.881 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985744+0000", 2024-04-19T01:05:22.881 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.882 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.882 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.882 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:22.882 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985744+0000", 2024-04-19T01:05:22.882 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.882 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.882 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.882 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:22.882 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985744+0000", 2024-04-19T01:05:22.882 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1699999999999999e-07 2024-04-19T01:05:22.882 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.882 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.883 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:22.883 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985747+0000", 2024-04-19T01:05:22.883 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3429999999999998e-06 2024-04-19T01:05:22.883 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.883 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.883 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:22.883 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985753+0000", 2024-04-19T01:05:22.883 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8619999999999999e-06 2024-04-19T01:05:22.883 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.883 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.883 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:22.884 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:13.985756+0000", 2024-04-19T01:05:22.884 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8820000000000003e-06 2024-04-19T01:05:22.884 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.884 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.884 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.884 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.002179+0000", 2024-04-19T01:05:22.884 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.01642302 2024-04-19T01:05:22.884 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.884 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.884 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:05:22.884 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.002278+0000", 2024-04-19T01:05:22.885 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.8548000000000006e-05 2024-04-19T01:05:22.885 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.885 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.885 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.885 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.029285+0000", 2024-04-19T01:05:22.885 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.027006935999999999 2024-04-19T01:05:22.885 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.885 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.885 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:22.885 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.029400+0000", 2024-04-19T01:05:22.885 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000115438 2024-04-19T01:05:22.885 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.886 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.886 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:22.886 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:14.032319+0000", 2024-04-19T01:05:22.886 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0029182470000000001 2024-04-19T01:05:22.886 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.886 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:22.886 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.886 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.886 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.886 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:354 3.as0 3:56c32ee7:::smithi15528762-47:head [copy-from ver 35] snapc 9=[8,7] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:05:22.886 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:18.544030+0000", 2024-04-19T01:05:22.886 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 64.321513386999996, 2024-04-19T01:05:22.886 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.049063250000000003, 2024-04-19T01:05:22.886 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:22.887 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:22.887 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:22.887 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:22.887 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:22.887 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:22.887 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 354 2024-04-19T01:05:22.887 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.887 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:22.887 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.887 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:22.887 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544030+0000", 2024-04-19T01:05:22.887 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.887 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.888 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.888 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:22.888 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544030+0000", 2024-04-19T01:05:22.888 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.888 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.888 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.888 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:22.888 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544033+0000", 2024-04-19T01:05:22.888 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.506e-06 2024-04-19T01:05:22.888 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.888 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.888 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:22.888 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544046+0000", 2024-04-19T01:05:22.888 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3208e-05 2024-04-19T01:05:22.888 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.889 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.889 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:22.889 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544082+0000", 2024-04-19T01:05:22.889 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5509999999999997e-05 2024-04-19T01:05:22.889 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.889 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.889 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:22.889 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544101+0000", 2024-04-19T01:05:22.889 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9500999999999998e-05 2024-04-19T01:05:22.889 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.889 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.889 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.889 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544230+0000", 2024-04-19T01:05:22.889 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000129114 2024-04-19T01:05:22.890 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.890 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.892 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:22.892 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.544428+0000", 2024-04-19T01:05:22.892 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00019776799999999999 2024-04-19T01:05:22.892 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.892 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.893 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:22.893 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.556395+0000", 2024-04-19T01:05:22.893 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011967261 2024-04-19T01:05:22.893 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.893 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.893 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:22.893 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.574011+0000", 2024-04-19T01:05:22.893 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017615654000000001 2024-04-19T01:05:22.893 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.893 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.893 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:22.894 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.592920+0000", 2024-04-19T01:05:22.894 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018908812000000001 2024-04-19T01:05:22.894 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.894 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.894 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:22.894 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:18.593093+0000", 2024-04-19T01:05:22.894 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00017291600000000001 2024-04-19T01:05:22.894 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.894 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:22.894 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.894 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.894 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.894 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:479 3.as0 3:5a488dbb:::smithi15528762-13:head [append 0~475136] snapc 12=[12,11,10] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-19T01:05:22.894 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:35.633850+0000", 2024-04-19T01:05:22.895 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 47.231693129, 2024-04-19T01:05:22.895 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.042497337000000003, 2024-04-19T01:05:22.895 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:22.895 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:22.895 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:22.895 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:22.895 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:22.895 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:22.895 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 479 2024-04-19T01:05:22.895 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.895 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:22.895 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.895 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:22.895 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.633850+0000", 2024-04-19T01:05:22.895 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.896 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.896 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.896 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:22.896 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.633850+0000", 2024-04-19T01:05:22.896 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.896 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.896 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.896 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:22.896 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.633852+0000", 2024-04-19T01:05:22.896 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.26e-06 2024-04-19T01:05:22.896 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.896 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.896 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:22.896 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.634421+0000", 2024-04-19T01:05:22.896 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000569058 2024-04-19T01:05:22.897 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.897 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.897 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:22.897 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.634441+0000", 2024-04-19T01:05:22.897 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0169e-05 2024-04-19T01:05:22.897 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.897 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.897 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:22.897 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.634452+0000", 2024-04-19T01:05:22.897 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0963e-05 2024-04-19T01:05:22.897 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.897 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.897 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.898 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.634540+0000", 2024-04-19T01:05:22.898 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.7563000000000002e-05 2024-04-19T01:05:22.898 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.898 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.898 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:05:22.898 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.634550+0000", 2024-04-19T01:05:22.898 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0491e-05 2024-04-19T01:05:22.898 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.898 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.898 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.898 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.653619+0000", 2024-04-19T01:05:22.898 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.019068284000000001 2024-04-19T01:05:22.898 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.898 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.899 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:22.899 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.653716+0000", 2024-04-19T01:05:22.899 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7546000000000004e-05 2024-04-19T01:05:22.899 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.899 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.899 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:22.899 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.654525+0000", 2024-04-19T01:05:22.899 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00080861300000000002 2024-04-19T01:05:22.899 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.899 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.899 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:22.899 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.674163+0000", 2024-04-19T01:05:22.899 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.019638666999999999 2024-04-19T01:05:22.899 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.900 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.900 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:22.900 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.676228+0000", 2024-04-19T01:05:22.900 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0020650379999999999 2024-04-19T01:05:22.900 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.900 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.900 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:22.900 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.676347+0000", 2024-04-19T01:05:22.900 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011868500000000001 2024-04-19T01:05:22.900 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.900 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:22.900 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.900 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.901 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.901 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:482 3.as0 3:5a488dbb:::smithi15528762-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-04-19T01:05:22.901 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:35.639943+0000", 2024-04-19T01:05:22.901 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 47.225600075999999, 2024-04-19T01:05:22.901 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.040997591999999999, 2024-04-19T01:05:22.901 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:22.901 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:22.901 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:05:22.901 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:22.901 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:22.901 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:22.901 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 482 2024-04-19T01:05:22.901 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.901 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:22.902 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.902 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:22.906 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639943+0000", 2024-04-19T01:05:22.906 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.906 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.906 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.906 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:22.906 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639943+0000", 2024-04-19T01:05:22.906 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.906 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.906 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.906 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:22.906 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639943+0000", 2024-04-19T01:05:22.907 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.89e-07 2024-04-19T01:05:22.907 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.907 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.907 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:22.907 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639945+0000", 2024-04-19T01:05:22.907 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9870000000000002e-06 2024-04-19T01:05:22.907 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.907 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.907 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:22.907 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639951+0000", 2024-04-19T01:05:22.907 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.9479999999999996e-06 2024-04-19T01:05:22.907 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.907 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.907 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:22.907 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.639956+0000", 2024-04-19T01:05:22.908 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8040000000000001e-06 2024-04-19T01:05:22.908 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.908 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.908 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.908 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.676400+0000", 2024-04-19T01:05:22.908 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.036444184999999997 2024-04-19T01:05:22.908 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.908 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.908 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:05:22.908 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.676473+0000", 2024-04-19T01:05:22.909 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2704000000000001e-05 2024-04-19T01:05:22.909 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.909 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.909 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.909 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.678615+0000", 2024-04-19T01:05:22.909 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0021422030000000001 2024-04-19T01:05:22.909 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.909 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.909 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:22.909 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.678811+0000", 2024-04-19T01:05:22.909 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000196026 2024-04-19T01:05:22.909 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.909 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.910 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:22.910 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:35.680940+0000", 2024-04-19T01:05:22.910 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0021293459999999998 2024-04-19T01:05:22.910 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.910 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:22.910 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.910 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.910 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.910 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:520 3.as0 3:58da707e:::smithi15528762-12:head [copy-from ver 38] snapc 16=[16,15,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-04-19T01:05:22.910 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:41.512306+0000", 2024-04-19T01:05:22.910 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 41.353237153000002, 2024-04-19T01:05:22.910 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.089648318000000005, 2024-04-19T01:05:22.910 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:22.910 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:22.911 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:22.911 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:22.911 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:22.911 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:22.911 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 520 2024-04-19T01:05:22.911 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.911 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:22.911 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.911 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:22.911 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512306+0000", 2024-04-19T01:05:22.911 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.911 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.911 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.911 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:22.912 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512306+0000", 2024-04-19T01:05:22.912 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.912 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.912 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.912 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:22.912 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512309+0000", 2024-04-19T01:05:22.912 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2509999999999999e-06 2024-04-19T01:05:22.912 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.912 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.913 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:22.913 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512326+0000", 2024-04-19T01:05:22.913 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6881999999999999e-05 2024-04-19T01:05:22.913 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.913 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.913 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:22.913 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512356+0000", 2024-04-19T01:05:22.913 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9920000000000002e-05 2024-04-19T01:05:22.913 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.913 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.913 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:22.914 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512371+0000", 2024-04-19T01:05:22.914 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5325999999999999e-05 2024-04-19T01:05:22.914 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.914 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.914 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.914 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512489+0000", 2024-04-19T01:05:22.914 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011835100000000001 2024-04-19T01:05:22.914 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.914 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.914 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op must wait for map", 2024-04-19T01:05:22.914 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512500+0000", 2024-04-19T01:05:22.915 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0437e-05 2024-04-19T01:05:22.915 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.915 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.915 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.915 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.514800+0000", 2024-04-19T01:05:22.915 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0023000719999999998 2024-04-19T01:05:22.915 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.915 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.915 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:22.915 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.514923+0000", 2024-04-19T01:05:22.915 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012253599999999999 2024-04-19T01:05:22.915 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.915 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.915 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:22.915 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.541802+0000", 2024-04-19T01:05:22.916 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.026879473000000001 2024-04-19T01:05:22.916 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.916 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.916 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:22.916 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.564261+0000", 2024-04-19T01:05:22.916 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.022458887 2024-04-19T01:05:22.916 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.916 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.916 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:22.916 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.601886+0000", 2024-04-19T01:05:22.916 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.037625204000000002 2024-04-19T01:05:22.916 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.916 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.916 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:22.917 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.601954+0000", 2024-04-19T01:05:22.917 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.7978999999999994e-05 2024-04-19T01:05:22.917 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.917 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:22.917 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.917 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.917 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.917 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:521 3.as0 3:58da707e:::smithi15528762-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e71)", 2024-04-19T01:05:22.917 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:04:41.512407+0000", 2024-04-19T01:05:22.917 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 41.353135731000002, 2024-04-19T01:05:22.917 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.051806598000000002, 2024-04-19T01:05:22.917 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:22.917 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:22.917 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:05:22.917 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:22.918 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:22.918 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:22.918 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 521 2024-04-19T01:05:22.918 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.918 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:22.918 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.918 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:22.918 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512407+0000", 2024-04-19T01:05:22.918 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.918 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.918 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.918 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:22.918 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512407+0000", 2024-04-19T01:05:22.918 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.918 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.919 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.919 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:22.919 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512408+0000", 2024-04-19T01:05:22.919 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.4799999999999997e-07 2024-04-19T01:05:22.919 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.919 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.919 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:22.919 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512411+0000", 2024-04-19T01:05:22.919 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5109999999999999e-06 2024-04-19T01:05:22.919 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.919 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.919 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:22.919 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512423+0000", 2024-04-19T01:05:22.920 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1188e-05 2024-04-19T01:05:22.920 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.920 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.920 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:22.920 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512430+0000", 2024-04-19T01:05:22.920 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.3540000000000002e-06 2024-04-19T01:05:22.920 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.920 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.920 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.920 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512598+0000", 2024-04-19T01:05:22.920 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00016804 2024-04-19T01:05:22.920 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.920 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.920 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:05:22.920 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.512605+0000", 2024-04-19T01:05:22.921 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.9290000000000001e-06 2024-04-19T01:05:22.921 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.921 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.921 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.921 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.515130+0000", 2024-04-19T01:05:22.921 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0025246299999999999 2024-04-19T01:05:22.921 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.921 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.921 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for blocked object", 2024-04-19T01:05:22.921 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.515181+0000", 2024-04-19T01:05:22.921 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.1100000000000002e-05 2024-04-19T01:05:22.921 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.921 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.921 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.922 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.563980+0000", 2024-04-19T01:05:22.922 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.048799373 2024-04-19T01:05:22.922 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.922 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.922 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:22.922 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.564087+0000", 2024-04-19T01:05:22.922 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000106827 2024-04-19T01:05:22.922 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.922 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.922 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:22.922 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:04:41.564214+0000", 2024-04-19T01:05:22.922 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012689799999999999 2024-04-19T01:05:22.922 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.923 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:22.923 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.923 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.923 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.923 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:723 3.as0 3:5a838d3b:::smithi15528762-46:head [append 0~573440] snapc 23=[23,22,21,1f,1c] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-19T01:05:22.923 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:05:07.585560+0000", 2024-04-19T01:05:22.923 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 15.279983173, 2024-04-19T01:05:22.923 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.052720560999999999, 2024-04-19T01:05:22.923 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:22.923 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:22.923 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:22.923 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:22.923 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:22.923 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:22.924 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 723 2024-04-19T01:05:22.924 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.924 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:22.924 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.924 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:22.924 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.585560+0000", 2024-04-19T01:05:22.924 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.924 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.924 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.924 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:22.924 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.585560+0000", 2024-04-19T01:05:22.924 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.924 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.924 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.924 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:22.925 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.585563+0000", 2024-04-19T01:05:22.925 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.083e-06 2024-04-19T01:05:22.925 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.925 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.925 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:22.925 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.586574+0000", 2024-04-19T01:05:22.925 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001011571 2024-04-19T01:05:22.925 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.925 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.925 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:22.925 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.586618+0000", 2024-04-19T01:05:22.925 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3565e-05 2024-04-19T01:05:22.925 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.925 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.926 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:22.926 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.586642+0000", 2024-04-19T01:05:22.926 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3626e-05 2024-04-19T01:05:22.926 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.926 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.926 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.926 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.593745+0000", 2024-04-19T01:05:22.926 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0071032020000000003 2024-04-19T01:05:22.926 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.926 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.926 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:22.926 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.594062+0000", 2024-04-19T01:05:22.926 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00031753199999999998 2024-04-19T01:05:22.926 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.927 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.927 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:22.927 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.596201+0000", 2024-04-19T01:05:22.927 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0021385660000000002 2024-04-19T01:05:22.927 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.927 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.927 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:22.927 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.609756+0000", 2024-04-19T01:05:22.927 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013555568 2024-04-19T01:05:22.927 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.927 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.927 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:22.927 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.638133+0000", 2024-04-19T01:05:22.927 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.028376714000000001 2024-04-19T01:05:22.928 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.928 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.928 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:22.928 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.638280+0000", 2024-04-19T01:05:22.928 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014713399999999999 2024-04-19T01:05:22.928 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.928 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:22.928 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.928 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.928 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.928 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:724 3.as0 3:5a838d3b:::smithi15528762-46:head [append 0~688128] snapc 23=[23,22,21,1f,1c] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-19T01:05:22.928 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:05:07.593788+0000", 2024-04-19T01:05:22.928 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 15.271755337, 2024-04-19T01:05:22.928 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.045244831999999999, 2024-04-19T01:05:22.928 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:22.929 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:22.929 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:22.929 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:22.929 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:22.929 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:22.929 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 724 2024-04-19T01:05:22.929 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.929 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:22.929 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.929 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:22.929 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.593788+0000", 2024-04-19T01:05:22.929 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.929 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.929 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.929 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:22.930 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.593788+0000", 2024-04-19T01:05:22.930 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.930 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.930 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.930 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:22.930 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.593790+0000", 2024-04-19T01:05:22.930 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8830000000000002e-06 2024-04-19T01:05:22.930 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.930 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.930 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:22.930 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.595229+0000", 2024-04-19T01:05:22.930 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014384829999999999 2024-04-19T01:05:22.930 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.930 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.931 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:22.931 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.595288+0000", 2024-04-19T01:05:22.931 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.9395000000000003e-05 2024-04-19T01:05:22.931 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.931 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.931 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:22.931 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.595311+0000", 2024-04-19T01:05:22.931 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3048999999999999e-05 2024-04-19T01:05:22.931 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.931 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.931 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.931 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.609854+0000", 2024-04-19T01:05:22.931 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014542670000000001 2024-04-19T01:05:22.931 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.931 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.932 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:22.932 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.610147+0000", 2024-04-19T01:05:22.932 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00029328699999999999 2024-04-19T01:05:22.932 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.932 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.932 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:22.932 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.612622+0000", 2024-04-19T01:05:22.932 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0024742700000000002 2024-04-19T01:05:22.932 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.932 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.932 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:22.932 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.638305+0000", 2024-04-19T01:05:22.932 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.025683606000000001 2024-04-19T01:05:22.932 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.933 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.933 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:22.933 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.638960+0000", 2024-04-19T01:05:22.933 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00065454499999999997 2024-04-19T01:05:22.933 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.933 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.933 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:22.933 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.639032+0000", 2024-04-19T01:05:22.933 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2644000000000004e-05 2024-04-19T01:05:22.933 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.933 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:22.933 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.933 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.933 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.933 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:725 3.as0 3:5a838d3b:::smithi15528762-46:head [append 0~294912] snapc 23=[23,22,21,1f,1c] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-19T01:05:22.934 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:05:07.597036+0000", 2024-04-19T01:05:22.934 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 15.268507107, 2024-04-19T01:05:22.934 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.042614699999999998, 2024-04-19T01:05:22.934 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:22.934 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:22.934 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:22.934 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:22.934 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:22.934 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:22.934 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 725 2024-04-19T01:05:22.935 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.935 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:22.935 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.935 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:22.938 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.597036+0000", 2024-04-19T01:05:22.939 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.939 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.939 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.939 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:22.939 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.597036+0000", 2024-04-19T01:05:22.939 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.939 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.939 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.939 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:22.939 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.597039+0000", 2024-04-19T01:05:22.939 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7070000000000001e-06 2024-04-19T01:05:22.939 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.939 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.939 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:22.940 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.597640+0000", 2024-04-19T01:05:22.940 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00060105900000000003 2024-04-19T01:05:22.940 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.940 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.940 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:22.940 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.597686+0000", 2024-04-19T01:05:22.940 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5961999999999998e-05 2024-04-19T01:05:22.940 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.940 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.940 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:22.940 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.597708+0000", 2024-04-19T01:05:22.940 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.2106000000000001e-05 2024-04-19T01:05:22.940 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.940 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.940 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.941 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.626571+0000", 2024-04-19T01:05:22.941 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.028863089000000001 2024-04-19T01:05:22.941 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.941 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.941 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:22.941 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.626891+0000", 2024-04-19T01:05:22.941 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000320549 2024-04-19T01:05:22.941 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.941 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.941 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:22.941 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.628735+0000", 2024-04-19T01:05:22.941 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0018436419999999999 2024-04-19T01:05:22.941 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.941 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.942 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:22.942 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.638642+0000", 2024-04-19T01:05:22.942 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0099071780000000009 2024-04-19T01:05:22.942 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.942 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.942 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:22.942 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.639528+0000", 2024-04-19T01:05:22.942 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00088558599999999995 2024-04-19T01:05:22.942 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.942 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.942 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:22.942 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.639651+0000", 2024-04-19T01:05:22.942 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000122822 2024-04-19T01:05:22.942 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.942 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:22.943 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.943 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.943 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.943 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:726 3.as0 3:5a838d3b:::smithi15528762-46:head [setxattr _header (57) in=7b] snapc 23=[23,22,21,1f,1c] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-19T01:05:22.943 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:05:07.597753+0000", 2024-04-19T01:05:22.943 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 15.267789988000001, 2024-04-19T01:05:22.943 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.046324369999999997, 2024-04-19T01:05:22.943 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:22.943 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:22.943 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:22.943 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:22.943 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:22.943 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:22.943 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 726 2024-04-19T01:05:22.943 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.944 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:22.944 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.944 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:22.944 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.597753+0000", 2024-04-19T01:05:22.944 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.944 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.944 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.945 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:22.945 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.597753+0000", 2024-04-19T01:05:22.945 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.945 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.945 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.945 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:22.945 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.597755+0000", 2024-04-19T01:05:22.945 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.716e-06 2024-04-19T01:05:22.945 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.945 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.945 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:22.946 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.597763+0000", 2024-04-19T01:05:22.946 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.5499999999999995e-06 2024-04-19T01:05:22.946 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.946 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.946 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:22.946 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.597783+0000", 2024-04-19T01:05:22.946 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9928999999999998e-05 2024-04-19T01:05:22.946 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.946 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.946 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:22.946 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.597798+0000", 2024-04-19T01:05:22.947 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5e-05 2024-04-19T01:05:22.947 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.947 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.947 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.950 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.640516+0000", 2024-04-19T01:05:22.951 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.042717828999999999 2024-04-19T01:05:22.951 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.951 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.951 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:22.951 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.640677+0000", 2024-04-19T01:05:22.951 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00016122900000000001 2024-04-19T01:05:22.951 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.951 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.951 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:22.951 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.641516+0000", 2024-04-19T01:05:22.951 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00083878200000000005 2024-04-19T01:05:22.951 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.951 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.951 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:22.952 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.643109+0000", 2024-04-19T01:05:22.952 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0015934149999999999 2024-04-19T01:05:22.952 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.952 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.952 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:22.952 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.643884+0000", 2024-04-19T01:05:22.952 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000774699 2024-04-19T01:05:22.952 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.952 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.952 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:22.952 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.644077+0000", 2024-04-19T01:05:22.952 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00019322100000000001 2024-04-19T01:05:22.952 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.952 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:22.952 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.953 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.953 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.953 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:727 3.as0 3:5a838d3b:::smithi15528762-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e97)", 2024-04-19T01:05:22.953 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:05:07.597819+0000", 2024-04-19T01:05:22.953 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 15.267724394, 2024-04-19T01:05:22.953 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.049201584, 2024-04-19T01:05:22.953 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:22.953 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:22.953 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:05:22.953 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:22.953 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:22.953 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:22.953 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 727 2024-04-19T01:05:22.953 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.953 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:22.954 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.954 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:22.954 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.597819+0000", 2024-04-19T01:05:22.954 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.954 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.954 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.954 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:22.954 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.597819+0000", 2024-04-19T01:05:22.955 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.955 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.955 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.955 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:22.955 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.597820+0000", 2024-04-19T01:05:22.955 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.364e-06 2024-04-19T01:05:22.955 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.955 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.955 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:22.955 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.597826+0000", 2024-04-19T01:05:22.955 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8459999999999996e-06 2024-04-19T01:05:22.956 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.956 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.956 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:22.956 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.597845+0000", 2024-04-19T01:05:22.956 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9239e-05 2024-04-19T01:05:22.956 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.956 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.956 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:22.956 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.597860+0000", 2024-04-19T01:05:22.956 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4538999999999999e-05 2024-04-19T01:05:22.956 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.956 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.956 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.956 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.641941+0000", 2024-04-19T01:05:22.956 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.044081126999999998 2024-04-19T01:05:22.957 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.957 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.957 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:05:22.957 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.642079+0000", 2024-04-19T01:05:22.957 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000138181 2024-04-19T01:05:22.957 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.957 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.957 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.957 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.644918+0000", 2024-04-19T01:05:22.958 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0028394900000000001 2024-04-19T01:05:22.958 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.958 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.958 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:22.958 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.644990+0000", 2024-04-19T01:05:22.958 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.1418e-05 2024-04-19T01:05:22.958 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.958 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.958 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:22.958 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:07.647020+0000", 2024-04-19T01:05:22.958 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00203038 2024-04-19T01:05:22.958 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.958 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:22.959 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.959 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.959 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.959 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:758 3.ds0 3:b9317ffa:::smithi15528762-1:head [append 0~606208] snapc 25=[25,24,22,21,1f] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-04-19T01:05:22.959 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:05:12.691197+0000", 2024-04-19T01:05:22.959 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 10.174346396000001, 2024-04-19T01:05:22.959 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.050627184999999998, 2024-04-19T01:05:22.959 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:22.959 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:22.959 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:22.959 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:22.959 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:22.959 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:22.959 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 758 2024-04-19T01:05:22.960 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.960 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:22.960 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.960 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:22.960 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.691197+0000", 2024-04-19T01:05:22.960 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.960 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.960 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.960 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:22.960 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.691197+0000", 2024-04-19T01:05:22.960 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.960 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.960 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.960 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:22.960 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.691199+0000", 2024-04-19T01:05:22.961 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.514e-06 2024-04-19T01:05:22.961 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.961 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.961 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:22.961 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.692032+0000", 2024-04-19T01:05:22.961 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00083304899999999999 2024-04-19T01:05:22.961 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.961 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.961 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:22.961 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.692054+0000", 2024-04-19T01:05:22.961 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.2402999999999999e-05 2024-04-19T01:05:22.961 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.961 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.961 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:22.961 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.692067+0000", 2024-04-19T01:05:22.962 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2865e-05 2024-04-19T01:05:22.962 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.962 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.962 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.962 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.692132+0000", 2024-04-19T01:05:22.962 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4171000000000006e-05 2024-04-19T01:05:22.962 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.962 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.962 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:22.962 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.692246+0000", 2024-04-19T01:05:22.962 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000114678 2024-04-19T01:05:22.962 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.962 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.962 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:22.963 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.693065+0000", 2024-04-19T01:05:22.963 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00081922799999999997 2024-04-19T01:05:22.963 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.963 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.963 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:22.963 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.714870+0000", 2024-04-19T01:05:22.963 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021805033000000001 2024-04-19T01:05:22.963 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.963 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.963 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:22.963 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.741744+0000", 2024-04-19T01:05:22.963 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.026873589 2024-04-19T01:05:22.963 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.963 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.964 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:22.964 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.741824+0000", 2024-04-19T01:05:22.964 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.9654999999999995e-05 2024-04-19T01:05:22.964 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.964 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:22.964 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.964 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.964 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.964 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:759 3.ds0 3:b9317ffa:::smithi15528762-1:head [append 0~507904] snapc 25=[25,24,22,21,1f] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-04-19T01:05:22.964 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:05:12.694108+0000", 2024-04-19T01:05:22.964 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 10.171434489999999, 2024-04-19T01:05:22.964 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.048052103999999998, 2024-04-19T01:05:22.964 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:22.964 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:22.964 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:22.965 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:22.965 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:22.965 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:22.965 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 759 2024-04-19T01:05:22.965 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.965 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:22.965 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.966 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:22.966 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.694108+0000", 2024-04-19T01:05:22.966 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.966 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.966 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.966 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:22.966 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.694108+0000", 2024-04-19T01:05:22.966 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.966 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.966 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.966 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:22.966 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.694112+0000", 2024-04-19T01:05:22.966 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2040000000000002e-06 2024-04-19T01:05:22.966 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.967 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.967 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:22.967 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.694934+0000", 2024-04-19T01:05:22.967 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00082255999999999996 2024-04-19T01:05:22.967 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.968 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.968 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:22.968 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.694968+0000", 2024-04-19T01:05:22.969 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.3569000000000003e-05 2024-04-19T01:05:22.969 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.969 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.969 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:22.969 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.694991+0000", 2024-04-19T01:05:22.969 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3580000000000001e-05 2024-04-19T01:05:22.969 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.970 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.970 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.970 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.697851+0000", 2024-04-19T01:05:22.970 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.002859735 2024-04-19T01:05:22.970 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.970 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.970 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:22.970 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.698100+0000", 2024-04-19T01:05:22.970 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00024859300000000002 2024-04-19T01:05:22.970 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.970 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.970 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:22.970 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.700128+0000", 2024-04-19T01:05:22.970 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0020280570000000002 2024-04-19T01:05:22.971 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.971 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.971 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:22.971 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.719878+0000", 2024-04-19T01:05:22.971 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.019750569999999999 2024-04-19T01:05:22.971 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.971 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.971 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:22.971 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.742071+0000", 2024-04-19T01:05:22.971 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.022192867000000002 2024-04-19T01:05:22.971 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.971 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.971 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:22.971 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.742161+0000", 2024-04-19T01:05:22.972 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9369000000000004e-05 2024-04-19T01:05:22.972 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.972 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:22.972 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.972 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.972 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.972 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:760 3.ds0 3:b9317ffa:::smithi15528762-1:head [append 0~573440] snapc 25=[25,24,22,21,1f] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-04-19T01:05:22.972 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:05:12.697334+0000", 2024-04-19T01:05:22.972 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 10.168208765999999, 2024-04-19T01:05:22.972 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.044974553, 2024-04-19T01:05:22.972 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:22.972 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:22.972 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:22.972 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:22.973 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:22.974 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:22.974 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 760 2024-04-19T01:05:22.975 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.975 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:22.975 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.975 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:22.975 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.697334+0000", 2024-04-19T01:05:22.975 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.975 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.975 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.975 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:22.975 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.697334+0000", 2024-04-19T01:05:22.976 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.976 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.976 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.976 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:22.976 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.697337+0000", 2024-04-19T01:05:22.976 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0730000000000001e-06 2024-04-19T01:05:22.977 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.977 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.977 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:22.977 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.698339+0000", 2024-04-19T01:05:22.977 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010022589999999999 2024-04-19T01:05:22.977 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.977 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.977 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:22.977 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.698375+0000", 2024-04-19T01:05:22.977 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5509000000000002e-05 2024-04-19T01:05:22.977 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.977 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.977 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:22.977 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.698398+0000", 2024-04-19T01:05:22.977 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3008999999999999e-05 2024-04-19T01:05:22.978 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.978 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.978 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.978 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.710078+0000", 2024-04-19T01:05:22.978 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011680295 2024-04-19T01:05:22.979 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.979 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.979 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:22.979 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.710153+0000", 2024-04-19T01:05:22.979 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5134000000000003e-05 2024-04-19T01:05:22.979 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.979 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.980 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:22.980 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.710841+0000", 2024-04-19T01:05:22.980 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00068707399999999997 2024-04-19T01:05:22.980 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.980 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.980 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:22.980 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.719981+0000", 2024-04-19T01:05:22.980 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0091403979999999992 2024-04-19T01:05:22.980 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.980 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.980 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:22.980 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.742242+0000", 2024-04-19T01:05:22.980 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.022261112999999999 2024-04-19T01:05:22.980 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.981 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.981 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:22.981 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.742309+0000", 2024-04-19T01:05:22.981 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.6688999999999998e-05 2024-04-19T01:05:22.981 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.981 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:22.981 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.981 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.981 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.981 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:761 3.ds0 3:b9317ffa:::smithi15528762-1:head [append 0~475136] snapc 25=[25,24,22,21,1f] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-04-19T01:05:22.981 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:05:12.700539+0000", 2024-04-19T01:05:22.981 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 10.165004055000001, 2024-04-19T01:05:22.981 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.042012834999999998, 2024-04-19T01:05:22.981 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:22.982 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:22.982 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:22.982 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:22.982 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:22.982 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:22.982 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 761 2024-04-19T01:05:22.982 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.982 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:22.982 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.982 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:22.982 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.700539+0000", 2024-04-19T01:05:22.982 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.982 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.982 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.983 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:22.983 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.700539+0000", 2024-04-19T01:05:22.983 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.983 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.983 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.983 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:22.983 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.700542+0000", 2024-04-19T01:05:22.983 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0230000000000001e-06 2024-04-19T01:05:22.983 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.983 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.983 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:22.983 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.701318+0000", 2024-04-19T01:05:22.983 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00077593599999999997 2024-04-19T01:05:22.983 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.983 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.984 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:22.984 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.701350+0000", 2024-04-19T01:05:22.984 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.1891999999999999e-05 2024-04-19T01:05:22.984 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.984 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.984 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:22.984 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.701365+0000", 2024-04-19T01:05:22.984 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.509e-05 2024-04-19T01:05:22.984 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.984 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.984 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.984 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.714895+0000", 2024-04-19T01:05:22.984 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013530388000000001 2024-04-19T01:05:22.984 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.984 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.985 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:22.985 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.715002+0000", 2024-04-19T01:05:22.985 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000106512 2024-04-19T01:05:22.985 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.985 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.985 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:22.985 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.715781+0000", 2024-04-19T01:05:22.985 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00077951400000000003 2024-04-19T01:05:22.985 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.985 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.985 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:22.985 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.719983+0000", 2024-04-19T01:05:22.985 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0042018849999999998 2024-04-19T01:05:22.985 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.986 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.986 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:22.986 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.742490+0000", 2024-04-19T01:05:22.986 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.022506894 2024-04-19T01:05:22.986 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.986 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.986 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:22.986 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.742552+0000", 2024-04-19T01:05:22.986 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1700999999999998e-05 2024-04-19T01:05:22.986 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.986 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:22.986 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.986 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.986 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.987 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:762 3.ds0 3:b9317ffa:::smithi15528762-1:head [setxattr _header (56) in=7b] snapc 25=[25,24,22,21,1f] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-04-19T01:05:22.987 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:05:12.701394+0000", 2024-04-19T01:05:22.987 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 10.164149235, 2024-04-19T01:05:22.987 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.041563461000000003, 2024-04-19T01:05:22.987 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:22.987 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:22.987 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:22.987 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:22.987 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:22.987 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:22.987 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 762 2024-04-19T01:05:22.987 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.987 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:22.987 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.987 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:22.988 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.701394+0000", 2024-04-19T01:05:22.988 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.988 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.988 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.988 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:22.988 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.701394+0000", 2024-04-19T01:05:22.988 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.988 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.988 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.988 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:22.988 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.701395+0000", 2024-04-19T01:05:22.988 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6360000000000001e-06 2024-04-19T01:05:22.988 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.988 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.989 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:22.989 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.701402+0000", 2024-04-19T01:05:22.989 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.8770000000000004e-06 2024-04-19T01:05:22.989 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.989 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.990 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:22.990 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.701423+0000", 2024-04-19T01:05:22.990 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0645999999999999e-05 2024-04-19T01:05:22.990 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.990 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.990 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:22.990 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.701440+0000", 2024-04-19T01:05:22.990 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7232999999999999e-05 2024-04-19T01:05:22.990 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.990 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.990 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.991 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.719045+0000", 2024-04-19T01:05:22.991 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017604583 2024-04-19T01:05:22.991 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.991 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.991 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:22.991 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.719113+0000", 2024-04-19T01:05:22.991 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.8189999999999996e-05 2024-04-19T01:05:22.991 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.992 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.992 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:22.992 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.719591+0000", 2024-04-19T01:05:22.992 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00047857800000000001 2024-04-19T01:05:22.992 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.992 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.992 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:22.992 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.720160+0000", 2024-04-19T01:05:22.992 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00056844700000000003 2024-04-19T01:05:22.992 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.992 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.992 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:22.992 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.742884+0000", 2024-04-19T01:05:22.992 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.022724468000000001 2024-04-19T01:05:22.992 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.993 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.993 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:22.993 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.742957+0000", 2024-04-19T01:05:22.993 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2803000000000001e-05 2024-04-19T01:05:22.993 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.993 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:22.993 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.993 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.993 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.993 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:763 3.ds0 3:b9317ffa:::smithi15528762-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e102)", 2024-04-19T01:05:22.993 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:05:12.701462+0000", 2024-04-19T01:05:22.993 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 10.164080431, 2024-04-19T01:05:22.993 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.044055500999999997, 2024-04-19T01:05:22.993 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:22.994 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:22.994 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2024-04-19T01:05:22.994 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:22.994 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:22.994 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:22.994 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 763 2024-04-19T01:05:22.994 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.994 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:22.994 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.994 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:22.994 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.701462+0000", 2024-04-19T01:05:22.994 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.994 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.994 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.995 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:22.995 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.701462+0000", 2024-04-19T01:05:22.995 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:22.995 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.995 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.995 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:22.995 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.701464+0000", 2024-04-19T01:05:22.995 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1030000000000001e-06 2024-04-19T01:05:22.995 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.995 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.995 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:22.995 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.701472+0000", 2024-04-19T01:05:22.995 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1040000000000007e-06 2024-04-19T01:05:22.995 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.995 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.996 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:22.996 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.701493+0000", 2024-04-19T01:05:22.996 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1639000000000001e-05 2024-04-19T01:05:22.996 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.996 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.996 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:22.996 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.701511+0000", 2024-04-19T01:05:22.996 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.721e-05 2024-04-19T01:05:22.996 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.996 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.996 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.996 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.720004+0000", 2024-04-19T01:05:22.996 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018493543000000001 2024-04-19T01:05:22.996 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.996 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.997 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2024-04-19T01:05:22.997 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.720055+0000", 2024-04-19T01:05:22.997 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0658999999999997e-05 2024-04-19T01:05:22.997 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.997 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.997 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:22.997 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.743422+0000", 2024-04-19T01:05:22.997 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.023366772000000001 2024-04-19T01:05:22.997 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.997 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.997 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:22.997 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.743610+0000", 2024-04-19T01:05:22.997 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00018861400000000001 2024-04-19T01:05:22.997 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.997 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.998 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:22.998 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:12.745518+0000", 2024-04-19T01:05:22.998 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001907857 2024-04-19T01:05:22.998 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.998 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:22.998 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:22.998 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.998 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:22.998 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4604.0:804 3.as0 3:56c32ee7:::smithi15528762-47:head [append 0~671744] snapc 27=[27,25,24,21] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-19T01:05:22.998 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2024-04-19T01:05:17.551636+0000", 2024-04-19T01:05:22.998 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.3139067019999997, 2024-04-19T01:05:22.998 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.043287586000000003, 2024-04-19T01:05:22.998 INFO:teuthology.orchestra.run.smithi033.stdout: "continuous": false, 2024-04-19T01:05:22.999 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2024-04-19T01:05:22.999 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:22.999 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2024-04-19T01:05:22.999 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4604", 2024-04-19T01:05:22.999 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:22.999 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 804 2024-04-19T01:05:22.999 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:22.999 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2024-04-19T01:05:23.000 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:23.000 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2024-04-19T01:05:23.000 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:17.551636+0000", 2024-04-19T01:05:23.000 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:23.000 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:23.000 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:23.000 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2024-04-19T01:05:23.000 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:17.551636+0000", 2024-04-19T01:05:23.000 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2024-04-19T01:05:23.000 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:23.000 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:23.000 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2024-04-19T01:05:23.000 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:17.551638+0000", 2024-04-19T01:05:23.000 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0549999999999998e-06 2024-04-19T01:05:23.000 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:23.001 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:23.001 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2024-04-19T01:05:23.001 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:17.552657+0000", 2024-04-19T01:05:23.001 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001019065 2024-04-19T01:05:23.001 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:23.001 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:23.001 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2024-04-19T01:05:23.001 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:17.552688+0000", 2024-04-19T01:05:23.001 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0703000000000001e-05 2024-04-19T01:05:23.001 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:23.001 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:23.001 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2024-04-19T01:05:23.001 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:17.552701+0000", 2024-04-19T01:05:23.001 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2792000000000001e-05 2024-04-19T01:05:23.001 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:23.002 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:23.002 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:23.002 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:17.552785+0000", 2024-04-19T01:05:23.002 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.4419000000000006e-05 2024-04-19T01:05:23.002 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:23.002 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:23.002 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op must wait for map", 2024-04-19T01:05:23.002 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:17.552797+0000", 2024-04-19T01:05:23.002 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1694e-05 2024-04-19T01:05:23.002 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:23.002 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:23.002 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2024-04-19T01:05:23.002 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:17.554872+0000", 2024-04-19T01:05:23.002 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.002074696 2024-04-19T01:05:23.003 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:23.003 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:23.003 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2024-04-19T01:05:23.003 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:17.554997+0000", 2024-04-19T01:05:23.003 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012554499999999999 2024-04-19T01:05:23.003 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:23.003 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:23.003 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2024-04-19T01:05:23.003 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:17.556280+0000", 2024-04-19T01:05:23.003 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0012828290000000001 2024-04-19T01:05:23.011 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:23.011 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:23.011 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2024-04-19T01:05:23.011 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:17.562238+0000", 2024-04-19T01:05:23.011 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0059576610000000004 2024-04-19T01:05:23.012 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:23.012 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:23.012 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2024-04-19T01:05:23.012 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:17.594834+0000", 2024-04-19T01:05:23.012 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.032596013 2024-04-19T01:05:23.012 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-04-19T01:05:23.012 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-04-19T01:05:23.012 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2024-04-19T01:05:23.012 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2024-04-19T01:05:17.594924+0000", 2024-04-19T01:05:23.012 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.0113999999999998e-05 2024-04-19T01:05:23.012 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:23.012 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:23.012 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:23.012 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-04-19T01:05:23.012 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-19T01:05:23.013 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:05:23.013 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-19T01:05:23.014 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:05:23.014+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:05:23.016 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-19T01:05:23.016 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:05:23.017 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-04-19T01:05:23.018 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":144,"stamp":"2024-04-19T01:05:22.524931+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":96125472,"num_objects":62,"num_object_clones":20,"num_object_copies":424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":2,"num_read":527,"num_read_kb":155077,"num_write":965,"num_write_kb":135492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":16465920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1503,"ondisk_log_size":1503,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":15,"num_per_pool_osds":15,"num_per_pool_omap_osds":15,"kb":1415577600,"kb_used":674228,"kb_used_data":184272,"kb_used_omap":23,"kb_used_meta":489704,"kb_avail":1414903372,"statfs":{"total":1449551462400,"available":1448861052928,"internally_reserved":0,"allocated":188694528,"data_stored":226105425,"data_compressed":2577763,"data_compressed_allocated":5316608,"data_compressed_original":11526144,"omap_allocated":23925,"internal_metadata":501457547},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12615680,"num_objects":3,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":15534,"num_write":58,"num_write_kb":17168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.995307"},"pg_stats":[{"pgid":"3.f","version":"106'81","reported_seq":452,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-19T01:05:18.533452+0000","last_change":"2024-04-19T01:05:18.499045+0000","last_active":"2024-04-19T01:05:18.533452+0000","last_peered":"2024-04-19T01:05:18.533452+0000","last_clean":"2024-04-19T01:05:18.533452+0000","last_became_active":"2024-04-19T01:05:09.542698+0000","last_became_peered":"2024-04-19T01:05:09.542698+0000","last_unstale":"2024-04-19T01:05:18.533452+0000","last_undegraded":"2024-04-19T01:05:18.533452+0000","last_fullsized":"2024-04-19T01:05:18.533452+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'81","last_scrub_stamp":"2024-04-19T01:05:18.498676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:18.498676+0000","objects_scrubbed":4,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:25.085878+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.014203057999999999,"stat_sum":{"num_bytes":8601600,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":19,"num_read_kb":5290,"num_write":51,"num_write_kb":8400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,13,5,11,7,1,2],"acting":[10,13,5,11,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"1c","length":"1"},{"start":"1f","length":"1"},{"start":"22","length":"1"},{"start":"26","length":"1"}]},{"pgid":"3.e","version":"109'53","reported_seq":461,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-19T01:05:21.523943+0000","last_change":"2024-04-19T01:05:19.570600+0000","last_active":"2024-04-19T01:05:21.523943+0000","last_peered":"2024-04-19T01:05:21.523943+0000","last_clean":"2024-04-19T01:05:21.523943+0000","last_became_active":"2024-04-19T01:04:03.238386+0000","last_became_peered":"2024-04-19T01:04:03.238386+0000","last_unstale":"2024-04-19T01:05:21.523943+0000","last_undegraded":"2024-04-19T01:05:21.523943+0000","last_fullsized":"2024-04-19T01:05:21.523943+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'45","last_scrub_stamp":"2024-04-19T01:05:05.862139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:05.862139+0000","objects_scrubbed":3,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:15.435599+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.026176436000000001,"stat_sum":{"num_bytes":2277376,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":22,"num_read_kb":8695,"num_write":17,"num_write_kb":5088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11,5,4,2,9,15],"acting":[13,11,5,4,2,9,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"1f","length":"1"},{"start":"21","length":"2"},{"start":"26","length":"1"}]},{"pgid":"3.d","version":"109'98","reported_seq":509,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-19T01:05:19.563431+0000","last_change":"2024-04-19T01:05:19.564433+0000","last_active":"2024-04-19T01:05:19.563431+0000","last_peered":"2024-04-19T01:05:19.563431+0000","last_clean":"2024-04-19T01:05:19.563431+0000","last_became_active":"2024-04-19T01:05:09.548237+0000","last_became_peered":"2024-04-19T01:05:09.548237+0000","last_unstale":"2024-04-19T01:05:19.563431+0000","last_undegraded":"2024-04-19T01:05:19.563431+0000","last_fullsized":"2024-04-19T01:05:19.563431+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:25.511234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.029187052000000002,"stat_sum":{"num_bytes":9125888,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":4329,"num_write":35,"num_write_kb":10000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":4751360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,9,1,8,2,6],"acting":[12,4,9,1,8,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"1c","length":"1"},{"start":"1f","length":"1"},{"start":"21","length":"2"},{"start":"26","length":"1"}]},{"pgid":"3.c","version":"109'80","reported_seq":462,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-19T01:05:19.555984+0000","last_change":"2024-04-19T01:05:19.556620+0000","last_active":"2024-04-19T01:05:19.555984+0000","last_peered":"2024-04-19T01:05:19.555984+0000","last_clean":"2024-04-19T01:05:19.555984+0000","last_became_active":"2024-04-19T01:05:09.550415+0000","last_became_peered":"2024-04-19T01:05:09.550415+0000","last_unstale":"2024-04-19T01:05:19.555984+0000","last_undegraded":"2024-04-19T01:05:19.555984+0000","last_fullsized":"2024-04-19T01:05:19.555984+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:24.890009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.008888623,"stat_sum":{"num_bytes":2883584,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":2454,"num_write":43,"num_write_kb":4144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,12,1,9,2,6],"acting":[11,10,12,1,9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"1c","length":"1"},{"start":"1f","length":"1"},{"start":"21","length":"2"},{"start":"26","length":"1"}]},{"pgid":"3.b","version":"105'56","reported_seq":442,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-19T01:05:18.533334+0000","last_change":"2024-04-19T01:05:16.664380+0000","last_active":"2024-04-19T01:05:18.533334+0000","last_peered":"2024-04-19T01:05:18.533334+0000","last_clean":"2024-04-19T01:05:18.533334+0000","last_became_active":"2024-04-19T01:04:02.781569+0000","last_became_peered":"2024-04-19T01:04:02.781569+0000","last_unstale":"2024-04-19T01:05:18.533334+0000","last_undegraded":"2024-04-19T01:05:18.533334+0000","last_fullsized":"2024-04-19T01:05:18.533334+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:28.241605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022240459999999999,"stat_sum":{"num_bytes":5095424,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":21,"num_read_kb":6897,"num_write":39,"num_write_kb":4976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11,15,1,4,14],"acting":[10,6,11,15,1,4,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"1c","length":"1"},{"start":"1f","length":"1"},{"start":"22","length":"1"},{"start":"26","length":"1"}]},{"pgid":"3.a","version":"109'110","reported_seq":500,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-19T01:05:19.551288+0000","last_change":"2024-04-19T01:05:19.554652+0000","last_active":"2024-04-19T01:05:19.551288+0000","last_peered":"2024-04-19T01:05:19.551288+0000","last_clean":"2024-04-19T01:05:19.551288+0000","last_became_active":"2024-04-19T01:04:03.323177+0000","last_became_peered":"2024-04-19T01:04:03.323177+0000","last_unstale":"2024-04-19T01:05:19.551288+0000","last_undegraded":"2024-04-19T01:05:19.551288+0000","last_fullsized":"2024-04-19T01:05:19.551288+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:20.938662+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.019394524,"stat_sum":{"num_bytes":8126464,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":32,"num_read_kb":5821,"num_write":37,"num_write_kb":15312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,7,3,13,1,8],"acting":[12,4,7,3,13,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"1c","length":"1"},{"start":"1f","length":"1"},{"start":"21","length":"2"},{"start":"26","length":"1"}]},{"pgid":"3.9","version":"109'144","reported_seq":558,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-19T01:05:21.524102+0000","last_change":"2024-04-19T01:05:19.570063+0000","last_active":"2024-04-19T01:05:21.524102+0000","last_peered":"2024-04-19T01:05:21.524102+0000","last_clean":"2024-04-19T01:05:21.524102+0000","last_became_active":"2024-04-19T01:04:03.027372+0000","last_became_peered":"2024-04-19T01:04:03.027372+0000","last_unstale":"2024-04-19T01:05:21.524102+0000","last_undegraded":"2024-04-19T01:05:21.524102+0000","last_fullsized":"2024-04-19T01:05:21.524102+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'132","last_scrub_stamp":"2024-04-19T01:05:07.894862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:07.894862+0000","objects_scrubbed":7,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:12.949429+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.025552390000000001,"stat_sum":{"num_bytes":12009472,"num_objects":7,"num_object_clones":1,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":50,"num_read_kb":23637,"num_write":93,"num_write_kb":15952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11,14,9,4,10,6],"acting":[13,11,14,9,4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"1f","length":"1"},{"start":"21","length":"2"},{"start":"26","length":"1"}]},{"pgid":"3.8","version":"109'65","reported_seq":490,"reported_epoch":109,"state":"active+clean+scrubbing","last_fresh":"2024-04-19T01:05:19.606786+0000","last_change":"2024-04-19T01:05:19.599145+0000","last_active":"2024-04-19T01:05:19.606786+0000","last_peered":"2024-04-19T01:05:19.606786+0000","last_clean":"2024-04-19T01:05:19.606786+0000","last_became_active":"2024-04-19T01:05:09.549364+0000","last_became_peered":"2024-04-19T01:05:09.549364+0000","last_unstale":"2024-04-19T01:05:19.606786+0000","last_undegraded":"2024-04-19T01:05:19.606786+0000","last_fullsized":"2024-04-19T01:05:19.606786+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074258500000000003,"stat_sum":{"num_bytes":3112960,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":1671,"num_write":63,"num_write_kb":5664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":3112960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,1,15,14,6,5],"acting":[11,10,1,15,14,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"1c","length":"1"},{"start":"1f","length":"1"},{"start":"21","length":"2"},{"start":"26","length":"1"}]},{"pgid":"1.0","version":"25'157","reported_seq":384,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-19T01:05:19.547608+0000","last_change":"2024-04-19T01:05:11.601754+0000","last_active":"2024-04-19T01:05:19.547608+0000","last_peered":"2024-04-19T01:05:19.547608+0000","last_clean":"2024-04-19T01:05:19.547608+0000","last_became_active":"2024-04-19T01:03:50.774078+0000","last_became_peered":"2024-04-19T01:03:50.774078+0000","last_unstale":"2024-04-19T01:05:19.547608+0000","last_undegraded":"2024-04-19T01:05:19.547608+0000","last_fullsized":"2024-04-19T01:05:19.547608+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'157","last_scrub_stamp":"2024-04-19T01:05:11.601630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:49.580851+0000","last_clean_scrub_stamp":"2024-04-19T01:05:11.601630+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:12.040402+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":187,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-19T01:05:19.547933+0000","last_change":"2024-04-19T01:05:07.549099+0000","last_active":"2024-04-19T01:05:19.547933+0000","last_peered":"2024-04-19T01:05:19.547933+0000","last_clean":"2024-04-19T01:05:19.547933+0000","last_became_active":"2024-04-19T01:05:06.529945+0000","last_became_peered":"2024-04-19T01:05:06.529945+0000","last_unstale":"2024-04-19T01:05:19.547933+0000","last_undegraded":"2024-04-19T01:05:19.547933+0000","last_fullsized":"2024-04-19T01:05:19.547933+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:07.548635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:07.548635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:16.001957+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"102'48","reported_seq":460,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-19T01:05:19.548862+0000","last_change":"2024-04-19T01:05:19.549663+0000","last_active":"2024-04-19T01:05:19.548862+0000","last_peered":"2024-04-19T01:05:19.548862+0000","last_clean":"2024-04-19T01:05:19.548862+0000","last_became_active":"2024-04-19T01:05:09.553946+0000","last_became_peered":"2024-04-19T01:05:09.553946+0000","last_unstale":"2024-04-19T01:05:19.548862+0000","last_undegraded":"2024-04-19T01:05:19.548862+0000","last_fullsized":"2024-04-19T01:05:19.548862+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:22.926716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001808849,"stat_sum":{"num_bytes":2260992,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":532,"num_write":12,"num_write_kb":2544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":2588672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,13,4,8,3,7],"acting":[11,5,13,4,8,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"1c","length":"1"},{"start":"1f","length":"1"},{"start":"21","length":"2"},{"start":"26","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":174,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-19T01:05:19.543505+0000","last_change":"2024-04-19T01:03:53.760075+0000","last_active":"2024-04-19T01:05:19.543505+0000","last_peered":"2024-04-19T01:05:19.543505+0000","last_clean":"2024-04-19T01:05:19.543505+0000","last_became_active":"2024-04-19T01:03:53.759326+0000","last_became_peered":"2024-04-19T01:03:53.759326+0000","last_unstale":"2024-04-19T01:05:19.543505+0000","last_undegraded":"2024-04-19T01:05:19.543505+0000","last_fullsized":"2024-04-19T01:05:19.543505+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:20.374505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"107'66","reported_seq":437,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-19T01:05:18.546669+0000","last_change":"2024-04-19T01:05:16.559999+0000","last_active":"2024-04-19T01:05:18.546669+0000","last_peered":"2024-04-19T01:05:18.546669+0000","last_clean":"2024-04-19T01:05:18.546669+0000","last_became_active":"2024-04-19T01:04:03.323727+0000","last_became_peered":"2024-04-19T01:04:03.323727+0000","last_unstale":"2024-04-19T01:05:18.546669+0000","last_undegraded":"2024-04-19T01:05:18.546669+0000","last_fullsized":"2024-04-19T01:05:18.546669+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:22.033100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080075259999999992,"stat_sum":{"num_bytes":3588096,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":14,"num_read_kb":2020,"num_write":53,"num_write_kb":4992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,11,9,1,7],"acting":[6,4,3,11,9,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1c","length":"1"},{"start":"1f","length":"1"},{"start":"22","length":"1"},{"start":"26","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":172,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-19T01:05:19.550680+0000","last_change":"2024-04-19T01:03:53.650434+0000","last_active":"2024-04-19T01:05:19.550680+0000","last_peered":"2024-04-19T01:05:19.550680+0000","last_clean":"2024-04-19T01:05:19.550680+0000","last_became_active":"2024-04-19T01:03:53.649606+0000","last_became_peered":"2024-04-19T01:03:53.649606+0000","last_unstale":"2024-04-19T01:05:19.550680+0000","last_undegraded":"2024-04-19T01:05:19.550680+0000","last_fullsized":"2024-04-19T01:05:19.550680+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"105'118","reported_seq":506,"reported_epoch":109,"state":"active+clean+snaptrim","last_fresh":"2024-04-19T01:05:19.544178+0000","last_change":"2024-04-19T01:05:19.544287+0000","last_active":"2024-04-19T01:05:19.544178+0000","last_peered":"2024-04-19T01:05:19.544178+0000","last_clean":"2024-04-19T01:05:19.544178+0000","last_became_active":"2024-04-19T01:04:03.322503+0000","last_became_peered":"2024-04-19T01:04:03.322503+0000","last_unstale":"2024-04-19T01:05:19.544178+0000","last_undegraded":"2024-04-19T01:05:19.544178+0000","last_fullsized":"2024-04-19T01:05:19.544178+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'96","last_scrub_stamp":"2024-04-19T01:05:06.447000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:06.447000+0000","objects_scrubbed":5,"log_size":118,"log_dups_size":0,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:24.065472+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.00160776,"stat_sum":{"num_bytes":9191424,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":27227,"num_write":39,"num_write_kb":10960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4,3,10,11,6],"acting":[1,9,4,3,10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1c","length":"1"},{"start":"1f","length":"1"},{"start":"22","length":"1"},{"start":"26","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":182,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-19T01:05:19.544110+0000","last_change":"2024-04-19T01:05:08.432764+0000","last_active":"2024-04-19T01:05:19.544110+0000","last_peered":"2024-04-19T01:05:19.544110+0000","last_clean":"2024-04-19T01:05:19.544110+0000","last_became_active":"2024-04-19T01:03:54.011460+0000","last_became_peered":"2024-04-19T01:03:54.011460+0000","last_unstale":"2024-04-19T01:05:19.544110+0000","last_undegraded":"2024-04-19T01:05:19.544110+0000","last_fullsized":"2024-04-19T01:05:19.544110+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:08.432519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:08.432519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:10.143460+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"109'103","reported_seq":523,"reported_epoch":109,"state":"active+clean+snaptrim","last_fresh":"2024-04-19T01:05:19.587652+0000","last_change":"2024-04-19T01:05:19.551156+0000","last_active":"2024-04-19T01:05:19.587652+0000","last_peered":"2024-04-19T01:05:19.587652+0000","last_clean":"2024-04-19T01:05:19.587652+0000","last_became_active":"2024-04-19T01:05:09.556011+0000","last_became_peered":"2024-04-19T01:05:09.556011+0000","last_unstale":"2024-04-19T01:05:19.587652+0000","last_undegraded":"2024-04-19T01:05:19.587652+0000","last_fullsized":"2024-04-19T01:05:19.587652+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001133712,"stat_sum":{"num_bytes":7176192,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":16881,"num_write":101,"num_write_kb":9712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":2932736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2,3,1,4,12],"acting":[15,8,2,3,1,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"1c","length":"1"},{"start":"1f","length":"1"},{"start":"22","length":"1"},{"start":"26","length":"1"}]},{"pgid":"3.5","version":"109'212","reported_seq":602,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-19T01:05:19.574816+0000","last_change":"2024-04-19T01:05:19.576519+0000","last_active":"2024-04-19T01:05:19.574816+0000","last_peered":"2024-04-19T01:05:19.574816+0000","last_clean":"2024-04-19T01:05:19.574816+0000","last_became_active":"2024-04-19T01:04:03.237886+0000","last_became_peered":"2024-04-19T01:04:03.237886+0000","last_unstale":"2024-04-19T01:05:19.574816+0000","last_undegraded":"2024-04-19T01:05:19.574816+0000","last_fullsized":"2024-04-19T01:05:19.574816+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.031479316,"stat_sum":{"num_bytes":13647872,"num_objects":9,"num_object_clones":3,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":65,"num_read_kb":18047,"num_write":126,"num_write_kb":22416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,14,11,10,2,5],"acting":[8,6,14,11,10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"1c","length":"1"},{"start":"1f","length":"1"},{"start":"21","length":"2"},{"start":"26","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":179,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-19T01:05:18.546317+0000","last_change":"2024-04-19T01:05:10.593158+0000","last_active":"2024-04-19T01:05:18.546317+0000","last_peered":"2024-04-19T01:05:18.546317+0000","last_clean":"2024-04-19T01:05:18.546317+0000","last_became_active":"2024-04-19T01:03:53.716040+0000","last_became_peered":"2024-04-19T01:03:53.716040+0000","last_unstale":"2024-04-19T01:05:18.546317+0000","last_undegraded":"2024-04-19T01:05:18.546317+0000","last_fullsized":"2024-04-19T01:05:18.546317+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:10.592906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:10.592906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:39.747266+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"107'87","reported_seq":481,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-19T01:05:18.533502+0000","last_change":"2024-04-19T01:05:16.665094+0000","last_active":"2024-04-19T01:05:18.533502+0000","last_peered":"2024-04-19T01:05:18.533502+0000","last_clean":"2024-04-19T01:05:18.533502+0000","last_became_active":"2024-04-19T01:04:03.322815+0000","last_became_peered":"2024-04-19T01:04:03.322815+0000","last_unstale":"2024-04-19T01:05:18.533502+0000","last_undegraded":"2024-04-19T01:05:18.533502+0000","last_fullsized":"2024-04-19T01:05:18.533502+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'81","last_scrub_stamp":"2024-04-19T01:05:10.447179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:10.447179+0000","objects_scrubbed":5,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:24.761632+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0050212269999999996,"stat_sum":{"num_bytes":5357568,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":39,"num_read_kb":13405,"num_write":24,"num_write_kb":8864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,13,14,3,15,1,5],"acting":[10,13,14,3,15,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"1c","length":"1"},{"start":"1f","length":"1"},{"start":"22","length":"1"},{"start":"26","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":183,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-19T01:05:21.106008+0000","last_change":"2024-04-19T01:05:21.106153+0000","last_active":"2024-04-19T01:05:21.106008+0000","last_peered":"2024-04-19T01:05:21.106008+0000","last_clean":"2024-04-19T01:05:21.106008+0000","last_became_active":"2024-04-19T01:03:53.771887+0000","last_became_peered":"2024-04-19T01:03:53.771887+0000","last_unstale":"2024-04-19T01:05:21.106008+0000","last_undegraded":"2024-04-19T01:05:21.106008+0000","last_fullsized":"2024-04-19T01:05:21.106008+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:21.105806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:21.105806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:40.011866+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"109'25","reported_seq":448,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-19T01:05:21.525519+0000","last_change":"2024-04-19T01:05:19.578396+0000","last_active":"2024-04-19T01:05:21.525519+0000","last_peered":"2024-04-19T01:05:21.525519+0000","last_clean":"2024-04-19T01:05:21.525519+0000","last_became_active":"2024-04-19T01:05:09.546523+0000","last_became_peered":"2024-04-19T01:05:09.546523+0000","last_unstale":"2024-04-19T01:05:21.525519+0000","last_undegraded":"2024-04-19T01:05:21.525519+0000","last_fullsized":"2024-04-19T01:05:21.525519+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'23","last_scrub_stamp":"2024-04-19T01:05:18.361237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:18.361237+0000","objects_scrubbed":2,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:23.365260+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.020132710000000002,"stat_sum":{"num_bytes":3080192,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":18053,"num_write":29,"num_write_kb":3008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":3080192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,7,5,11,6,9],"acting":[14,13,7,5,11,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"1f","length":"1"},{"start":"21","length":"2"},{"start":"26","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":185,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-19T01:05:21.524801+0000","last_change":"2024-04-19T01:05:11.353464+0000","last_active":"2024-04-19T01:05:21.524801+0000","last_peered":"2024-04-19T01:05:21.524801+0000","last_clean":"2024-04-19T01:05:21.524801+0000","last_became_active":"2024-04-19T01:03:53.735549+0000","last_became_peered":"2024-04-19T01:03:53.735549+0000","last_unstale":"2024-04-19T01:05:21.524801+0000","last_undegraded":"2024-04-19T01:05:21.524801+0000","last_fullsized":"2024-04-19T01:05:21.524801+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:11.353213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:11.353213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:29.090848+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13],"acting":[14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":404,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-19T01:05:21.525303+0000","last_change":"2024-04-19T01:05:19.560596+0000","last_active":"2024-04-19T01:05:21.525303+0000","last_peered":"2024-04-19T01:05:21.525303+0000","last_clean":"2024-04-19T01:05:21.525303+0000","last_became_active":"2024-04-19T01:05:06.543387+0000","last_became_peered":"2024-04-19T01:05:06.543387+0000","last_unstale":"2024-04-19T01:05:21.525303+0000","last_undegraded":"2024-04-19T01:05:21.525303+0000","last_fullsized":"2024-04-19T01:05:21.525303+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:19.329813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:19.329813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:36.140684+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0020017569999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7,15,8,11,1,10],"acting":[14,7,15,8,11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"1f","length":"1"},{"start":"21","length":"2"},{"start":"26","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":174,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-19T01:05:19.543462+0000","last_change":"2024-04-19T01:03:53.939441+0000","last_active":"2024-04-19T01:05:19.543462+0000","last_peered":"2024-04-19T01:05:19.543462+0000","last_clean":"2024-04-19T01:05:19.543462+0000","last_became_active":"2024-04-19T01:03:53.938774+0000","last_became_peered":"2024-04-19T01:03:53.938774+0000","last_unstale":"2024-04-19T01:05:19.543462+0000","last_undegraded":"2024-04-19T01:05:19.543462+0000","last_fullsized":"2024-04-19T01:05:19.543462+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:20.374551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":95535104,"num_objects":60,"num_object_clones":20,"num_object_copies":420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":2,"num_read":391,"num_read_kb":154959,"num_write":762,"num_write_kb":132032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":16465920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":178888704,"data_stored":219090944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1346,"ondisk_log_size":1346,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":13},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":15546,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":14,"up_from":24,"seq":103079215126,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46900,"kb_used_data":11560,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94324940,"statfs":{"total":96636764160,"available":96588738560,"internally_reserved":0,"allocated":11837440,"data_stored":12913175,"data_compressed":171899,"data_compressed_allocated":352256,"data_compressed_original":704512,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,5,6,7,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":13,"up_from":24,"seq":103079215127,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47896,"kb_used_data":12556,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94323944,"statfs":{"total":96636764160,"available":96587718656,"internally_reserved":0,"allocated":12857344,"data_stored":14346775,"data_compressed":171899,"data_compressed_allocated":352256,"data_compressed_original":704512,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":9,"up_from":24,"seq":103079215128,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46880,"kb_used_data":11536,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94324960,"statfs":{"total":96636764160,"available":96588759040,"internally_reserved":0,"allocated":11812864,"data_stored":15254088,"data_compressed":173932,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[1,2,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313238,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46452,"kb_used_data":15268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325388,"statfs":{"total":96636764160,"available":96589197312,"internally_reserved":0,"allocated":15634432,"data_stored":17234301,"data_compressed":167834,"data_compressed_allocated":344064,"data_compressed_original":688128,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280532,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46900,"kb_used_data":15716,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324940,"statfs":{"total":96636764160,"available":96588738560,"internally_reserved":0,"allocated":16093184,"data_stored":19532157,"data_compressed":167834,"data_compressed_allocated":344064,"data_compressed_original":688128,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,3,4,5,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280534,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47608,"kb_used_data":16420,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324232,"statfs":{"total":96636764160,"available":96588013568,"internally_reserved":0,"allocated":16814080,"data_stored":21086638,"data_compressed":169865,"data_compressed_allocated":348160,"data_compressed_original":696320,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,6,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280533,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36956,"kb_used_data":9936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334884,"statfs":{"total":96636764160,"available":96598921216,"internally_reserved":0,"allocated":10174464,"data_stored":12992942,"data_compressed":169865,"data_compressed_allocated":348160,"data_compressed_original":696320,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[2,4,5,6,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280533,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46776,"kb_used_data":15588,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325064,"statfs":{"total":96636764160,"available":96588865536,"internally_reserved":0,"allocated":15962112,"data_stored":20177326,"data_compressed":169865,"data_compressed_allocated":348160,"data_compressed_original":696320,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280533,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41820,"kb_used_data":10632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330020,"statfs":{"total":96636764160,"available":96593940480,"internally_reserved":0,"allocated":10887168,"data_stored":12350414,"data_compressed":177638,"data_compressed_allocated":380928,"data_compressed_original":1208320,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,6,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280534,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42256,"kb_used_data":11068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329584,"statfs":{"total":96636764160,"available":96593494016,"internally_reserved":0,"allocated":11333632,"data_stored":14307758,"data_compressed":169865,"data_compressed_allocated":348160,"data_compressed_original":696320,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,5,6,7,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280532,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67360,"kb_used_data":18132,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94304480,"statfs":{"total":96636764160,"available":96567787520,"internally_reserved":0,"allocated":18567168,"data_stored":22154295,"data_compressed":179672,"data_compressed_allocated":385024,"data_compressed_original":1216512,"omap_allocated":1596,"internal_metadata":50395588},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280532,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34920,"kb_used_data":7900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336920,"statfs":{"total":96636764160,"available":96601006080,"internally_reserved":0,"allocated":8089600,"data_stored":10803735,"data_compressed":171899,"data_compressed_allocated":352256,"data_compressed_original":704512,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7,8,9,10,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280533,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33860,"kb_used_data":6840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337980,"statfs":{"total":96636764160,"available":96602091520,"internally_reserved":0,"allocated":7004160,"data_stored":7430574,"data_compressed":169865,"data_compressed_allocated":348160,"data_compressed_original":696320,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":24,"seq":103079215126,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47828,"kb_used_data":12484,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94324012,"statfs":{"total":96636764160,"available":96587788288,"internally_reserved":0,"allocated":12783616,"data_stored":15397448,"data_compressed":173932,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":0,"seq":103079215127,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":24,"seq":103079215126,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39816,"kb_used_data":8636,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332024,"statfs":{"total":96636764160,"available":96595992576,"internally_reserved":0,"allocated":8843264,"data_stored":10123799,"data_compressed":171899,"data_compressed_allocated":352256,"data_compressed_original":704512,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5787648,"data_stored":6684672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":15564800,"data_stored":19558400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11509760,"data_stored":14434304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":9121792,"data_stored":11661312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":15200256,"data_stored":19136512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":10752000,"data_stored":12378112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15433728,"data_stored":19591168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":7581696,"data_stored":9170944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":10452992,"data_stored":13148160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":10539008,"data_stored":14290944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":14585856,"data_stored":16904192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":17604608,"data_stored":21024768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":6828032,"data_stored":9265152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":11595776,"data_stored":13393920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":10575872,"data_stored":11960320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":5754880,"data_stored":6488064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T01:05:23.020 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2024-04-19T01:05:23.020 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 26 2024-04-19T01:05:23.116 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:05:23.116+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:05:23.153 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:05:23.153 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [], 2024-04-19T01:05:23.153 INFO:teuthology.orchestra.run.smithi110.stdout: "num_ops": 0 2024-04-19T01:05:23.153 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:05:23.167 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-19T01:05:23.217 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-04-19T01:05:23.217 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T01:05:23.217 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:05:23.216+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:05:23.239 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-04-19T01:05:23.239 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T01:05:23.262 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-04-19T01:05:23.265 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T01:05:23.277 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-04-19T01:05:23.278 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T01:05:23.294 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-04-19T01:05:23.295 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T01:05:23.316 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-04-19T01:05:23.316 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T01:05:23.317 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:05:23.316+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:05:23.338 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-04-19T01:05:23.338 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T01:05:23.341 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:23.341 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:23.355 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-19T01:05:23.359 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-04-19T01:05:23.360 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T01:05:23.378 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2024-04-19T01:05:23.378 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T01:05:23.396 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2024-04-19T01:05:23.397 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T01:05:23.419 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2024-04-19T01:05:23.419 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T01:05:23.419 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:05:23.416+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:05:23.440 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2024-04-19T01:05:23.440 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T01:05:23.459 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2024-04-19T01:05:23.459 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-19T01:05:23.481 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:23.481 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:23.482 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2024-04-19T01:05:23.482 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-19T01:05:23.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:05:23.481+0000 7f6fcffff640 0 -- 172.21.15.33:0/881019358 >> [v2:172.21.15.118:6800/506416463,v1:172.21.15.118:6801/506416463] conn(0x7f6fd0220580 msgr2=0x7f6fd0222970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-19T01:05:23.495 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-19T01:05:23.520 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:05:23.518+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:05:23.576 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2024-04-19T01:05:23.576 INFO:teuthology.orchestra.run.smithi033.stderr:set pool 3 pg_num to 26 2024-04-19T01:05:23.583 INFO:tasks.rados.rados.0.smithi155.stdout:359: done (3 left) 2024-04-19T01:05:23.583 INFO:tasks.rados.rados.0.smithi155.stdout:360: setattr oid 38 current snap is 41 2024-04-19T01:05:23.583 INFO:tasks.rados.rados.0.smithi155.stdout:356: expect (ObjNum 130 snap 37 seq_num 130) 2024-04-19T01:05:23.614 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:23.614 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:23.624 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:05:23.621+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:05:23.628 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-19T01:05:23.677 INFO:tasks.rados.rados.0.smithi155.stdout:357: expect (ObjNum 24 snap 0 seq_num 24) 2024-04-19T01:05:23.691 INFO:tasks.rados.rados.0.smithi155.stdout:358: finishing write tid 1 to smithi15528762-10 2024-04-19T01:05:23.691 INFO:tasks.rados.rados.0.smithi155.stdout:358: finishing write tid 2 to smithi15528762-10 2024-04-19T01:05:23.691 INFO:tasks.rados.rados.0.smithi155.stdout:358: finishing write tid 3 to smithi15528762-10 2024-04-19T01:05:23.691 INFO:tasks.rados.rados.0.smithi155.stdout:358: finishing write tid 4 to smithi15528762-10 2024-04-19T01:05:23.691 INFO:tasks.rados.rados.0.smithi155.stdout:358: finishing write tid 5 to smithi15528762-10 2024-04-19T01:05:23.691 INFO:tasks.rados.rados.0.smithi155.stdout:358: finishing write tid 6 to smithi15528762-10 2024-04-19T01:05:23.691 INFO:tasks.rados.rados.0.smithi155.stdout:358: oid 10 updating version 0 to 55 2024-04-19T01:05:23.691 INFO:tasks.rados.rados.0.smithi155.stdout:358: oid 10 updating version 55 to 56 2024-04-19T01:05:23.691 INFO:tasks.rados.rados.0.smithi155.stdout:358: oid 10 updating version 56 to 58 2024-04-19T01:05:23.691 INFO:tasks.rados.rados.0.smithi155.stdout:358: oid 10 version 58 is already newer than 57 2024-04-19T01:05:23.691 INFO:tasks.rados.rados.0.smithi155.stdout:358: oid 10 updating version 58 to 59 2024-04-19T01:05:23.692 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 10 v 59 (ObjNum 140 snap 41 seq_num 140) dirty exists 2024-04-19T01:05:23.692 INFO:tasks.rados.rados.0.smithi155.stdout:358: left oid 10 (ObjNum 140 snap 41 seq_num 140) 2024-04-19T01:05:23.692 INFO:tasks.rados.rados.0.smithi155.stdout:356: done (3 left) 2024-04-19T01:05:23.692 INFO:tasks.rados.rados.0.smithi155.stdout:357: done (2 left) 2024-04-19T01:05:23.692 INFO:tasks.rados.rados.0.smithi155.stdout:358: done (1 left) 2024-04-19T01:05:23.692 INFO:tasks.rados.rados.0.smithi155.stdout:361: delete oid 34 current snap is 41 2024-04-19T01:05:23.703 INFO:tasks.rados.rados.0.smithi155.stdout:361: done (1 left) 2024-04-19T01:05:23.704 INFO:tasks.rados.rados.0.smithi155.stdout:362: snap_create 2024-04-19T01:05:23.704 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 38 v 215 (ObjNum 129 snap 0 seq_num 1052568752) dirty exists 2024-04-19T01:05:23.714 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2024-04-19T01:05:23.715 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-19T01:05:23.723 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:05:23.724+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:05:23.745 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2024-04-19T01:05:23.746 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-19T01:05:23.754 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:23.755 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:23.768 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-19T01:05:23.824 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:05:23.824+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:05:23.900 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:23.900 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:23.915 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-19T01:05:23.956 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:05:23.956+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:05:24.027 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:05:24.026+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:05:24.078 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:24.078 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:24.086 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-19T01:05:24.128 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:05:24.130+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:05:24.214 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:24.214 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:24.234 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-19T01:05:24.235 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:05:24.230+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:05:24.335 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:05:24.332+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:05:24.358 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:24.359 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:24.372 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-04-19T01:05:24.435 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:05:24.434+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:05:24.497 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:24.497 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:24.509 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-04-19T01:05:24.537 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:05:24.536+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:05:24.589 INFO:tasks.rados.rados.0.smithi155.stdout:360: done (1 left) 2024-04-19T01:05:24.590 INFO:tasks.rados.rados.0.smithi155.stdout:362: done (0 left) 2024-04-19T01:05:24.590 INFO:tasks.rados.rados.0.smithi155.stdout:363: copy_from oid 3 from oid 13 current snap is 42 2024-04-19T01:05:24.590 INFO:tasks.rados.rados.0.smithi155.stdout:364: snap_remove snap 34 2024-04-19T01:05:24.630 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:24.630 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:24.640 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:05:24.636+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:05:24.646 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-04-19T01:05:24.739 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:05:24.734+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:05:24.772 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:24.772 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:24.786 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-19T01:05:24.840 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:05:24.837+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:05:24.917 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:24.918 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:24.932 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-19T01:05:24.942 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:05:24.937+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:05:25.042 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:05:25.040+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:05:25.054 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:25.055 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:25.069 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-19T01:05:25.143 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:05:25.140+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:05:25.191 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:25.192 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:25.205 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-19T01:05:25.244 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:05:25.244+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:05:25.328 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:25.329 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:25.343 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-19T01:05:25.344 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:05:25.342+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:05:25.446 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:05:25.448+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:05:25.464 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:25.465 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:25.478 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-19T01:05:25.547 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:05:25.545+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:05:25.608 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-19T01:05:25.612 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:25.612 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:25.631 INFO:tasks.rados.rados.0.smithi155.stdout:364: done (1 left) 2024-04-19T01:05:25.631 INFO:tasks.rados.rados.0.smithi155.stdout:365: read oid 43 snap -1 2024-04-19T01:05:25.631 INFO:tasks.rados.rados.0.smithi155.stdout:365: expect (ObjNum 12 snap 0 seq_num 12) 2024-04-19T01:05:25.631 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 41 current snap is 42 2024-04-19T01:05:25.632 INFO:tasks.rados.rados.0.smithi155.stdout:366: seq_num 142 ranges {0=442368,442368=720896,1163264=540672,1703936=688128} 2024-04-19T01:05:25.632 INFO:tasks.rados.rados.0.smithi155.stdout:366: writing smithi15528762-41 from 0 to 442368 tid 1 2024-04-19T01:05:25.635 INFO:tasks.rados.rados.0.smithi155.stdout:366: writing smithi15528762-41 from 442368 to 1163264 tid 2 2024-04-19T01:05:25.637 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-19T01:05:25.640 INFO:tasks.rados.rados.0.smithi155.stdout:366: writing smithi15528762-41 from 1163264 to 1703936 tid 3 2024-04-19T01:05:25.646 INFO:tasks.rados.rados.0.smithi155.stdout:366: writing smithi15528762-41 from 1703936 to 2392064 tid 4 2024-04-19T01:05:25.647 INFO:tasks.rados.rados.0.smithi155.stdout:367: rollback oid 45 current snap is 42 2024-04-19T01:05:25.647 INFO:tasks.rados.rados.0.smithi155.stdout:rollback oid 45 to 40 2024-04-19T01:05:25.647 INFO:tasks.rados.rados.0.smithi155.stdout:363: finishing copy_from to smithi15528762-3 2024-04-19T01:05:25.647 INFO:tasks.rados.rados.0.smithi155.stdout:363: got expected ENOENT (src dne) 2024-04-19T01:05:25.647 INFO:tasks.rados.rados.0.smithi155.stdout:363: finishing copy_from racing read to smithi15528762-3 2024-04-19T01:05:25.647 INFO:tasks.rados.rados.0.smithi155.stdout:363: done (3 left) 2024-04-19T01:05:25.647 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 6 current snap is 42 2024-04-19T01:05:25.647 INFO:tasks.rados.rados.0.smithi155.stdout:368: seq_num 143 ranges {4620288=458752,5079040=475136,5554176=655360,6209536=671744,6881280=147456} 2024-04-19T01:05:25.648 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:05:25.648+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:05:25.687 INFO:tasks.rados.rados.0.smithi155.stdout:368: writing smithi15528762-6 from 4620288 to 5079040 tid 1 2024-04-19T01:05:25.690 INFO:tasks.rados.rados.0.smithi155.stdout:368: writing smithi15528762-6 from 5079040 to 5554176 tid 2 2024-04-19T01:05:25.693 INFO:tasks.rados.rados.0.smithi155.stdout:368: writing smithi15528762-6 from 5554176 to 6209536 tid 3 2024-04-19T01:05:25.697 INFO:tasks.rados.rados.0.smithi155.stdout:368: writing smithi15528762-6 from 6209536 to 6881280 tid 4 2024-04-19T01:05:25.699 INFO:tasks.rados.rados.0.smithi155.stdout:368: writing smithi15528762-6 from 6881280 to 7028736 tid 5 2024-04-19T01:05:25.699 INFO:tasks.rados.rados.0.smithi155.stdout:369: rollback oid 5 current snap is 42 2024-04-19T01:05:25.699 INFO:tasks.rados.rados.0.smithi155.stdout:rollback oid 5 to 38 2024-04-19T01:05:25.699 INFO:tasks.rados.rados.0.smithi155.stdout:370: delete oid 1 current snap is 42 2024-04-19T01:05:25.705 INFO:tasks.rados.rados.0.smithi155.stdout:370: done (5 left) 2024-04-19T01:05:25.705 INFO:tasks.rados.rados.0.smithi155.stdout:371: delete oid 30 current snap is 42 2024-04-19T01:05:25.726 INFO:tasks.rados.rados.0.smithi155.stdout:368: finishing write tid 1 to smithi15528762-6 2024-04-19T01:05:25.739 INFO:tasks.rados.rados.0.smithi155.stdout:368: finishing write tid 2 to smithi15528762-6 2024-04-19T01:05:25.739 INFO:tasks.rados.rados.0.smithi155.stdout:368: finishing write tid 3 to smithi15528762-6 2024-04-19T01:05:25.740 INFO:tasks.rados.rados.0.smithi155.stdout:368: finishing write tid 4 to smithi15528762-6 2024-04-19T01:05:25.740 INFO:tasks.rados.rados.0.smithi155.stdout:368: finishing write tid 5 to smithi15528762-6 2024-04-19T01:05:25.740 INFO:tasks.rados.rados.0.smithi155.stdout:368: finishing write tid 6 to smithi15528762-6 2024-04-19T01:05:25.743 INFO:tasks.rados.rados.0.smithi155.stdout:368: finishing write tid 7 to smithi15528762-6 2024-04-19T01:05:25.743 INFO:tasks.rados.rados.0.smithi155.stdout:368: oid 6 updating version 0 to 97 2024-04-19T01:05:25.743 INFO:tasks.rados.rados.0.smithi155.stdout:368: oid 6 updating version 97 to 101 2024-04-19T01:05:25.744 INFO:tasks.rados.rados.0.smithi155.stdout:368: oid 6 version 101 is already newer than 100 2024-04-19T01:05:25.744 INFO:tasks.rados.rados.0.smithi155.stdout:368: oid 6 version 101 is already newer than 98 2024-04-19T01:05:25.744 INFO:tasks.rados.rados.0.smithi155.stdout:368: oid 6 version 101 is already newer than 99 2024-04-19T01:05:25.744 INFO:tasks.rados.rados.0.smithi155.stdout:368: oid 6 updating version 101 to 102 2024-04-19T01:05:25.744 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 6 v 102 (ObjNum 143 snap 42 seq_num 143) dirty exists 2024-04-19T01:05:25.744 INFO:tasks.rados.rados.0.smithi155.stdout:368: left oid 6 (ObjNum 143 snap 42 seq_num 143) 2024-04-19T01:05:25.750 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:05:25.748+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:05:25.770 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:25.771 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:25.777 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-19T01:05:25.778 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-19T01:05:25.850 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:05:25.849+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:05:25.900 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:25.900 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:25.914 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-19T01:05:25.918 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-04-19T01:05:25.918 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T01:05:25.939 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-04-19T01:05:25.939 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T01:05:25.957 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-04-19T01:05:25.957 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T01:05:25.959 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:05:25.958+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:05:25.977 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-04-19T01:05:25.978 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T01:05:25.995 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-04-19T01:05:25.995 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T01:05:26.016 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-04-19T01:05:26.017 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T01:05:26.038 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:26.038 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:26.039 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-04-19T01:05:26.039 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T01:05:26.052 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-04-19T01:05:26.053 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:05:26.052+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:05:26.060 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-04-19T01:05:26.061 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T01:05:26.079 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2024-04-19T01:05:26.080 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T01:05:26.102 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2024-04-19T01:05:26.102 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T01:05:26.125 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2024-04-19T01:05:26.125 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T01:05:26.148 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2024-04-19T01:05:26.149 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T01:05:26.154 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:05:26.152+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:05:26.168 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2024-04-19T01:05:26.169 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-19T01:05:26.192 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2024-04-19T01:05:26.192 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-19T01:05:26.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:05:26.189+0000 7f7015336640 0 -- 172.21.15.33:0/388303628 >> [v2:172.21.15.118:6800/506416463,v1:172.21.15.118:6801/506416463] conn(0x7f701021e390 msgr2=0x7f7010220780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-19T01:05:26.217 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:26.217 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:26.231 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-04-19T01:05:26.275 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:05:26.274+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:05:26.349 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:26.350 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:26.355 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:05:26.356+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:05:26.363 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-04-19T01:05:26.416 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2024-04-19T01:05:26.416 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-19T01:05:26.451 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2024-04-19T01:05:26.451 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-19T01:05:26.456 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:05:26.453+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:05:26.519 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2024-04-19T01:05:26.533 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:05:26.533 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-04-19T01:05:26.533 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-04-19T01:05:26.533 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-04-19T01:05:26.533 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.533 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s2 44/31 ECSubWrite(tid=82, reqid=client.4604.0:280, at_version=44'50, trim_to=0'0, roll_forward_to=43'48))", 2024-04-19T01:05:26.533 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:14.882222+0000", 2024-04-19T01:05:26.533 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 71.650339521999996, 2024-04-19T01:05:26.533 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.035827673999999997, 2024-04-19T01:05:26.533 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:26.533 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:26.533 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:26.534 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:26.534 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.534 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:26.534 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882222+0000", 2024-04-19T01:05:26.534 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:26.534 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.534 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.534 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:26.534 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882222+0000", 2024-04-19T01:05:26.534 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:26.534 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.534 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.534 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:26.535 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882222+0000", 2024-04-19T01:05:26.535 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7999999999999995e-07 2024-04-19T01:05:26.535 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.535 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.535 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:26.535 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882709+0000", 2024-04-19T01:05:26.535 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00048730799999999999 2024-04-19T01:05:26.535 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.535 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.535 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:26.535 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882742+0000", 2024-04-19T01:05:26.535 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2428000000000002e-05 2024-04-19T01:05:26.535 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.535 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.536 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:26.536 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.882765+0000", 2024-04-19T01:05:26.536 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2693000000000001e-05 2024-04-19T01:05:26.536 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.536 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.536 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:26.536 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.883062+0000", 2024-04-19T01:05:26.536 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029777100000000001 2024-04-19T01:05:26.536 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.536 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.536 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:26.536 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.883088+0000", 2024-04-19T01:05:26.536 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6018999999999999e-05 2024-04-19T01:05:26.536 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.537 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.537 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:26.537 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.917995+0000", 2024-04-19T01:05:26.537 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.034906988999999999 2024-04-19T01:05:26.537 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.537 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.537 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:26.537 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.918049+0000", 2024-04-19T01:05:26.537 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3885999999999999e-05 2024-04-19T01:05:26.537 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:26.537 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:26.537 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:26.537 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.537 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.538 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 59/31 ECSubWrite(tid=66, reqid=client.4604.0:423, at_version=59'51, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:05:26.538 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.541618+0000", 2024-04-19T01:05:26.538 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 56.990943440000002, 2024-04-19T01:05:26.538 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.040796200999999997, 2024-04-19T01:05:26.538 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:26.538 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:26.538 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:26.538 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:26.538 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.538 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:26.538 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.541618+0000", 2024-04-19T01:05:26.538 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:26.538 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.538 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.539 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:26.539 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.541618+0000", 2024-04-19T01:05:26.539 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:26.539 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.539 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.539 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:26.539 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.541618+0000", 2024-04-19T01:05:26.539 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9699999999999995e-07 2024-04-19T01:05:26.539 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.539 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.539 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:26.539 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.542215+0000", 2024-04-19T01:05:26.539 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00059629300000000002 2024-04-19T01:05:26.539 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.540 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.540 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:26.540 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.542247+0000", 2024-04-19T01:05:26.540 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2339000000000003e-05 2024-04-19T01:05:26.540 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.540 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.540 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:26.540 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.542272+0000", 2024-04-19T01:05:26.540 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4973e-05 2024-04-19T01:05:26.540 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.540 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.540 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:26.540 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.542445+0000", 2024-04-19T01:05:26.540 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017274599999999999 2024-04-19T01:05:26.540 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.541 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.541 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:26.541 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.542468+0000", 2024-04-19T01:05:26.541 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3689e-05 2024-04-19T01:05:26.541 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.541 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.541 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:26.541 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.582330+0000", 2024-04-19T01:05:26.541 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.039861946000000002 2024-04-19T01:05:26.541 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.541 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.541 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:26.541 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.582414+0000", 2024-04-19T01:05:26.541 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3517999999999996e-05 2024-04-19T01:05:26.541 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:26.542 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:26.542 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:26.542 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.542 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.542 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 59/31 ECSubWrite(tid=67, reqid=client.4604.0:424, at_version=59'52, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:05:26.542 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.549371+0000", 2024-04-19T01:05:26.542 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 56.983189617000001, 2024-04-19T01:05:26.542 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.033134352999999998, 2024-04-19T01:05:26.542 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:26.542 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:26.542 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:26.542 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:26.542 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.543 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:26.543 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549371+0000", 2024-04-19T01:05:26.543 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:26.543 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.543 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.543 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:26.543 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549371+0000", 2024-04-19T01:05:26.543 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:26.543 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.543 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.543 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:26.543 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549375+0000", 2024-04-19T01:05:26.543 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0869999999999998e-06 2024-04-19T01:05:26.543 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.543 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.544 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:26.544 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549729+0000", 2024-04-19T01:05:26.545 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000354225 2024-04-19T01:05:26.545 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.545 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.545 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:26.545 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549769+0000", 2024-04-19T01:05:26.545 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0488999999999998e-05 2024-04-19T01:05:26.545 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.545 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.545 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:26.545 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549793+0000", 2024-04-19T01:05:26.545 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3702000000000001e-05 2024-04-19T01:05:26.545 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.546 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.546 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:26.546 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.561556+0000", 2024-04-19T01:05:26.546 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011763018 2024-04-19T01:05:26.546 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.546 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.546 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:26.546 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.561583+0000", 2024-04-19T01:05:26.546 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7243e-05 2024-04-19T01:05:26.546 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.546 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.546 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:26.546 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.582445+0000", 2024-04-19T01:05:26.546 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020862084999999999 2024-04-19T01:05:26.547 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.547 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.547 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:26.547 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.582506+0000", 2024-04-19T01:05:26.547 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0504000000000002e-05 2024-04-19T01:05:26.547 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:26.547 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:26.547 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:26.547 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.547 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.547 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.4s2 71/31 ECSubWrite(tid=89, reqid=client.4604.0:518, at_version=71'22, trim_to=0'0, roll_forward_to=52'20))", 2024-04-19T01:05:26.547 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:41.545110+0000", 2024-04-19T01:05:26.547 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 44.987450666000001, 2024-04-19T01:05:26.547 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.049869387000000001, 2024-04-19T01:05:26.548 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:26.548 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:26.548 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:26.548 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:26.548 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.548 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:26.548 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.545110+0000", 2024-04-19T01:05:26.548 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:26.548 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.548 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.548 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:26.548 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.545110+0000", 2024-04-19T01:05:26.548 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:26.548 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.548 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.549 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:26.549 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.545111+0000", 2024-04-19T01:05:26.549 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3399999999999999e-07 2024-04-19T01:05:26.549 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.549 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.549 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:26.549 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.546538+0000", 2024-04-19T01:05:26.549 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0014274330000000001 2024-04-19T01:05:26.549 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.549 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.549 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:26.549 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.546582+0000", 2024-04-19T01:05:26.549 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3155000000000003e-05 2024-04-19T01:05:26.549 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.550 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.550 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:26.550 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.546610+0000", 2024-04-19T01:05:26.550 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8807e-05 2024-04-19T01:05:26.550 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.550 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.550 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:26.550 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.546756+0000", 2024-04-19T01:05:26.550 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014540399999999999 2024-04-19T01:05:26.550 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.550 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.550 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:26.550 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.546778+0000", 2024-04-19T01:05:26.550 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1722999999999999e-05 2024-04-19T01:05:26.551 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.551 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.551 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:26.551 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.594901+0000", 2024-04-19T01:05:26.551 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.048123725999999999 2024-04-19T01:05:26.551 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.551 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.551 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:26.551 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:41.594980+0000", 2024-04-19T01:05:26.551 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8604999999999997e-05 2024-04-19T01:05:26.551 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:26.551 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:26.551 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:26.551 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.551 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.552 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 76/31 ECSubWrite(tid=97, reqid=client.4604.0:568, at_version=76'74, trim_to=0'0, roll_forward_to=74'72))", 2024-04-19T01:05:26.552 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:46.580144+0000", 2024-04-19T01:05:26.552 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 39.952417025000003, 2024-04-19T01:05:26.552 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.068404570999999997, 2024-04-19T01:05:26.552 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:26.552 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:26.552 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:26.552 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:26.552 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.552 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:26.552 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.580144+0000", 2024-04-19T01:05:26.552 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:26.552 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.552 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.553 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:26.553 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.580144+0000", 2024-04-19T01:05:26.553 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:26.553 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.553 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.553 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:26.553 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.580145+0000", 2024-04-19T01:05:26.553 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8599999999999998e-07 2024-04-19T01:05:26.553 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.553 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.553 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:26.553 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.580959+0000", 2024-04-19T01:05:26.553 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00081373299999999997 2024-04-19T01:05:26.553 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.554 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.554 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:26.554 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.581006+0000", 2024-04-19T01:05:26.554 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7481999999999997e-05 2024-04-19T01:05:26.554 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.554 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.554 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:26.554 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.581034+0000", 2024-04-19T01:05:26.554 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8293000000000002e-05 2024-04-19T01:05:26.554 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.554 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.554 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:26.554 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.581198+0000", 2024-04-19T01:05:26.554 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016329100000000001 2024-04-19T01:05:26.555 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.555 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.555 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:26.555 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.581223+0000", 2024-04-19T01:05:26.555 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5656e-05 2024-04-19T01:05:26.555 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.555 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.555 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:26.555 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.648474+0000", 2024-04-19T01:05:26.556 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.067250402000000001 2024-04-19T01:05:26.556 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.556 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.556 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:26.556 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.648549+0000", 2024-04-19T01:05:26.556 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5028000000000001e-05 2024-04-19T01:05:26.556 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:26.556 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:26.556 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:26.556 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.556 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.556 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 76/31 ECSubWrite(tid=98, reqid=client.4604.0:569, at_version=76'75, trim_to=0'0, roll_forward_to=74'72))", 2024-04-19T01:05:26.556 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:46.587805+0000", 2024-04-19T01:05:26.557 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 39.944755647000001, 2024-04-19T01:05:26.557 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.060794462000000001, 2024-04-19T01:05:26.557 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:26.557 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:26.557 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:26.557 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:26.557 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.557 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:26.557 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.587805+0000", 2024-04-19T01:05:26.557 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:26.557 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.557 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.557 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:26.558 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.587805+0000", 2024-04-19T01:05:26.558 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:26.558 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.558 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.558 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:26.558 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.587808+0000", 2024-04-19T01:05:26.559 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8399999999999999e-06 2024-04-19T01:05:26.559 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.559 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.560 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:26.560 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.588154+0000", 2024-04-19T01:05:26.560 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00034547400000000002 2024-04-19T01:05:26.560 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.560 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.560 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:26.560 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.588204+0000", 2024-04-19T01:05:26.560 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0482000000000002e-05 2024-04-19T01:05:26.560 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.560 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.560 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:26.560 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.588227+0000", 2024-04-19T01:05:26.560 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2339000000000001e-05 2024-04-19T01:05:26.560 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.561 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.561 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:26.561 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.596418+0000", 2024-04-19T01:05:26.561 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0081914320000000002 2024-04-19T01:05:26.561 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.561 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.561 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:26.561 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.596444+0000", 2024-04-19T01:05:26.561 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5953e-05 2024-04-19T01:05:26.561 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.562 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.562 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:26.562 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.648559+0000", 2024-04-19T01:05:26.562 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.052114797999999997 2024-04-19T01:05:26.562 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.562 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.562 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:26.562 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.648600+0000", 2024-04-19T01:05:26.562 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1143999999999998e-05 2024-04-19T01:05:26.562 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:26.562 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:26.562 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:26.562 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.562 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.563 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 76/31 ECSubWrite(tid=99, reqid=client.4604.0:570, at_version=76'76, trim_to=0'0, roll_forward_to=74'72))", 2024-04-19T01:05:26.563 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:46.594492+0000", 2024-04-19T01:05:26.563 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 39.938069255999999, 2024-04-19T01:05:26.563 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.054401448999999998, 2024-04-19T01:05:26.563 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:26.563 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:26.563 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:26.563 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:26.563 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.563 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:26.563 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.594492+0000", 2024-04-19T01:05:26.563 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:26.563 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.563 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.564 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:26.564 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.594492+0000", 2024-04-19T01:05:26.564 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:26.564 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.564 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.564 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:26.564 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.594495+0000", 2024-04-19T01:05:26.564 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6769999999999999e-06 2024-04-19T01:05:26.564 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.564 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.564 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:26.564 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.594850+0000", 2024-04-19T01:05:26.564 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000355247 2024-04-19T01:05:26.564 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.564 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.565 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:26.565 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.594886+0000", 2024-04-19T01:05:26.565 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6545999999999999e-05 2024-04-19T01:05:26.565 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.565 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.565 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:26.565 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.594911+0000", 2024-04-19T01:05:26.565 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4366e-05 2024-04-19T01:05:26.565 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.566 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.566 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:26.566 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.613951+0000", 2024-04-19T01:05:26.566 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019039865999999999 2024-04-19T01:05:26.566 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.566 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.566 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:26.566 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.613968+0000", 2024-04-19T01:05:26.566 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7825999999999998e-05 2024-04-19T01:05:26.566 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.566 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.566 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:26.566 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.648836+0000", 2024-04-19T01:05:26.567 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.034867926 2024-04-19T01:05:26.567 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.567 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.567 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:26.567 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.648893+0000", 2024-04-19T01:05:26.567 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6994999999999999e-05 2024-04-19T01:05:26.567 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:26.567 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:26.567 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:26.567 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.567 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.567 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 76/31 ECSubWrite(tid=100, reqid=client.4604.0:571, at_version=76'77, trim_to=0'0, roll_forward_to=74'72))", 2024-04-19T01:05:26.567 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:46.600988+0000", 2024-04-19T01:05:26.567 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 39.931573208000003, 2024-04-19T01:05:26.568 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.047952294999999999, 2024-04-19T01:05:26.568 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:26.568 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:26.568 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:26.568 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:26.568 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.568 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:26.568 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.600988+0000", 2024-04-19T01:05:26.568 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:26.568 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.568 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.568 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:26.568 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.600988+0000", 2024-04-19T01:05:26.568 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:26.569 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.569 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.569 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:26.569 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.600990+0000", 2024-04-19T01:05:26.569 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3439999999999999e-06 2024-04-19T01:05:26.569 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.569 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.569 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:26.569 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.601037+0000", 2024-04-19T01:05:26.569 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6548999999999998e-05 2024-04-19T01:05:26.569 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.569 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.569 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:26.570 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.601082+0000", 2024-04-19T01:05:26.570 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4978999999999997e-05 2024-04-19T01:05:26.570 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.570 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.570 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:26.570 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.601105+0000", 2024-04-19T01:05:26.570 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3519999999999998e-05 2024-04-19T01:05:26.570 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.570 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.570 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:26.570 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.621093+0000", 2024-04-19T01:05:26.570 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0199878 2024-04-19T01:05:26.570 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.570 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.570 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:26.571 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.621110+0000", 2024-04-19T01:05:26.571 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7316000000000002e-05 2024-04-19T01:05:26.571 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.571 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.571 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:26.571 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.648899+0000", 2024-04-19T01:05:26.571 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.027788891 2024-04-19T01:05:26.571 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.571 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.571 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:26.571 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.648940+0000", 2024-04-19T01:05:26.571 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0896000000000003e-05 2024-04-19T01:05:26.571 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:26.571 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:26.572 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:26.572 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.572 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.572 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 76/31 ECSubWrite(tid=101, reqid=client.4604.0:573, at_version=76'79, trim_to=0'0, roll_forward_to=74'72))", 2024-04-19T01:05:26.572 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:46.612179+0000", 2024-04-19T01:05:26.572 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 39.920381866, 2024-04-19T01:05:26.572 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.036832701000000002, 2024-04-19T01:05:26.572 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:26.572 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:26.572 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:26.572 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:26.572 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.572 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:26.572 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.612179+0000", 2024-04-19T01:05:26.573 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:26.573 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.573 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.573 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:26.573 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.612179+0000", 2024-04-19T01:05:26.573 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:26.573 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.573 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.573 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:26.573 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.612181+0000", 2024-04-19T01:05:26.573 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8759999999999999e-06 2024-04-19T01:05:26.573 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.573 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.573 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:26.573 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.612643+0000", 2024-04-19T01:05:26.574 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00046178400000000002 2024-04-19T01:05:26.574 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.574 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.574 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:26.574 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.612676+0000", 2024-04-19T01:05:26.574 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3339e-05 2024-04-19T01:05:26.574 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.574 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.574 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:26.574 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.612694+0000", 2024-04-19T01:05:26.574 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7655999999999998e-05 2024-04-19T01:05:26.574 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.574 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.574 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:26.574 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.621352+0000", 2024-04-19T01:05:26.575 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0086576210000000008 2024-04-19T01:05:26.575 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.575 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.575 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:26.575 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.621360+0000", 2024-04-19T01:05:26.575 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4009999999999997e-06 2024-04-19T01:05:26.575 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.575 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.575 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:26.575 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.648969+0000", 2024-04-19T01:05:26.576 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.027608669999999998 2024-04-19T01:05:26.576 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.576 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.576 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:26.576 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:46.649012+0000", 2024-04-19T01:05:26.576 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3353999999999999e-05 2024-04-19T01:05:26.576 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:26.576 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:26.576 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:26.576 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.576 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.576 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 79/31 ECSubWrite(tid=115, reqid=client.4604.0:591, at_version=79'90, trim_to=0'0, roll_forward_to=79'86))", 2024-04-19T01:05:26.576 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:49.657905+0000", 2024-04-19T01:05:26.576 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 36.874656111999997, 2024-04-19T01:05:26.577 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.043682177000000003, 2024-04-19T01:05:26.577 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:26.577 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:26.577 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:26.577 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:26.577 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.577 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:26.577 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.657905+0000", 2024-04-19T01:05:26.577 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:26.577 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.577 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.577 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:26.577 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.657905+0000", 2024-04-19T01:05:26.577 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:26.577 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.578 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.578 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:26.578 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.657907+0000", 2024-04-19T01:05:26.578 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2869999999999999e-06 2024-04-19T01:05:26.578 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.578 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.578 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:26.578 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658833+0000", 2024-04-19T01:05:26.578 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00092593299999999999 2024-04-19T01:05:26.578 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.578 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.578 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:26.578 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658867+0000", 2024-04-19T01:05:26.578 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4048999999999999e-05 2024-04-19T01:05:26.579 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.579 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.579 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:26.579 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658883+0000", 2024-04-19T01:05:26.579 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5502000000000002e-05 2024-04-19T01:05:26.579 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.579 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.579 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:26.579 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.659042+0000", 2024-04-19T01:05:26.579 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015962800000000001 2024-04-19T01:05:26.579 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.579 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.579 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:26.579 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.659073+0000", 2024-04-19T01:05:26.580 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0232000000000001e-05 2024-04-19T01:05:26.580 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.580 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.580 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:26.580 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.701480+0000", 2024-04-19T01:05:26.580 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.042407621999999999 2024-04-19T01:05:26.580 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.580 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.580 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:26.580 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.701587+0000", 2024-04-19T01:05:26.580 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010692399999999999 2024-04-19T01:05:26.580 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:26.580 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:26.580 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:26.580 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.581 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.581 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 79/31 ECSubWrite(tid=116, reqid=client.4604.0:592, at_version=79'91, trim_to=0'0, roll_forward_to=79'88))", 2024-04-19T01:05:26.581 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:49.666213+0000", 2024-04-19T01:05:26.581 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 36.866348219000002, 2024-04-19T01:05:26.581 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.035451589999999998, 2024-04-19T01:05:26.581 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:26.581 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:26.581 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:26.581 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:26.581 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.581 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:26.581 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.666213+0000", 2024-04-19T01:05:26.581 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:26.581 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.581 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.582 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:26.582 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.666213+0000", 2024-04-19T01:05:26.582 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:26.582 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.582 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.582 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:26.582 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.666215+0000", 2024-04-19T01:05:26.582 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.95e-06 2024-04-19T01:05:26.582 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.582 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.582 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:26.582 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.666782+0000", 2024-04-19T01:05:26.582 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00056714299999999999 2024-04-19T01:05:26.582 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.583 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.583 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:26.583 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.666830+0000", 2024-04-19T01:05:26.583 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7787000000000002e-05 2024-04-19T01:05:26.583 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.583 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.583 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:26.583 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.666852+0000", 2024-04-19T01:05:26.583 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2303e-05 2024-04-19T01:05:26.583 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.583 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.583 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:26.583 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.677400+0000", 2024-04-19T01:05:26.583 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010547503999999999 2024-04-19T01:05:26.583 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.584 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.584 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:26.584 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.677425+0000", 2024-04-19T01:05:26.584 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5644999999999999e-05 2024-04-19T01:05:26.584 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.584 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.584 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:26.584 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.701599+0000", 2024-04-19T01:05:26.584 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024174079000000001 2024-04-19T01:05:26.584 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.584 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.584 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:26.584 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.701664+0000", 2024-04-19T01:05:26.584 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5178999999999994e-05 2024-04-19T01:05:26.585 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:26.585 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:26.585 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:26.585 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.585 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.585 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.4s2 79/31 ECSubWrite(tid=111, reqid=client.4604.0:605, at_version=79'36, trim_to=0'0, roll_forward_to=79'34))", 2024-04-19T01:05:26.585 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:49.681460+0000", 2024-04-19T01:05:26.585 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 36.851101290000003, 2024-04-19T01:05:26.585 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.032313512000000003, 2024-04-19T01:05:26.585 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:26.585 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:26.585 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:26.585 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:26.586 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.586 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:26.586 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.681460+0000", 2024-04-19T01:05:26.586 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:26.586 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.586 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.586 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:26.586 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.681460+0000", 2024-04-19T01:05:26.586 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:26.587 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.587 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.587 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:26.587 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.681462+0000", 2024-04-19T01:05:26.587 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8950000000000001e-06 2024-04-19T01:05:26.587 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.587 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.587 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:26.587 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.682022+0000", 2024-04-19T01:05:26.587 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00055996400000000003 2024-04-19T01:05:26.587 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.587 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.587 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:26.587 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.682075+0000", 2024-04-19T01:05:26.587 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3100000000000003e-05 2024-04-19T01:05:26.588 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.588 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.588 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:26.588 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.682096+0000", 2024-04-19T01:05:26.588 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0973000000000001e-05 2024-04-19T01:05:26.588 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.588 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.588 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:26.588 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.682234+0000", 2024-04-19T01:05:26.588 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013790499999999999 2024-04-19T01:05:26.588 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.588 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.588 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:26.588 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.682258+0000", 2024-04-19T01:05:26.588 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4528e-05 2024-04-19T01:05:26.589 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.589 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.589 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:26.589 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.713716+0000", 2024-04-19T01:05:26.589 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.03145746 2024-04-19T01:05:26.589 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.589 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.589 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:26.589 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.713773+0000", 2024-04-19T01:05:26.589 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7686999999999998e-05 2024-04-19T01:05:26.589 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:26.589 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:26.589 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:26.589 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.590 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.590 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 81/31 ECSubWrite(tid=122, reqid=client.4604.0:626, at_version=81'96, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:05:26.590 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:51.594239+0000", 2024-04-19T01:05:26.590 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 34.938322096, 2024-04-19T01:05:26.590 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.043938814999999999, 2024-04-19T01:05:26.590 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:26.590 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:26.590 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:26.590 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:26.590 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.590 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:26.590 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.594239+0000", 2024-04-19T01:05:26.590 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:26.590 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.590 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.591 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:26.591 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.594239+0000", 2024-04-19T01:05:26.591 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:26.591 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.591 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.591 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:26.591 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.594240+0000", 2024-04-19T01:05:26.591 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6300000000000005e-07 2024-04-19T01:05:26.591 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.591 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.591 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:26.591 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.595111+0000", 2024-04-19T01:05:26.591 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00087121599999999996 2024-04-19T01:05:26.591 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.591 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.592 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:26.592 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.595145+0000", 2024-04-19T01:05:26.592 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3698000000000002e-05 2024-04-19T01:05:26.592 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.592 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.592 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:26.592 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.595166+0000", 2024-04-19T01:05:26.592 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1081999999999999e-05 2024-04-19T01:05:26.592 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.592 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.592 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:26.592 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.595312+0000", 2024-04-19T01:05:26.592 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014611299999999999 2024-04-19T01:05:26.593 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.593 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.593 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:26.593 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.595336+0000", 2024-04-19T01:05:26.593 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4057999999999999e-05 2024-04-19T01:05:26.593 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.593 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.593 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:26.593 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.638124+0000", 2024-04-19T01:05:26.593 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.042788228999999997 2024-04-19T01:05:26.593 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.593 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.593 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:26.593 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.638178+0000", 2024-04-19T01:05:26.593 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3755999999999998e-05 2024-04-19T01:05:26.594 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:26.594 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:26.594 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:26.594 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.595 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.595 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 81/31 ECSubWrite(tid=123, reqid=client.4604.0:627, at_version=81'97, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:05:26.595 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:51.600586+0000", 2024-04-19T01:05:26.595 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 34.931975104000003, 2024-04-19T01:05:26.595 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.037646446, 2024-04-19T01:05:26.595 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:26.595 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:26.595 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:26.595 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:26.595 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.595 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:26.595 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.600586+0000", 2024-04-19T01:05:26.595 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:26.596 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.596 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.596 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:26.596 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.600586+0000", 2024-04-19T01:05:26.596 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:26.596 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.596 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.596 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:26.597 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.600588+0000", 2024-04-19T01:05:26.597 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0999999999999998e-06 2024-04-19T01:05:26.597 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.597 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.597 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:26.597 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.600966+0000", 2024-04-19T01:05:26.597 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000377878 2024-04-19T01:05:26.597 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.597 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.597 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:26.597 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.601011+0000", 2024-04-19T01:05:26.597 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4920999999999999e-05 2024-04-19T01:05:26.597 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.597 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.598 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:26.598 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.601032+0000", 2024-04-19T01:05:26.598 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1192999999999999e-05 2024-04-19T01:05:26.598 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.598 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.598 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:26.598 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.613182+0000", 2024-04-19T01:05:26.598 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012149900999999999 2024-04-19T01:05:26.598 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.598 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.598 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:26.598 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.613204+0000", 2024-04-19T01:05:26.598 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1761999999999999e-05 2024-04-19T01:05:26.598 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.599 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.599 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:26.599 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.638194+0000", 2024-04-19T01:05:26.599 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024990413 2024-04-19T01:05:26.599 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.599 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.599 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:26.599 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.638232+0000", 2024-04-19T01:05:26.599 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8278000000000002e-05 2024-04-19T01:05:26.599 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:26.599 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:26.599 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:26.599 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.599 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.599 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 81/31 ECSubWrite(tid=124, reqid=client.4604.0:628, at_version=81'98, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:05:26.600 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:51.606371+0000", 2024-04-19T01:05:26.600 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 34.926190234000003, 2024-04-19T01:05:26.600 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.031905021999999998, 2024-04-19T01:05:26.600 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:26.600 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:26.600 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:26.600 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:26.600 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.600 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:26.600 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.606371+0000", 2024-04-19T01:05:26.600 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:26.600 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.600 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.600 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:26.600 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.606371+0000", 2024-04-19T01:05:26.601 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:26.601 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.601 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.601 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:26.601 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.606373+0000", 2024-04-19T01:05:26.601 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.108e-06 2024-04-19T01:05:26.601 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.601 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.601 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:26.601 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.606762+0000", 2024-04-19T01:05:26.601 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000389398 2024-04-19T01:05:26.601 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.601 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.601 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:26.601 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.606802+0000", 2024-04-19T01:05:26.602 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0098000000000001e-05 2024-04-19T01:05:26.602 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.602 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.602 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:26.602 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.606825+0000", 2024-04-19T01:05:26.602 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.247e-05 2024-04-19T01:05:26.602 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.602 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.602 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:26.602 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.626279+0000", 2024-04-19T01:05:26.602 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019454050000000001 2024-04-19T01:05:26.602 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.602 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.602 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:26.603 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.626301+0000", 2024-04-19T01:05:26.603 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2011e-05 2024-04-19T01:05:26.603 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.603 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.603 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:26.603 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.638239+0000", 2024-04-19T01:05:26.603 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011937830999999999 2024-04-19T01:05:26.603 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.603 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.603 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:26.603 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.638276+0000", 2024-04-19T01:05:26.603 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7055999999999999e-05 2024-04-19T01:05:26.603 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:26.603 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:26.603 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:26.604 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.604 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.604 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 81/31 ECSubWrite(tid=125, reqid=client.4604.0:629, at_version=81'99, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:05:26.604 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:51.610678+0000", 2024-04-19T01:05:26.604 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 34.921883196000003, 2024-04-19T01:05:26.604 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.046619226999999999, 2024-04-19T01:05:26.604 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:26.604 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:26.604 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:26.604 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:26.604 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.604 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:26.604 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.610678+0000", 2024-04-19T01:05:26.604 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:26.605 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.605 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.605 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:26.605 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.610678+0000", 2024-04-19T01:05:26.605 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:26.605 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.605 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.605 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:26.605 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.610680+0000", 2024-04-19T01:05:26.605 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0499999999999999e-06 2024-04-19T01:05:26.605 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.605 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.605 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:26.605 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.610729+0000", 2024-04-19T01:05:26.605 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8906999999999998e-05 2024-04-19T01:05:26.606 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.606 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.606 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:26.606 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.610776+0000", 2024-04-19T01:05:26.606 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7250999999999999e-05 2024-04-19T01:05:26.606 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.606 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.606 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:26.606 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.610800+0000", 2024-04-19T01:05:26.606 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4094999999999999e-05 2024-04-19T01:05:26.606 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.606 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.606 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:26.607 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.637697+0000", 2024-04-19T01:05:26.607 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.026896798 2024-04-19T01:05:26.607 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.607 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.607 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:26.607 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.637715+0000", 2024-04-19T01:05:26.607 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7923000000000001e-05 2024-04-19T01:05:26.607 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.607 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.608 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:26.608 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.657204+0000", 2024-04-19T01:05:26.608 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.01948861 2024-04-19T01:05:26.608 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.608 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.608 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:26.608 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.657297+0000", 2024-04-19T01:05:26.608 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3592999999999997e-05 2024-04-19T01:05:26.608 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:26.608 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:26.608 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:26.608 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.608 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.608 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 81/31 ECSubWrite(tid=126, reqid=client.4604.0:631, at_version=81'101, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:05:26.609 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:51.619126+0000", 2024-04-19T01:05:26.609 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 34.913434711000001, 2024-04-19T01:05:26.609 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.038912264000000002, 2024-04-19T01:05:26.609 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:26.609 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:26.609 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:26.609 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:26.609 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.609 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:26.609 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.619126+0000", 2024-04-19T01:05:26.609 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:26.609 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.609 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.609 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:26.609 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.619126+0000", 2024-04-19T01:05:26.610 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:26.610 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.610 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.610 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:26.610 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.619128+0000", 2024-04-19T01:05:26.610 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0889999999999998e-06 2024-04-19T01:05:26.610 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.610 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.610 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:26.610 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.619962+0000", 2024-04-19T01:05:26.610 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00083304099999999997 2024-04-19T01:05:26.610 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.610 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.610 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:26.611 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.620004+0000", 2024-04-19T01:05:26.611 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2035e-05 2024-04-19T01:05:26.611 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.611 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.611 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:26.611 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.620029+0000", 2024-04-19T01:05:26.611 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5429e-05 2024-04-19T01:05:26.611 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.611 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.611 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:26.611 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.638344+0000", 2024-04-19T01:05:26.611 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018315498999999999 2024-04-19T01:05:26.611 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.611 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.611 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:26.612 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.638358+0000", 2024-04-19T01:05:26.612 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3392e-05 2024-04-19T01:05:26.612 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.612 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.612 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:26.612 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.657967+0000", 2024-04-19T01:05:26.612 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019608961000000001 2024-04-19T01:05:26.612 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.612 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.612 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:26.612 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:51.658039+0000", 2024-04-19T01:05:26.612 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1817999999999996e-05 2024-04-19T01:05:26.612 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:26.612 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:26.612 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:26.613 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.613 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.613 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s2 92/31 ECSubWrite(tid=202, reqid=client.4604.0:691, at_version=92'124, trim_to=0'0, roll_forward_to=89'122))", 2024-04-19T01:05:26.613 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:05:02.633260+0000", 2024-04-19T01:05:26.613 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 23.899301576999999, 2024-04-19T01:05:26.613 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.043633751999999998, 2024-04-19T01:05:26.613 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:26.613 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:26.613 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:26.613 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:26.613 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.613 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:26.613 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.633260+0000", 2024-04-19T01:05:26.613 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:26.614 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.614 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.614 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:26.614 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.633260+0000", 2024-04-19T01:05:26.614 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:26.614 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.614 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.614 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:26.614 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.633261+0000", 2024-04-19T01:05:26.614 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.11e-06 2024-04-19T01:05:26.614 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.614 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.614 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:26.614 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.634187+0000", 2024-04-19T01:05:26.614 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00092616400000000005 2024-04-19T01:05:26.615 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.615 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.615 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:26.615 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.634217+0000", 2024-04-19T01:05:26.615 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9978999999999999e-05 2024-04-19T01:05:26.615 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.615 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.615 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:26.615 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.634241+0000", 2024-04-19T01:05:26.615 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3788999999999999e-05 2024-04-19T01:05:26.615 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.615 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.615 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:26.615 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.634479+0000", 2024-04-19T01:05:26.615 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00023878300000000001 2024-04-19T01:05:26.616 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.616 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.616 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:26.616 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.634513+0000", 2024-04-19T01:05:26.616 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3605000000000001e-05 2024-04-19T01:05:26.616 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.616 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.616 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:26.616 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.676795+0000", 2024-04-19T01:05:26.616 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.042281883999999999 2024-04-19T01:05:26.616 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.616 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.616 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:26.616 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.676893+0000", 2024-04-19T01:05:26.617 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8437999999999995e-05 2024-04-19T01:05:26.617 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:26.617 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:26.617 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:26.617 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.617 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.617 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s2 92/31 ECSubWrite(tid=203, reqid=client.4604.0:692, at_version=92'125, trim_to=0'0, roll_forward_to=89'122))", 2024-04-19T01:05:26.617 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:05:02.647807+0000", 2024-04-19T01:05:26.617 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 23.884754415, 2024-04-19T01:05:26.618 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.040516416, 2024-04-19T01:05:26.618 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:26.618 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:26.618 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:26.618 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:26.618 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.618 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:26.618 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.647807+0000", 2024-04-19T01:05:26.618 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:26.618 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.618 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.618 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:26.618 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.647807+0000", 2024-04-19T01:05:26.618 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:26.619 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.619 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.619 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:26.619 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.647810+0000", 2024-04-19T01:05:26.619 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.96e-06 2024-04-19T01:05:26.619 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.619 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.619 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:26.619 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.648104+0000", 2024-04-19T01:05:26.619 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029466300000000002 2024-04-19T01:05:26.619 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.619 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.619 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:26.619 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.648155+0000", 2024-04-19T01:05:26.620 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0992000000000002e-05 2024-04-19T01:05:26.620 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.620 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.620 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:26.620 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.648180+0000", 2024-04-19T01:05:26.620 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5159000000000001e-05 2024-04-19T01:05:26.620 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.620 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.620 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:26.620 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.654587+0000", 2024-04-19T01:05:26.620 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0064061989999999996 2024-04-19T01:05:26.620 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.620 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.620 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:26.620 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.654615+0000", 2024-04-19T01:05:26.620 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7841999999999999e-05 2024-04-19T01:05:26.621 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.621 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.621 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:26.621 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.688247+0000", 2024-04-19T01:05:26.621 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.033632044 2024-04-19T01:05:26.621 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.621 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.621 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:26.621 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:02.688323+0000", 2024-04-19T01:05:26.621 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6557000000000006e-05 2024-04-19T01:05:26.621 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:26.621 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:26.621 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:26.621 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.622 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.622 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s2 107/31 ECSubWrite(tid=213, reqid=client.4604.0:811, at_version=107'200, trim_to=0'0, roll_forward_to=106'199))", 2024-04-19T01:05:26.622 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:05:17.593711+0000", 2024-04-19T01:05:26.622 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 8.9388500390000001, 2024-04-19T01:05:26.622 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.033833528000000002, 2024-04-19T01:05:26.622 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:26.622 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:26.622 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:26.622 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:26.622 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.622 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:26.622 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:17.593711+0000", 2024-04-19T01:05:26.622 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:26.622 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.623 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.623 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:26.623 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:17.593711+0000", 2024-04-19T01:05:26.623 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:26.623 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.623 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.623 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:26.623 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:17.593711+0000", 2024-04-19T01:05:26.623 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9700000000000002e-07 2024-04-19T01:05:26.623 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.623 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.623 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:26.623 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:17.594364+0000", 2024-04-19T01:05:26.623 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00065229899999999996 2024-04-19T01:05:26.623 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.624 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.624 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:26.624 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:17.594384+0000", 2024-04-19T01:05:26.624 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0024999999999999e-05 2024-04-19T01:05:26.624 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.624 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.624 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:26.624 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:17.594406+0000", 2024-04-19T01:05:26.624 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2008e-05 2024-04-19T01:05:26.624 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.624 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.624 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:26.624 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:17.594537+0000", 2024-04-19T01:05:26.624 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013070399999999999 2024-04-19T01:05:26.624 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.625 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.625 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:26.625 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:17.594549+0000", 2024-04-19T01:05:26.625 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2167000000000001e-05 2024-04-19T01:05:26.625 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.625 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.625 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:26.625 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:17.627508+0000", 2024-04-19T01:05:26.625 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.032959307 2024-04-19T01:05:26.625 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:26.625 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:26.625 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:26.625 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:17.627545+0000", 2024-04-19T01:05:26.625 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6621000000000001e-05 2024-04-19T01:05:26.625 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:26.626 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:26.626 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:26.626 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:26.626 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:26.626 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:05:26.629 INFO:tasks.rados.rados.0.smithi155.stdout:367: finishing rollback tid 0 to smithi15528762-45 2024-04-19T01:05:26.629 INFO:tasks.rados.rados.0.smithi155.stdout:368: done (5 left) 2024-04-19T01:05:26.629 INFO:tasks.rados.rados.0.smithi155.stdout:371: done (4 left) 2024-04-19T01:05:26.629 INFO:tasks.rados.rados.0.smithi155.stdout:372: read oid 19 snap 38 2024-04-19T01:05:26.629 INFO:tasks.rados.rados.0.smithi155.stdout:372: expect deleted 2024-04-19T01:05:26.630 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 28 current snap is 42 2024-04-19T01:05:26.630 INFO:tasks.rados.rados.0.smithi155.stdout:373: seq_num 146 ranges {0=49152} 2024-04-19T01:05:26.630 INFO:tasks.rados.rados.0.smithi155.stdout:373: writing smithi15528762-28 from 0 to 49152 tid 1 2024-04-19T01:05:26.630 INFO:tasks.rados.rados.0.smithi155.stdout:374: snap_remove snap 41 2024-04-19T01:05:26.630 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-19T01:05:26.630 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:05:26.626+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:05:26.728 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:05:26.726+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:05:26.748 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:05:26.748 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:05:26.748 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-04-19T01:05:26.748 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:05:26.762 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-19T01:05:26.829 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:05:26.825+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:05:26.886 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:05:26.886 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:05:26.886 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-04-19T01:05:26.886 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-04-19T01:05:26.886 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:05:26.900 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-19T01:05:26.930 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:05:26.930+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:05:27.022 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:05:27.022 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-04-19T01:05:27.022 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-04-19T01:05:27.022 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-04-19T01:05:27.022 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.022 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.0s3 47/31 ECSubWrite(tid=32, reqid=client.4604.0:308, at_version=47'29, trim_to=0'0, roll_forward_to=47'28))", 2024-04-19T01:05:27.022 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:17.565539+0000", 2024-04-19T01:05:27.022 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 69.455271113999999, 2024-04-19T01:05:27.022 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.018109618000000001, 2024-04-19T01:05:27.022 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:27.022 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:27.023 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:27.023 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:27.023 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.023 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:27.023 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.565539+0000", 2024-04-19T01:05:27.023 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.023 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.023 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.023 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:27.023 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.565539+0000", 2024-04-19T01:05:27.023 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.023 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.023 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.024 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:27.024 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.565541+0000", 2024-04-19T01:05:27.024 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.04e-06 2024-04-19T01:05:27.024 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.024 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.024 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:27.024 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.566554+0000", 2024-04-19T01:05:27.024 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0010129029999999999 2024-04-19T01:05:27.024 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.024 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.024 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:27.024 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.566591+0000", 2024-04-19T01:05:27.025 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7061999999999999e-05 2024-04-19T01:05:27.025 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.025 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.025 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:27.025 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.566613+0000", 2024-04-19T01:05:27.025 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.145e-05 2024-04-19T01:05:27.025 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.025 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.026 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:27.026 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.566752+0000", 2024-04-19T01:05:27.026 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000138868 2024-04-19T01:05:27.026 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.026 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.026 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:27.026 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.566783+0000", 2024-04-19T01:05:27.026 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0772000000000003e-05 2024-04-19T01:05:27.026 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.027 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.027 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:27.027 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.583581+0000", 2024-04-19T01:05:27.027 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.016798124000000001 2024-04-19T01:05:27.027 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.027 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.027 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:27.027 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.583649+0000", 2024-04-19T01:05:27.027 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.8398999999999993e-05 2024-04-19T01:05:27.027 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.027 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.027 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.027 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.027 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.028 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as3 48/31 ECSubWrite(tid=71, reqid=client.4604.0:354, at_version=48'10, trim_to=0'0, roll_forward_to=34'8))", 2024-04-19T01:05:27.028 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.556344+0000", 2024-04-19T01:05:27.028 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 68.464466349000006, 2024-04-19T01:05:27.028 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.030027702, 2024-04-19T01:05:27.028 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:27.028 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:27.028 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:27.028 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:27.028 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.028 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:27.028 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.556344+0000", 2024-04-19T01:05:27.028 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.028 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.028 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.028 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:27.029 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.556344+0000", 2024-04-19T01:05:27.029 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.029 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.029 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.029 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:27.029 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.556345+0000", 2024-04-19T01:05:27.029 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1199999999999999e-07 2024-04-19T01:05:27.029 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.029 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.030 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:27.030 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.557285+0000", 2024-04-19T01:05:27.030 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00094096299999999996 2024-04-19T01:05:27.030 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.030 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.030 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:27.030 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.557335+0000", 2024-04-19T01:05:27.030 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9669000000000002e-05 2024-04-19T01:05:27.030 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.030 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.031 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:27.031 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.557358+0000", 2024-04-19T01:05:27.031 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3099999999999999e-05 2024-04-19T01:05:27.031 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.031 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.031 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:27.031 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.557490+0000", 2024-04-19T01:05:27.031 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013213400000000001 2024-04-19T01:05:27.031 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.031 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.031 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:27.031 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.557512+0000", 2024-04-19T01:05:27.031 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1603999999999998e-05 2024-04-19T01:05:27.031 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.032 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.032 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:27.032 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.586316+0000", 2024-04-19T01:05:27.032 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.028803714000000001 2024-04-19T01:05:27.032 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.032 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.032 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:27.032 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.586372+0000", 2024-04-19T01:05:27.032 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6205999999999997e-05 2024-04-19T01:05:27.032 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.032 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.032 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.032 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.032 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.033 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.4s3 81/31 ECSubWrite(tid=117, reqid=client.4604.0:618, at_version=81'41, trim_to=0'0, roll_forward_to=79'39))", 2024-04-19T01:05:27.033 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:51.536744+0000", 2024-04-19T01:05:27.033 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 35.484066792999997, 2024-04-19T01:05:27.033 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.018283939999999999, 2024-04-19T01:05:27.033 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:27.033 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:27.033 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:27.033 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:27.033 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.033 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:27.033 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.536744+0000", 2024-04-19T01:05:27.033 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.033 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.033 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.034 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:27.034 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.536744+0000", 2024-04-19T01:05:27.034 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.034 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.034 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.034 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:27.034 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.536744+0000", 2024-04-19T01:05:27.034 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6600000000000002e-07 2024-04-19T01:05:27.034 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.034 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.034 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:27.034 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.537473+0000", 2024-04-19T01:05:27.034 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00072840000000000003 2024-04-19T01:05:27.034 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.035 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.035 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:27.035 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.537505+0000", 2024-04-19T01:05:27.035 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2746000000000001e-05 2024-04-19T01:05:27.035 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.035 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.035 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:27.036 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.537536+0000", 2024-04-19T01:05:27.036 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0562000000000003e-05 2024-04-19T01:05:27.036 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.036 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.036 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:27.036 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.537733+0000", 2024-04-19T01:05:27.036 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019705800000000001 2024-04-19T01:05:27.036 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.036 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.036 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:27.036 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.537762+0000", 2024-04-19T01:05:27.036 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8958999999999998e-05 2024-04-19T01:05:27.036 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.037 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.038 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:27.038 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.554926+0000", 2024-04-19T01:05:27.038 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.017164338000000001 2024-04-19T01:05:27.038 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.038 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.038 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:27.038 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.555028+0000", 2024-04-19T01:05:27.038 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000101511 2024-04-19T01:05:27.038 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.038 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.038 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.038 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.038 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.038 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.2s5 86/31 ECSubWrite(tid=265, reqid=client.4604.0:655, at_version=86'35, trim_to=0'0, roll_forward_to=86'33))", 2024-04-19T01:05:27.039 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:56.580820+0000", 2024-04-19T01:05:27.039 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 30.439990765000001, 2024-04-19T01:05:27.039 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.030714183999999999, 2024-04-19T01:05:27.039 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:27.039 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:27.039 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:27.039 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:27.039 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.039 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:27.039 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.580820+0000", 2024-04-19T01:05:27.039 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.039 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.039 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.040 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:27.040 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.580820+0000", 2024-04-19T01:05:27.040 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.040 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.040 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.040 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:27.040 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.580823+0000", 2024-04-19T01:05:27.040 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8090000000000001e-06 2024-04-19T01:05:27.040 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.040 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.040 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:27.040 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.581340+0000", 2024-04-19T01:05:27.040 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00051707599999999999 2024-04-19T01:05:27.040 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.041 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.041 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:27.041 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.581384+0000", 2024-04-19T01:05:27.041 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4369999999999997e-05 2024-04-19T01:05:27.041 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.041 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.041 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:27.041 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.581409+0000", 2024-04-19T01:05:27.041 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4913999999999999e-05 2024-04-19T01:05:27.041 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.041 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.041 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:27.041 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.581604+0000", 2024-04-19T01:05:27.041 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019476500000000001 2024-04-19T01:05:27.042 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.042 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.042 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:27.042 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.581631+0000", 2024-04-19T01:05:27.042 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6979e-05 2024-04-19T01:05:27.042 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.042 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.042 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:27.043 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.611484+0000", 2024-04-19T01:05:27.043 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.029853356000000001 2024-04-19T01:05:27.043 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.043 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.043 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:27.043 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.611534+0000", 2024-04-19T01:05:27.043 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9914999999999999e-05 2024-04-19T01:05:27.043 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.043 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.043 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.043 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.043 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.043 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDPGPush(3.3s3 97/96 [PushOp(3:cbbd4e2f:::smithi15528762-23:head, version: 89'66, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:cbbd4e2f:::smithi15528762-23:head@89'66, size: 2129920, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-04-19T01:05:27.043 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:07.705295+0000", 2024-04-19T01:05:27.044 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 19.315515830999999, 2024-04-19T01:05:27.044 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.017631539000000002, 2024-04-19T01:05:27.044 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:27.044 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:27.044 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:27.044 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:27.044 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.044 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:27.044 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.705295+0000", 2024-04-19T01:05:27.044 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.044 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.044 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.044 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:27.044 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.705295+0000", 2024-04-19T01:05:27.045 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.045 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.045 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.045 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:27.045 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.705296+0000", 2024-04-19T01:05:27.045 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6470000000000001e-06 2024-04-19T01:05:27.045 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.045 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.045 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:27.045 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.705649+0000", 2024-04-19T01:05:27.045 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00035229599999999999 2024-04-19T01:05:27.045 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.045 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.046 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:27.046 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.705693+0000", 2024-04-19T01:05:27.046 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4459000000000002e-05 2024-04-19T01:05:27.046 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.046 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.046 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:27.046 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.705719+0000", 2024-04-19T01:05:27.046 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6242999999999999e-05 2024-04-19T01:05:27.046 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.046 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.046 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:27.046 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.705901+0000", 2024-04-19T01:05:27.046 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00018181999999999999 2024-04-19T01:05:27.047 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.047 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.047 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:27.047 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.722926+0000", 2024-04-19T01:05:27.047 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.017025074000000001 2024-04-19T01:05:27.047 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.047 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.047 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.047 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.047 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.047 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDPGPush(3.3s3 97/96 [PushOp(3:cbbd4e2f:::smithi15528762-23:head, version: 89'66, data_included: [262144~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:cbbd4e2f:::smithi15528762-23:head@89'66, size: 2129920, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-04-19T01:05:27.047 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:07.726191+0000", 2024-04-19T01:05:27.047 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 19.294619189999999, 2024-04-19T01:05:27.047 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.017422659, 2024-04-19T01:05:27.048 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:27.048 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:27.048 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:27.048 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:27.048 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.048 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:27.048 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.726191+0000", 2024-04-19T01:05:27.048 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.048 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.048 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.048 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:27.048 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.726191+0000", 2024-04-19T01:05:27.048 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.049 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.049 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.049 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:27.049 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.726193+0000", 2024-04-19T01:05:27.049 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.683e-06 2024-04-19T01:05:27.049 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.049 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.049 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:27.049 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.726548+0000", 2024-04-19T01:05:27.049 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00035464300000000003 2024-04-19T01:05:27.049 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.049 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.049 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:27.049 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.726592+0000", 2024-04-19T01:05:27.049 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4492e-05 2024-04-19T01:05:27.050 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.050 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.050 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:27.050 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.726621+0000", 2024-04-19T01:05:27.050 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.887e-05 2024-04-19T01:05:27.050 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.050 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.050 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:27.050 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.726799+0000", 2024-04-19T01:05:27.050 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017782299999999999 2024-04-19T01:05:27.050 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.050 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.050 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:27.051 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:07.743614+0000", 2024-04-19T01:05:27.051 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.016815147999999999 2024-04-19T01:05:27.051 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.051 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.051 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.051 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.051 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.051 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.3s3 102/98 ECSubWrite(tid=99, reqid=client.4604.0:745, at_version=102'79, trim_to=0'0, roll_forward_to=102'77))", 2024-04-19T01:05:27.051 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:12.604332+0000", 2024-04-19T01:05:27.051 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.416478569000001, 2024-04-19T01:05:27.051 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.022063187000000001, 2024-04-19T01:05:27.051 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:27.051 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:27.051 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:27.051 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:27.052 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.052 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:27.052 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.604332+0000", 2024-04-19T01:05:27.052 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.052 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.052 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.052 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:27.052 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.604332+0000", 2024-04-19T01:05:27.052 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.052 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.052 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.052 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:27.052 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.604334+0000", 2024-04-19T01:05:27.053 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.84e-06 2024-04-19T01:05:27.053 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.053 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.053 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:27.053 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.604500+0000", 2024-04-19T01:05:27.053 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016646200000000001 2024-04-19T01:05:27.053 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.053 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.053 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:27.053 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.604535+0000", 2024-04-19T01:05:27.053 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4708e-05 2024-04-19T01:05:27.053 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.053 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.053 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:27.053 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.604557+0000", 2024-04-19T01:05:27.054 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1750999999999999e-05 2024-04-19T01:05:27.054 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.054 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.054 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:27.054 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.604824+0000", 2024-04-19T01:05:27.054 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00026735399999999997 2024-04-19T01:05:27.054 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.054 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.054 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:27.054 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.604854+0000", 2024-04-19T01:05:27.054 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9716000000000001e-05 2024-04-19T01:05:27.054 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.054 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.054 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:27.055 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.626336+0000", 2024-04-19T01:05:27.055 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.021482283000000001 2024-04-19T01:05:27.055 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.055 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.055 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:27.055 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.626395+0000", 2024-04-19T01:05:27.055 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9073000000000002e-05 2024-04-19T01:05:27.055 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.055 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.055 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.055 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.055 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.055 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.0s3 105/31 ECSubWrite(tid=126, reqid=osd.1.0:126, at_version=105'116, trim_to=0'0, roll_forward_to=104'114))", 2024-04-19T01:05:27.055 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:15.668817+0000", 2024-04-19T01:05:27.056 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 11.351993593, 2024-04-19T01:05:27.056 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.016289637999999999, 2024-04-19T01:05:27.056 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:27.056 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:27.056 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:27.056 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:27.056 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.056 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:27.056 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:15.668817+0000", 2024-04-19T01:05:27.056 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.056 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.056 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.057 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:27.057 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:15.668817+0000", 2024-04-19T01:05:27.057 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.057 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.057 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.057 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:27.057 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:15.668818+0000", 2024-04-19T01:05:27.057 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3e-07 2024-04-19T01:05:27.057 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.057 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.057 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:27.057 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:15.668869+0000", 2024-04-19T01:05:27.057 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.1685999999999999e-05 2024-04-19T01:05:27.058 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.058 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.058 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:27.058 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:15.668917+0000", 2024-04-19T01:05:27.058 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7185999999999999e-05 2024-04-19T01:05:27.058 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.058 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.058 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:27.058 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:15.668943+0000", 2024-04-19T01:05:27.058 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6152999999999998e-05 2024-04-19T01:05:27.058 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.058 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.058 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:27.058 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:15.669134+0000", 2024-04-19T01:05:27.059 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019097 2024-04-19T01:05:27.059 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.059 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.059 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:27.059 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:15.669163+0000", 2024-04-19T01:05:27.059 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8999000000000001e-05 2024-04-19T01:05:27.059 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.059 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.059 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:27.059 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:15.685044+0000", 2024-04-19T01:05:27.059 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.015881802 2024-04-19T01:05:27.059 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.059 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.060 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:27.060 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:15.685107+0000", 2024-04-19T01:05:27.060 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2112000000000004e-05 2024-04-19T01:05:27.060 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.060 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.060 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.060 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.060 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.060 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.1s2 105/31 ECSubWrite(tid=65, reqid=client.4604.0:794, at_version=105'62, trim_to=0'0, roll_forward_to=105'60))", 2024-04-19T01:05:27.060 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:15.704614+0000", 2024-04-19T01:05:27.060 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 11.316196318999999, 2024-04-19T01:05:27.060 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.041431383000000002, 2024-04-19T01:05:27.060 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:27.061 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:27.061 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:27.061 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:27.061 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.061 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:27.061 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:15.704614+0000", 2024-04-19T01:05:27.061 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.061 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.061 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.061 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:27.061 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:15.704614+0000", 2024-04-19T01:05:27.061 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.061 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.061 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.062 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:27.062 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:15.704619+0000", 2024-04-19T01:05:27.062 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2480000000000004e-06 2024-04-19T01:05:27.062 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.062 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.062 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:27.062 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:15.708478+0000", 2024-04-19T01:05:27.062 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0038583979999999999 2024-04-19T01:05:27.062 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.062 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.062 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:27.062 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:15.708516+0000", 2024-04-19T01:05:27.062 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8618999999999997e-05 2024-04-19T01:05:27.063 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.063 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.063 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:27.063 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:15.708539+0000", 2024-04-19T01:05:27.063 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2497000000000002e-05 2024-04-19T01:05:27.063 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.063 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.063 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:27.063 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:15.708688+0000", 2024-04-19T01:05:27.063 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000149223 2024-04-19T01:05:27.063 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.063 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.063 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:27.063 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:15.708710+0000", 2024-04-19T01:05:27.063 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1752000000000001e-05 2024-04-19T01:05:27.064 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.064 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.064 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:27.064 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:15.745964+0000", 2024-04-19T01:05:27.064 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.037254469999999998 2024-04-19T01:05:27.064 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.064 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.064 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:27.064 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:15.746046+0000", 2024-04-19T01:05:27.064 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.1175999999999997e-05 2024-04-19T01:05:27.064 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.064 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.064 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.064 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.064 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.065 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as3 107/31 ECSubWrite(tid=232, reqid=client.4604.0:804, at_version=107'102, trim_to=0'0, roll_forward_to=105'100))", 2024-04-19T01:05:27.065 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:17.557620+0000", 2024-04-19T01:05:27.065 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 9.4631907670000004, 2024-04-19T01:05:27.065 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.025646450000000001, 2024-04-19T01:05:27.065 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:27.065 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:27.065 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:27.065 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:27.065 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.065 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:27.065 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.557620+0000", 2024-04-19T01:05:27.065 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.065 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.065 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.066 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:27.066 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.557620+0000", 2024-04-19T01:05:27.066 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.066 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.066 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.066 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:27.066 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.557621+0000", 2024-04-19T01:05:27.066 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4e-07 2024-04-19T01:05:27.066 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.066 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.066 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:27.066 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.558104+0000", 2024-04-19T01:05:27.066 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00048326499999999997 2024-04-19T01:05:27.066 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.067 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.067 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:27.067 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.558133+0000", 2024-04-19T01:05:27.067 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9544999999999999e-05 2024-04-19T01:05:27.067 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.067 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.067 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:27.067 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.558153+0000", 2024-04-19T01:05:27.067 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9885e-05 2024-04-19T01:05:27.067 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.067 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.067 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:27.067 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.558308+0000", 2024-04-19T01:05:27.067 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015419199999999999 2024-04-19T01:05:27.068 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.068 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.068 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:27.068 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.558339+0000", 2024-04-19T01:05:27.068 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1259e-05 2024-04-19T01:05:27.068 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.068 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.068 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:27.068 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.583169+0000", 2024-04-19T01:05:27.068 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.024830305 2024-04-19T01:05:27.068 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.068 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.068 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:27.068 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.583266+0000", 2024-04-19T01:05:27.069 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7158999999999996e-05 2024-04-19T01:05:27.069 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.069 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.069 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.069 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.069 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.069 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as3 107/31 ECSubWrite(tid=233, reqid=client.4604.0:805, at_version=107'103, trim_to=0'0, roll_forward_to=105'100))", 2024-04-19T01:05:27.069 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:17.568058+0000", 2024-04-19T01:05:27.069 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 9.4527527229999997, 2024-04-19T01:05:27.069 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.028936986000000001, 2024-04-19T01:05:27.069 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:27.069 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:27.069 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:27.070 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:27.070 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.070 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:27.070 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.568058+0000", 2024-04-19T01:05:27.070 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.070 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.070 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.070 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:27.070 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.568058+0000", 2024-04-19T01:05:27.070 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.070 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.070 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.071 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:27.071 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.568060+0000", 2024-04-19T01:05:27.071 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0559999999999999e-06 2024-04-19T01:05:27.071 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.071 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.071 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:27.071 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.568183+0000", 2024-04-19T01:05:27.071 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012334800000000001 2024-04-19T01:05:27.071 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.071 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.071 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:27.071 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.568216+0000", 2024-04-19T01:05:27.071 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2360000000000002e-05 2024-04-19T01:05:27.071 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.072 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.072 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:27.072 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.568232+0000", 2024-04-19T01:05:27.072 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6163000000000001e-05 2024-04-19T01:05:27.072 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.072 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.072 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:27.072 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.572977+0000", 2024-04-19T01:05:27.072 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0047450770000000003 2024-04-19T01:05:27.072 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.072 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.072 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:27.072 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.573009+0000", 2024-04-19T01:05:27.073 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1850000000000002e-05 2024-04-19T01:05:27.073 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.073 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.073 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:27.073 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.596958+0000", 2024-04-19T01:05:27.073 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.023949179000000001 2024-04-19T01:05:27.073 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.073 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.073 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:27.073 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.596995+0000", 2024-04-19T01:05:27.073 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6952999999999997e-05 2024-04-19T01:05:27.074 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.074 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.074 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.158 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.158 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.158 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as3 107/31 ECSubWrite(tid=234, reqid=client.4604.0:806, at_version=107'104, trim_to=0'0, roll_forward_to=105'100))", 2024-04-19T01:05:27.158 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:17.577536+0000", 2024-04-19T01:05:27.158 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 9.4432740850000005, 2024-04-19T01:05:27.159 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.019487529999999999, 2024-04-19T01:05:27.159 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:27.159 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:27.159 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:27.159 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:27.159 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.159 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:27.159 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.577536+0000", 2024-04-19T01:05:27.159 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.159 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.159 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.159 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:27.159 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.577536+0000", 2024-04-19T01:05:27.159 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.159 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.160 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.160 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:27.160 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.577539+0000", 2024-04-19T01:05:27.160 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1440000000000001e-06 2024-04-19T01:05:27.160 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.160 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.160 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:27.160 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.577938+0000", 2024-04-19T01:05:27.160 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00039962299999999999 2024-04-19T01:05:27.160 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.160 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.160 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:27.160 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.577976+0000", 2024-04-19T01:05:27.160 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7392999999999999e-05 2024-04-19T01:05:27.161 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.161 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.161 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:27.161 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.577999+0000", 2024-04-19T01:05:27.161 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3465999999999999e-05 2024-04-19T01:05:27.161 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.161 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.161 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:27.161 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.583347+0000", 2024-04-19T01:05:27.161 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.005348378 2024-04-19T01:05:27.161 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.161 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.161 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:27.161 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.583365+0000", 2024-04-19T01:05:27.162 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7754e-05 2024-04-19T01:05:27.162 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.162 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.162 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:27.162 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.597002+0000", 2024-04-19T01:05:27.162 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.013636921 2024-04-19T01:05:27.162 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.162 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.162 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:27.162 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.597024+0000", 2024-04-19T01:05:27.162 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1851000000000001e-05 2024-04-19T01:05:27.162 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.162 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.162 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.162 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.163 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.163 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.4s3 107/31 ECSubWrite(tid=196, reqid=client.4604.0:819, at_version=107'87, trim_to=0'0, roll_forward_to=102'85))", 2024-04-19T01:05:27.163 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:17.630405+0000", 2024-04-19T01:05:27.163 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 9.3904057049999992, 2024-04-19T01:05:27.163 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.033293358000000002, 2024-04-19T01:05:27.163 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:27.163 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:27.163 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:27.163 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:27.163 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.163 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:27.163 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.630405+0000", 2024-04-19T01:05:27.163 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.163 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.163 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.164 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:27.164 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.630405+0000", 2024-04-19T01:05:27.164 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.164 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.164 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.164 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:27.164 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.630408+0000", 2024-04-19T01:05:27.164 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7549999999999999e-06 2024-04-19T01:05:27.164 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.164 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.164 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:27.164 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.631340+0000", 2024-04-19T01:05:27.165 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00093285000000000004 2024-04-19T01:05:27.166 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.166 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.166 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:27.166 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.631384+0000", 2024-04-19T01:05:27.166 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3813999999999998e-05 2024-04-19T01:05:27.166 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.166 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.166 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:27.166 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.631409+0000", 2024-04-19T01:05:27.166 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5119000000000001e-05 2024-04-19T01:05:27.167 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.167 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.167 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:27.167 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.631570+0000", 2024-04-19T01:05:27.167 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016047899999999999 2024-04-19T01:05:27.167 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.167 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.167 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:27.167 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.631590+0000", 2024-04-19T01:05:27.167 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0245999999999999e-05 2024-04-19T01:05:27.167 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.167 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.167 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:27.167 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.663654+0000", 2024-04-19T01:05:27.167 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.032064174000000001 2024-04-19T01:05:27.168 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.168 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.168 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:27.168 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.663698+0000", 2024-04-19T01:05:27.168 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3921000000000002e-05 2024-04-19T01:05:27.168 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.168 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.168 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.168 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.168 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.168 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.3s3 116/115 ECSubWrite(tid=132, reqid=osd.15.0:132, at_version=116'105, trim_to=0'0, roll_forward_to=0'0))", 2024-04-19T01:05:27.168 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:26.557012+0000", 2024-04-19T01:05:27.168 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.46379900499999999, 2024-04-19T01:05:27.168 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.020672883999999999, 2024-04-19T01:05:27.169 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:27.169 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:27.169 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:27.169 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:27.169 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.169 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:27.169 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.557012+0000", 2024-04-19T01:05:27.169 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.169 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.169 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.169 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:27.169 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.557012+0000", 2024-04-19T01:05:27.169 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.169 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.170 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.170 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:27.170 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.557013+0000", 2024-04-19T01:05:27.170 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5099999999999999e-06 2024-04-19T01:05:27.170 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.170 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.170 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:27.170 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.557079+0000", 2024-04-19T01:05:27.170 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5488999999999996e-05 2024-04-19T01:05:27.170 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.170 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.170 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:27.170 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.557107+0000", 2024-04-19T01:05:27.170 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8833e-05 2024-04-19T01:05:27.171 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.171 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.171 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:27.171 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.557138+0000", 2024-04-19T01:05:27.171 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0562999999999999e-05 2024-04-19T01:05:27.171 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.171 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.171 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:27.171 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.557368+0000", 2024-04-19T01:05:27.171 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000229867 2024-04-19T01:05:27.171 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.171 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.171 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:27.171 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.557397+0000", 2024-04-19T01:05:27.171 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8903000000000001e-05 2024-04-19T01:05:27.172 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.172 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.172 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:27.172 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.577601+0000", 2024-04-19T01:05:27.172 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.020204247000000002 2024-04-19T01:05:27.172 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.172 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.172 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:27.172 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.577684+0000", 2024-04-19T01:05:27.172 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.3472000000000003e-05 2024-04-19T01:05:27.172 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.172 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.172 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.172 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.172 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.173 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.1s2 116/115 ECSubWrite(tid=75, reqid=client.4604.0:898, at_version=116'71, trim_to=0'0, roll_forward_to=0'0))", 2024-04-19T01:05:27.173 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:26.560008+0000", 2024-04-19T01:05:27.173 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.46080229900000003, 2024-04-19T01:05:27.173 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.035299738999999997, 2024-04-19T01:05:27.173 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:27.173 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:27.173 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:27.173 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:27.173 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.173 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:27.173 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.560008+0000", 2024-04-19T01:05:27.173 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.173 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.173 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.173 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:27.174 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.560008+0000", 2024-04-19T01:05:27.174 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.174 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.174 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.175 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:27.175 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.560010+0000", 2024-04-19T01:05:27.175 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2230000000000001e-06 2024-04-19T01:05:27.175 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.175 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.175 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:27.175 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.560238+0000", 2024-04-19T01:05:27.175 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000227607 2024-04-19T01:05:27.175 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.175 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.175 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:27.175 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.560281+0000", 2024-04-19T01:05:27.175 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3092e-05 2024-04-19T01:05:27.175 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.176 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.176 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:27.176 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.560343+0000", 2024-04-19T01:05:27.176 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2246e-05 2024-04-19T01:05:27.176 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.176 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.176 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:27.176 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.560546+0000", 2024-04-19T01:05:27.176 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00020250699999999999 2024-04-19T01:05:27.176 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.176 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.176 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:27.176 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.560574+0000", 2024-04-19T01:05:27.176 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7623e-05 2024-04-19T01:05:27.176 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.177 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.177 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:27.177 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.595242+0000", 2024-04-19T01:05:27.177 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.034668446999999998 2024-04-19T01:05:27.177 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.177 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.177 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:27.177 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.595308+0000", 2024-04-19T01:05:27.177 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5994000000000005e-05 2024-04-19T01:05:27.177 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.177 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.177 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.177 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.177 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.177 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.3s3 116/115 ECSubWrite(tid=133, reqid=osd.15.0:133, at_version=116'107, trim_to=0'0, roll_forward_to=0'0))", 2024-04-19T01:05:27.178 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:26.560520+0000", 2024-04-19T01:05:27.178 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.46029088299999998, 2024-04-19T01:05:27.178 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.021743205000000002, 2024-04-19T01:05:27.178 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:27.178 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:27.178 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:27.178 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:27.178 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.178 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:27.178 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.560520+0000", 2024-04-19T01:05:27.178 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.178 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.178 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.178 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:27.178 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.560520+0000", 2024-04-19T01:05:27.179 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.179 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.179 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.179 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:27.179 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.560521+0000", 2024-04-19T01:05:27.179 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4330000000000001e-06 2024-04-19T01:05:27.179 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.179 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.179 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:27.180 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.560602+0000", 2024-04-19T01:05:27.180 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.1398999999999998e-05 2024-04-19T01:05:27.180 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.180 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.180 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:27.180 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.560629+0000", 2024-04-19T01:05:27.180 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6916000000000001e-05 2024-04-19T01:05:27.180 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.180 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.180 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:27.180 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.560651+0000", 2024-04-19T01:05:27.180 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1367000000000001e-05 2024-04-19T01:05:27.180 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.180 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.181 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:27.181 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.560885+0000", 2024-04-19T01:05:27.181 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00023431299999999999 2024-04-19T01:05:27.181 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.181 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.181 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:27.181 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.560909+0000", 2024-04-19T01:05:27.181 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3748999999999999e-05 2024-04-19T01:05:27.181 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.181 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.181 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:27.181 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.582202+0000", 2024-04-19T01:05:27.181 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.021292999999999999 2024-04-19T01:05:27.181 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.182 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.182 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:27.182 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.582263+0000", 2024-04-19T01:05:27.182 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1027999999999999e-05 2024-04-19T01:05:27.182 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.182 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.182 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.182 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.182 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.182 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.4s3 116/115 ECSubWrite(tid=226, reqid=osd.10.0:226, at_version=116'93, trim_to=0'0, roll_forward_to=0'0))", 2024-04-19T01:05:27.182 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:26.565575+0000", 2024-04-19T01:05:27.182 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.45523514799999998, 2024-04-19T01:05:27.182 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.016639518999999998, 2024-04-19T01:05:27.182 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:27.182 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:27.183 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:27.183 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:27.183 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.183 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:27.183 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.565575+0000", 2024-04-19T01:05:27.183 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.183 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.183 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.183 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:27.183 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.565575+0000", 2024-04-19T01:05:27.183 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.183 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.183 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.183 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:27.184 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.565578+0000", 2024-04-19T01:05:27.184 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3e-06 2024-04-19T01:05:27.184 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.184 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.184 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:27.184 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.565626+0000", 2024-04-19T01:05:27.184 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8139000000000001e-05 2024-04-19T01:05:27.184 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.184 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.184 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:27.184 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.565669+0000", 2024-04-19T01:05:27.185 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3173999999999997e-05 2024-04-19T01:05:27.185 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.185 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.185 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:27.185 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.565690+0000", 2024-04-19T01:05:27.185 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1206e-05 2024-04-19T01:05:27.185 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.185 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.185 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:27.185 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.567219+0000", 2024-04-19T01:05:27.185 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0015290289999999999 2024-04-19T01:05:27.185 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.185 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.186 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:27.186 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.567239+0000", 2024-04-19T01:05:27.186 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9504000000000002e-05 2024-04-19T01:05:27.186 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.186 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.186 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:27.186 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.582133+0000", 2024-04-19T01:05:27.186 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.014894387 2024-04-19T01:05:27.186 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.186 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.186 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:27.186 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.582215+0000", 2024-04-19T01:05:27.186 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.1780000000000006e-05 2024-04-19T01:05:27.186 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.186 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.187 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.187 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.187 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.187 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.1s2 116/115 ECSubWrite(tid=76, reqid=client.4604.0:899, at_version=116'72, trim_to=0'0, roll_forward_to=0'0))", 2024-04-19T01:05:27.187 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:26.575504+0000", 2024-04-19T01:05:27.187 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.44530668800000001, 2024-04-19T01:05:27.187 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.028759377999999999, 2024-04-19T01:05:27.187 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:27.187 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:27.187 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:27.187 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:27.187 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.187 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:27.187 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.575504+0000", 2024-04-19T01:05:27.188 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.188 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.188 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.188 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:27.188 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.575504+0000", 2024-04-19T01:05:27.188 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.188 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.188 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.188 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:27.188 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.575506+0000", 2024-04-19T01:05:27.189 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.733e-06 2024-04-19T01:05:27.189 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.189 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.189 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:27.189 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.575825+0000", 2024-04-19T01:05:27.189 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00031976800000000001 2024-04-19T01:05:27.189 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.189 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.189 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:27.189 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.575864+0000", 2024-04-19T01:05:27.189 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8439999999999998e-05 2024-04-19T01:05:27.189 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.190 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.190 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:27.190 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.575885+0000", 2024-04-19T01:05:27.190 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.084e-05 2024-04-19T01:05:27.190 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.190 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.190 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:27.190 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.576288+0000", 2024-04-19T01:05:27.190 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00040385399999999999 2024-04-19T01:05:27.190 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.190 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.190 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:27.191 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.576309+0000", 2024-04-19T01:05:27.191 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0800000000000001e-05 2024-04-19T01:05:27.191 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.191 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.191 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:27.191 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.604222+0000", 2024-04-19T01:05:27.191 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.027912719999999999 2024-04-19T01:05:27.191 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.191 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.191 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:27.191 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.604263+0000", 2024-04-19T01:05:27.191 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1223000000000002e-05 2024-04-19T01:05:27.191 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.191 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.191 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.192 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.192 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.192 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.1s2 116/115 ECSubWrite(tid=77, reqid=client.4604.0:900, at_version=116'73, trim_to=0'0, roll_forward_to=0'0))", 2024-04-19T01:05:27.192 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:26.593022+0000", 2024-04-19T01:05:27.192 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.42778855100000002, 2024-04-19T01:05:27.192 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.019245337000000001, 2024-04-19T01:05:27.192 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:27.192 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:27.192 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:27.192 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:27.192 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.192 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:27.192 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.593022+0000", 2024-04-19T01:05:27.192 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.193 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.193 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.193 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:27.193 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.593022+0000", 2024-04-19T01:05:27.193 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.193 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.193 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.193 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:27.193 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.593023+0000", 2024-04-19T01:05:27.193 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.145e-06 2024-04-19T01:05:27.193 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.193 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.193 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:27.193 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.593382+0000", 2024-04-19T01:05:27.193 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00035875000000000002 2024-04-19T01:05:27.194 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.194 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.194 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:27.194 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.593402+0000", 2024-04-19T01:05:27.194 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0350999999999999e-05 2024-04-19T01:05:27.194 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.194 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.194 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:27.194 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.593411+0000", 2024-04-19T01:05:27.194 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2269999999999999e-06 2024-04-19T01:05:27.194 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.194 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.194 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:27.194 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.595558+0000", 2024-04-19T01:05:27.195 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0021465030000000001 2024-04-19T01:05:27.195 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.195 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.195 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:27.195 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.595576+0000", 2024-04-19T01:05:27.195 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8403999999999999e-05 2024-04-19T01:05:27.195 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.195 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.195 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:27.195 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.612150+0000", 2024-04-19T01:05:27.195 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.016574037999999999 2024-04-19T01:05:27.195 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.195 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.195 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:27.196 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.612267+0000", 2024-04-19T01:05:27.196 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000116919 2024-04-19T01:05:27.196 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.196 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.196 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.196 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.196 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.196 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.1s2 116/115 ECSubWrite(tid=78, reqid=client.4604.0:901, at_version=116'74, trim_to=0'0, roll_forward_to=0'0))", 2024-04-19T01:05:27.196 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:26.598054+0000", 2024-04-19T01:05:27.196 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.422756189, 2024-04-19T01:05:27.197 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.018563427, 2024-04-19T01:05:27.197 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:27.197 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:27.197 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:27.197 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:27.197 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.197 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:27.197 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.598054+0000", 2024-04-19T01:05:27.197 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.197 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.197 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.197 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:27.197 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.598054+0000", 2024-04-19T01:05:27.197 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.198 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.198 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.198 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:27.198 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.598056+0000", 2024-04-19T01:05:27.198 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.37e-06 2024-04-19T01:05:27.198 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.198 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.198 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:27.198 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.598215+0000", 2024-04-19T01:05:27.198 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000159758 2024-04-19T01:05:27.198 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.198 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.198 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:27.198 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.598248+0000", 2024-04-19T01:05:27.199 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2302999999999999e-05 2024-04-19T01:05:27.199 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.199 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.199 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:27.199 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.598265+0000", 2024-04-19T01:05:27.199 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7668000000000001e-05 2024-04-19T01:05:27.199 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.199 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.199 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:27.199 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.604548+0000", 2024-04-19T01:05:27.199 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0062824250000000003 2024-04-19T01:05:27.199 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.199 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.199 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:27.200 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.604558+0000", 2024-04-19T01:05:27.200 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0346999999999999e-05 2024-04-19T01:05:27.200 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.200 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.200 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:27.200 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.616572+0000", 2024-04-19T01:05:27.200 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.012013776 2024-04-19T01:05:27.200 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.200 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.200 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:27.200 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.616618+0000", 2024-04-19T01:05:27.200 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5779999999999999e-05 2024-04-19T01:05:27.200 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.200 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.201 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.201 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.201 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.201 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:05:27.202 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-19T01:05:27.203 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:05:27.200+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:05:27.303 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:05:27.301+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:05:27.371 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:05:27.371 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:05:27.371 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-04-19T01:05:27.371 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:05:27.385 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-19T01:05:27.433 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:05:27.432+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:05:27.503 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:05:27.503 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:05:27.503 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-04-19T01:05:27.503 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-04-19T01:05:27.503 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:05:27.504 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:05:27.506+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:05:27.519 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-19T01:05:27.606 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:05:27.604+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:05:27.607 INFO:tasks.rados.rados.0.smithi155.stdout:374: done (6 left) 2024-04-19T01:05:27.607 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 49 current snap is 42 2024-04-19T01:05:27.607 INFO:tasks.rados.rados.0.smithi155.stdout:375: seq_num 147 ranges {819200=65536} 2024-04-19T01:05:27.615 INFO:tasks.rados.rados.0.smithi155.stdout:375: writing smithi15528762-49 from 819200 to 884736 tid 1 2024-04-19T01:05:27.615 INFO:tasks.rados.rados.0.smithi155.stdout:376: snap_create 2024-04-19T01:05:27.615 INFO:tasks.rados.rados.0.smithi155.stdout:365: expect (ObjNum 12 snap 0 seq_num 12) 2024-04-19T01:05:27.627 INFO:tasks.rados.rados.0.smithi155.stdout:367: finishing rollback tid 1 to smithi15528762-45 2024-04-19T01:05:27.627 INFO:tasks.rados.rados.0.smithi155.stdout:369: finishing rollback tid 0 to smithi15528762-5 2024-04-19T01:05:27.628 INFO:tasks.rados.rados.0.smithi155.stdout:373: finishing write tid 1 to smithi15528762-28 2024-04-19T01:05:27.628 INFO:tasks.rados.rados.0.smithi155.stdout:367: finishing rollback tid 2 to smithi15528762-45 2024-04-19T01:05:27.628 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 45 v 73 (ObjNum 85 snap 14 seq_num 85) dirty exists 2024-04-19T01:05:27.628 INFO:tasks.rados.rados.0.smithi155.stdout:373: finishing write tid 2 to smithi15528762-28 2024-04-19T01:05:27.628 INFO:tasks.rados.rados.0.smithi155.stdout:366: finishing write tid 1 to smithi15528762-41 2024-04-19T01:05:27.628 INFO:tasks.rados.rados.0.smithi155.stdout:366: finishing write tid 2 to smithi15528762-41 2024-04-19T01:05:27.628 INFO:tasks.rados.rados.0.smithi155.stdout:366: finishing write tid 3 to smithi15528762-41 2024-04-19T01:05:27.628 INFO:tasks.rados.rados.0.smithi155.stdout:373: finishing write tid 3 to smithi15528762-28 2024-04-19T01:05:27.628 INFO:tasks.rados.rados.0.smithi155.stdout:373: oid 28 updating version 0 to 85 2024-04-19T01:05:27.628 INFO:tasks.rados.rados.0.smithi155.stdout:373: oid 28 updating version 85 to 86 2024-04-19T01:05:27.628 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 28 v 86 (ObjNum 146 snap 42 seq_num 146) dirty exists 2024-04-19T01:05:27.628 INFO:tasks.rados.rados.0.smithi155.stdout:373: left oid 28 (ObjNum 146 snap 42 seq_num 146) 2024-04-19T01:05:27.628 INFO:tasks.rados.rados.0.smithi155.stdout:369: finishing rollback tid 1 to smithi15528762-5 2024-04-19T01:05:27.628 INFO:tasks.rados.rados.0.smithi155.stdout:369: finishing rollback tid 2 to smithi15528762-5 2024-04-19T01:05:27.629 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 5 v 126 (ObjNum 72 snap 8 seq_num 72) dirty exists 2024-04-19T01:05:27.629 INFO:tasks.rados.rados.0.smithi155.stdout:366: finishing write tid 4 to smithi15528762-41 2024-04-19T01:05:27.629 INFO:tasks.rados.rados.0.smithi155.stdout:366: finishing write tid 5 to smithi15528762-41 2024-04-19T01:05:27.629 INFO:tasks.rados.rados.0.smithi155.stdout:366: finishing write tid 6 to smithi15528762-41 2024-04-19T01:05:27.629 INFO:tasks.rados.rados.0.smithi155.stdout:366: oid 41 updating version 0 to 75 2024-04-19T01:05:27.629 INFO:tasks.rados.rados.0.smithi155.stdout:366: oid 41 version 75 is already newer than 71 2024-04-19T01:05:27.629 INFO:tasks.rados.rados.0.smithi155.stdout:366: oid 41 version 75 is already newer than 72 2024-04-19T01:05:27.629 INFO:tasks.rados.rados.0.smithi155.stdout:366: oid 41 version 75 is already newer than 74 2024-04-19T01:05:27.629 INFO:tasks.rados.rados.0.smithi155.stdout:366: oid 41 version 75 is already newer than 73 2024-04-19T01:05:27.629 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 41 v 75 (ObjNum 142 snap 42 seq_num 142) dirty exists 2024-04-19T01:05:27.629 INFO:tasks.rados.rados.0.smithi155.stdout:366: left oid 41 (ObjNum 142 snap 42 seq_num 142) 2024-04-19T01:05:27.629 INFO:tasks.rados.rados.0.smithi155.stdout:375: finishing write tid 1 to smithi15528762-49 2024-04-19T01:05:27.634 INFO:tasks.rados.rados.0.smithi155.stdout:375: finishing write tid 2 to smithi15528762-49 2024-04-19T01:05:27.638 INFO:tasks.rados.rados.0.smithi155.stdout:375: finishing write tid 3 to smithi15528762-49 2024-04-19T01:05:27.638 INFO:tasks.rados.rados.0.smithi155.stdout:375: oid 49 updating version 0 to 222 2024-04-19T01:05:27.638 INFO:tasks.rados.rados.0.smithi155.stdout:375: oid 49 version 222 is already newer than 221 2024-04-19T01:05:27.638 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 49 v 222 (ObjNum 147 snap 42 seq_num 147) dirty exists 2024-04-19T01:05:27.638 INFO:tasks.rados.rados.0.smithi155.stdout:375: left oid 49 (ObjNum 147 snap 42 seq_num 147) 2024-04-19T01:05:27.702 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:05:27.702 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-04-19T01:05:27.702 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-04-19T01:05:27.702 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-04-19T01:05:27.702 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.702 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 47/31 ECSubWrite(tid=61, reqid=client.4604.0:317, at_version=47'16, trim_to=0'0, roll_forward_to=45'14))", 2024-04-19T01:05:27.702 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:17.604486+0000", 2024-04-19T01:05:27.702 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 70.096324437999996, 2024-04-19T01:05:27.702 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.025312589, 2024-04-19T01:05:27.703 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:27.703 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:27.703 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:27.703 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:27.703 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.703 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:27.703 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.604486+0000", 2024-04-19T01:05:27.703 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.703 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.703 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.703 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:27.703 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.604486+0000", 2024-04-19T01:05:27.703 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.703 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.704 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.704 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:27.704 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.604486+0000", 2024-04-19T01:05:27.704 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.9999999999999997e-07 2024-04-19T01:05:27.704 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.704 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.704 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:27.704 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.604960+0000", 2024-04-19T01:05:27.704 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00047350300000000001 2024-04-19T01:05:27.704 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.704 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.704 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:27.704 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.605002+0000", 2024-04-19T01:05:27.704 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1949000000000001e-05 2024-04-19T01:05:27.704 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.705 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.705 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:27.705 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.605032+0000", 2024-04-19T01:05:27.705 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9865999999999998e-05 2024-04-19T01:05:27.705 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.705 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.705 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:27.705 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.605289+0000", 2024-04-19T01:05:27.705 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00025766500000000002 2024-04-19T01:05:27.705 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.705 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.705 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:27.705 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.605322+0000", 2024-04-19T01:05:27.705 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2845999999999997e-05 2024-04-19T01:05:27.706 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.706 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.706 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:27.706 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.629741+0000", 2024-04-19T01:05:27.706 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.024419335 2024-04-19T01:05:27.706 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.706 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.706 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:27.706 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.629798+0000", 2024-04-19T01:05:27.706 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6725000000000003e-05 2024-04-19T01:05:27.706 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.706 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.706 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.706 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.707 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.707 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 47/31 ECSubWrite(tid=62, reqid=client.4604.0:318, at_version=47'17, trim_to=0'0, roll_forward_to=45'14))", 2024-04-19T01:05:27.707 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:17.611532+0000", 2024-04-19T01:05:27.707 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 70.089277523999996, 2024-04-19T01:05:27.707 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.018327896, 2024-04-19T01:05:27.707 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:27.707 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:27.707 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:27.707 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:27.707 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.707 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:27.707 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.611532+0000", 2024-04-19T01:05:27.707 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.707 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.707 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.708 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:27.708 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.611532+0000", 2024-04-19T01:05:27.708 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.708 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.708 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.708 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:27.708 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.611534+0000", 2024-04-19T01:05:27.708 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.979e-06 2024-04-19T01:05:27.708 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.708 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.708 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:27.708 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.611863+0000", 2024-04-19T01:05:27.708 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00032853100000000002 2024-04-19T01:05:27.709 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.709 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.709 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:27.709 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.611907+0000", 2024-04-19T01:05:27.709 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4425000000000002e-05 2024-04-19T01:05:27.709 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.709 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.709 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:27.709 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.611935+0000", 2024-04-19T01:05:27.709 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7993000000000001e-05 2024-04-19T01:05:27.709 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.709 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.709 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:27.709 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.617071+0000", 2024-04-19T01:05:27.710 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0051354069999999998 2024-04-19T01:05:27.710 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.710 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.710 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:27.710 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.617091+0000", 2024-04-19T01:05:27.710 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0061999999999999e-05 2024-04-19T01:05:27.710 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.710 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.710 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:27.711 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.629827+0000", 2024-04-19T01:05:27.711 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.012735943 2024-04-19T01:05:27.711 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.711 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.711 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:27.711 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:17.629860+0000", 2024-04-19T01:05:27.711 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3556000000000002e-05 2024-04-19T01:05:27.711 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.711 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.711 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.711 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.711 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.711 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 48/31 ECSubWrite(tid=76, reqid=client.4604.0:345, at_version=48'22, trim_to=0'0, roll_forward_to=47'19))", 2024-04-19T01:05:27.711 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.537945+0000", 2024-04-19T01:05:27.712 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 69.162864548000002, 2024-04-19T01:05:27.712 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.034986066000000003, 2024-04-19T01:05:27.712 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:27.712 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:27.712 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:27.712 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:27.712 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.712 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:27.712 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.537945+0000", 2024-04-19T01:05:27.712 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.712 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.712 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.712 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:27.712 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.537945+0000", 2024-04-19T01:05:27.713 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.713 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.713 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.713 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:27.713 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.537948+0000", 2024-04-19T01:05:27.713 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6929999999999999e-06 2024-04-19T01:05:27.713 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.713 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.713 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:27.713 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.538485+0000", 2024-04-19T01:05:27.713 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00053661900000000001 2024-04-19T01:05:27.713 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.713 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.714 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:27.714 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.538530+0000", 2024-04-19T01:05:27.714 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5423000000000002e-05 2024-04-19T01:05:27.714 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.714 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.714 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:27.714 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.538558+0000", 2024-04-19T01:05:27.714 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7569999999999999e-05 2024-04-19T01:05:27.714 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.714 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.714 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:27.715 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.538778+0000", 2024-04-19T01:05:27.715 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000220478 2024-04-19T01:05:27.715 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.715 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.715 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:27.715 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.538806+0000", 2024-04-19T01:05:27.715 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8197000000000001e-05 2024-04-19T01:05:27.715 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.715 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.715 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:27.715 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.572854+0000", 2024-04-19T01:05:27.715 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.034047315000000002 2024-04-19T01:05:27.715 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.715 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.715 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:27.716 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.572931+0000", 2024-04-19T01:05:27.716 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7770999999999998e-05 2024-04-19T01:05:27.716 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.716 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.716 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.716 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.716 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.716 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 48/31 ECSubWrite(tid=78, reqid=client.4604.0:346, at_version=48'23, trim_to=0'0, roll_forward_to=48'21))", 2024-04-19T01:05:27.716 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.548797+0000", 2024-04-19T01:05:27.716 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 69.152012482999993, 2024-04-19T01:05:27.716 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.024207922, 2024-04-19T01:05:27.716 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:27.716 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:27.717 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:27.717 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:27.717 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.717 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:27.717 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.548797+0000", 2024-04-19T01:05:27.717 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.717 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.717 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.717 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:27.717 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.548797+0000", 2024-04-19T01:05:27.717 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.717 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.717 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.717 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:27.718 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.548799+0000", 2024-04-19T01:05:27.718 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.604e-06 2024-04-19T01:05:27.718 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.718 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.718 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:27.718 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.548991+0000", 2024-04-19T01:05:27.718 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019153400000000001 2024-04-19T01:05:27.718 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.718 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.718 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:27.718 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.549011+0000", 2024-04-19T01:05:27.718 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0332000000000001e-05 2024-04-19T01:05:27.718 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.718 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.718 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:27.719 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.549025+0000", 2024-04-19T01:05:27.719 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4260000000000001e-05 2024-04-19T01:05:27.719 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.719 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.719 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:27.719 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.552893+0000", 2024-04-19T01:05:27.719 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0038674460000000001 2024-04-19T01:05:27.719 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.719 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.719 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:27.719 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.552906+0000", 2024-04-19T01:05:27.719 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3105000000000001e-05 2024-04-19T01:05:27.719 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.719 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.720 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:27.720 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.572950+0000", 2024-04-19T01:05:27.720 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.020043787 2024-04-19T01:05:27.720 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.720 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.720 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:27.720 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.573005+0000", 2024-04-19T01:05:27.720 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.5853999999999998e-05 2024-04-19T01:05:27.720 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.720 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.720 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.720 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.720 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.720 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 48/31 ECSubWrite(tid=79, reqid=client.4604.0:347, at_version=48'24, trim_to=0'0, roll_forward_to=48'21))", 2024-04-19T01:05:27.720 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.555132+0000", 2024-04-19T01:05:27.721 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 69.145678051000004, 2024-04-19T01:05:27.721 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.018366424999999999, 2024-04-19T01:05:27.721 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:27.721 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:27.721 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:27.721 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:27.721 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.721 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:27.721 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.555132+0000", 2024-04-19T01:05:27.721 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.721 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.721 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.721 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:27.721 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.555132+0000", 2024-04-19T01:05:27.721 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.722 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.722 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.722 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:27.722 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.555133+0000", 2024-04-19T01:05:27.722 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.42e-06 2024-04-19T01:05:27.722 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.722 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.722 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:27.722 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.555208+0000", 2024-04-19T01:05:27.722 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4532999999999999e-05 2024-04-19T01:05:27.722 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.722 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.722 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:27.722 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.555254+0000", 2024-04-19T01:05:27.723 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6031e-05 2024-04-19T01:05:27.723 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.723 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.723 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:27.723 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.555268+0000", 2024-04-19T01:05:27.723 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4068e-05 2024-04-19T01:05:27.723 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.723 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.723 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:27.723 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.571711+0000", 2024-04-19T01:05:27.723 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.016442656999999999 2024-04-19T01:05:27.723 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.723 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.723 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:27.723 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.571720+0000", 2024-04-19T01:05:27.724 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1370000000000008e-06 2024-04-19T01:05:27.724 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.724 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.724 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:27.724 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.573461+0000", 2024-04-19T01:05:27.724 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0017408180000000001 2024-04-19T01:05:27.724 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.724 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.724 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:27.724 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.573498+0000", 2024-04-19T01:05:27.724 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7761e-05 2024-04-19T01:05:27.724 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.724 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.725 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.725 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.725 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.725 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as2 48/31 ECSubWrite(tid=71, reqid=client.4604.0:354, at_version=48'10, trim_to=0'0, roll_forward_to=34'8))", 2024-04-19T01:05:27.725 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.556963+0000", 2024-04-19T01:05:27.725 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 69.143847332999997, 2024-04-19T01:05:27.725 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.029734679, 2024-04-19T01:05:27.725 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:27.725 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:27.725 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:27.725 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:27.725 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.725 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:27.725 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.556963+0000", 2024-04-19T01:05:27.725 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.726 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.726 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.726 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:27.726 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.556963+0000", 2024-04-19T01:05:27.726 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.726 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.726 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.726 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:27.726 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.556963+0000", 2024-04-19T01:05:27.726 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5499999999999998e-07 2024-04-19T01:05:27.726 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.726 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.726 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:27.727 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.558128+0000", 2024-04-19T01:05:27.727 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011650180000000001 2024-04-19T01:05:27.727 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.727 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.727 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:27.727 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.558199+0000", 2024-04-19T01:05:27.727 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0630999999999995e-05 2024-04-19T01:05:27.727 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.727 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.727 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:27.727 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.558221+0000", 2024-04-19T01:05:27.727 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2546999999999999e-05 2024-04-19T01:05:27.727 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.727 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.727 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:27.728 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.558350+0000", 2024-04-19T01:05:27.728 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012838699999999999 2024-04-19T01:05:27.728 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.728 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.728 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:27.728 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.558370+0000", 2024-04-19T01:05:27.728 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0106999999999999e-05 2024-04-19T01:05:27.728 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.728 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.728 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:27.728 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.586643+0000", 2024-04-19T01:05:27.728 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.028273007999999999 2024-04-19T01:05:27.728 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.729 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.729 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:27.729 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.586697+0000", 2024-04-19T01:05:27.729 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4326000000000001e-05 2024-04-19T01:05:27.729 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.729 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.729 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.729 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.729 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.729 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.fs4 48/31 ECSubWrite(tid=80, reqid=client.4604.0:348, at_version=48'25, trim_to=0'0, roll_forward_to=48'21))", 2024-04-19T01:05:27.729 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.559574+0000", 2024-04-19T01:05:27.729 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 69.141236061000001, 2024-04-19T01:05:27.729 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.016989658000000001, 2024-04-19T01:05:27.729 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:27.730 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:27.730 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:27.730 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:27.730 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.730 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:27.730 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.559574+0000", 2024-04-19T01:05:27.730 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.730 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.730 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.730 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:27.730 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.559574+0000", 2024-04-19T01:05:27.730 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.731 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.731 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.731 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:27.731 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.559574+0000", 2024-04-19T01:05:27.731 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9400000000000001e-07 2024-04-19T01:05:27.731 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.731 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.731 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:27.731 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.559589+0000", 2024-04-19T01:05:27.731 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4873e-05 2024-04-19T01:05:27.731 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.731 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.731 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:27.732 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.559597+0000", 2024-04-19T01:05:27.732 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.1089999999999998e-06 2024-04-19T01:05:27.732 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.732 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.732 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:27.732 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.559605+0000", 2024-04-19T01:05:27.732 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.267e-06 2024-04-19T01:05:27.732 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.732 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.732 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:27.732 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.573085+0000", 2024-04-19T01:05:27.732 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.013480373 2024-04-19T01:05:27.732 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.733 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.733 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:27.733 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.573107+0000", 2024-04-19T01:05:27.733 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2031999999999998e-05 2024-04-19T01:05:27.734 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.734 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.734 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:27.734 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.576518+0000", 2024-04-19T01:05:27.734 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.003411458 2024-04-19T01:05:27.734 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.734 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.734 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:27.734 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.576564+0000", 2024-04-19T01:05:27.734 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5151999999999997e-05 2024-04-19T01:05:27.734 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.734 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.734 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.735 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.735 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.735 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.2s6 86/31 ECSubWrite(tid=265, reqid=client.4604.0:655, at_version=86'35, trim_to=0'0, roll_forward_to=86'33))", 2024-04-19T01:05:27.735 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:56.581157+0000", 2024-04-19T01:05:27.735 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 31.119653267, 2024-04-19T01:05:27.735 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.034472488000000003, 2024-04-19T01:05:27.735 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:27.735 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:27.735 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:27.735 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:27.735 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.735 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:27.735 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.581157+0000", 2024-04-19T01:05:27.735 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.736 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.736 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.736 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:27.736 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.581157+0000", 2024-04-19T01:05:27.736 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.736 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.736 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.736 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:27.736 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.581159+0000", 2024-04-19T01:05:27.736 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5770000000000001e-06 2024-04-19T01:05:27.736 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.736 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.736 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:27.736 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.581453+0000", 2024-04-19T01:05:27.737 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00029377899999999998 2024-04-19T01:05:27.737 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.737 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.737 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:27.737 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.581497+0000", 2024-04-19T01:05:27.737 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4053e-05 2024-04-19T01:05:27.737 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.737 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.737 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:27.737 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.581519+0000", 2024-04-19T01:05:27.737 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2090999999999999e-05 2024-04-19T01:05:27.737 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.737 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.737 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:27.738 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.581679+0000", 2024-04-19T01:05:27.738 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000159946 2024-04-19T01:05:27.738 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.738 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.738 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:27.738 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.581705+0000", 2024-04-19T01:05:27.738 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5462000000000001e-05 2024-04-19T01:05:27.738 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.738 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.738 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:27.739 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.615490+0000", 2024-04-19T01:05:27.739 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.033785002000000001 2024-04-19T01:05:27.739 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.739 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.739 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:27.739 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.615629+0000", 2024-04-19T01:05:27.739 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013957800000000001 2024-04-19T01:05:27.739 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.739 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.739 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.739 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.739 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.740 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.2s6 86/31 ECSubWrite(tid=266, reqid=client.4604.0:656, at_version=86'36, trim_to=0'0, roll_forward_to=86'33))", 2024-04-19T01:05:27.740 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:56.598018+0000", 2024-04-19T01:05:27.740 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 31.102792050000001, 2024-04-19T01:05:27.740 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.025308352999999999, 2024-04-19T01:05:27.740 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:27.740 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:27.740 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:27.740 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:27.740 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.740 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:27.740 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.598018+0000", 2024-04-19T01:05:27.740 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.740 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.741 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.741 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:27.741 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.598018+0000", 2024-04-19T01:05:27.741 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.741 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.741 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.741 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:27.741 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.598020+0000", 2024-04-19T01:05:27.741 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9700000000000002e-06 2024-04-19T01:05:27.741 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.741 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.741 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:27.741 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.598315+0000", 2024-04-19T01:05:27.741 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00029499300000000001 2024-04-19T01:05:27.742 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.742 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.742 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:27.742 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.598358+0000", 2024-04-19T01:05:27.742 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3599e-05 2024-04-19T01:05:27.742 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.742 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.742 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:27.742 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.598382+0000", 2024-04-19T01:05:27.742 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3051999999999999e-05 2024-04-19T01:05:27.742 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.742 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.742 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:27.742 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.601433+0000", 2024-04-19T01:05:27.743 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00305128 2024-04-19T01:05:27.743 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.743 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.743 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:27.743 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.601460+0000", 2024-04-19T01:05:27.743 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7327000000000001e-05 2024-04-19T01:05:27.743 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.743 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.743 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:27.743 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.623242+0000", 2024-04-19T01:05:27.743 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.021782341 2024-04-19T01:05:27.744 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.744 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.744 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:27.744 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.623326+0000", 2024-04-19T01:05:27.744 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.3790999999999998e-05 2024-04-19T01:05:27.744 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.744 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.744 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.744 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.744 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.744 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as2 107/31 ECSubWrite(tid=232, reqid=client.4604.0:804, at_version=107'102, trim_to=0'0, roll_forward_to=105'100))", 2024-04-19T01:05:27.744 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:17.557846+0000", 2024-04-19T01:05:27.744 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.142964057, 2024-04-19T01:05:27.745 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.023289740999999999, 2024-04-19T01:05:27.745 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:27.745 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:27.745 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:27.745 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:27.745 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.745 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:27.745 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.557846+0000", 2024-04-19T01:05:27.745 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.745 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.745 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.745 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:27.745 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.557846+0000", 2024-04-19T01:05:27.745 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.746 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.746 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.746 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:27.746 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.557847+0000", 2024-04-19T01:05:27.746 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4000000000000001e-07 2024-04-19T01:05:27.746 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.747 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.747 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:27.747 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.558435+0000", 2024-04-19T01:05:27.747 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00058847700000000001 2024-04-19T01:05:27.747 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.747 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.747 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:27.747 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.558476+0000", 2024-04-19T01:05:27.747 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0543000000000001e-05 2024-04-19T01:05:27.747 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.747 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.747 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:27.747 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.558502+0000", 2024-04-19T01:05:27.748 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6409000000000001e-05 2024-04-19T01:05:27.748 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.748 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.748 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:27.748 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.558639+0000", 2024-04-19T01:05:27.748 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013686500000000001 2024-04-19T01:05:27.748 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.748 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.748 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:27.748 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.558659+0000", 2024-04-19T01:05:27.748 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0573000000000001e-05 2024-04-19T01:05:27.749 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.749 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.749 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:27.749 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.581074+0000", 2024-04-19T01:05:27.749 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.022414508999999999 2024-04-19T01:05:27.749 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.749 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.749 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:27.749 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.581136+0000", 2024-04-19T01:05:27.750 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1724999999999996e-05 2024-04-19T01:05:27.750 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.750 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.750 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.750 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.750 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.750 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as2 107/31 ECSubWrite(tid=233, reqid=client.4604.0:805, at_version=107'103, trim_to=0'0, roll_forward_to=105'100))", 2024-04-19T01:05:27.750 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:17.568142+0000", 2024-04-19T01:05:27.750 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.132668249, 2024-04-19T01:05:27.750 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.032828224000000003, 2024-04-19T01:05:27.750 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:27.750 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:27.751 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:27.751 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:27.751 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.751 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:27.751 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.568142+0000", 2024-04-19T01:05:27.751 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.751 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.751 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.751 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:27.751 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.568142+0000", 2024-04-19T01:05:27.751 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.752 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.752 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.752 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:27.752 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.568144+0000", 2024-04-19T01:05:27.752 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1459999999999999e-06 2024-04-19T01:05:27.752 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.752 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.752 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:27.752 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.568479+0000", 2024-04-19T01:05:27.752 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00033551499999999999 2024-04-19T01:05:27.753 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.753 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.753 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:27.753 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.568509+0000", 2024-04-19T01:05:27.753 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9586000000000001e-05 2024-04-19T01:05:27.753 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.753 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.753 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:27.753 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.568529+0000", 2024-04-19T01:05:27.753 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9979999999999998e-05 2024-04-19T01:05:27.753 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.753 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.753 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:27.754 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.571753+0000", 2024-04-19T01:05:27.754 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.003224106 2024-04-19T01:05:27.754 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.754 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.754 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:27.754 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.571773+0000", 2024-04-19T01:05:27.754 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9553000000000001e-05 2024-04-19T01:05:27.754 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.754 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.754 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:27.754 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.600909+0000", 2024-04-19T01:05:27.754 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.029136351000000001 2024-04-19T01:05:27.754 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.755 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.755 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:27.755 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.600970+0000", 2024-04-19T01:05:27.755 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0986999999999997e-05 2024-04-19T01:05:27.755 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.755 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.755 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.755 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.755 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.755 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as2 107/31 ECSubWrite(tid=234, reqid=client.4604.0:806, at_version=107'104, trim_to=0'0, roll_forward_to=105'100))", 2024-04-19T01:05:27.755 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:17.577440+0000", 2024-04-19T01:05:27.755 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.123369890999999, 2024-04-19T01:05:27.755 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.023569943999999999, 2024-04-19T01:05:27.755 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:27.756 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:27.756 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:27.756 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:27.756 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.756 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:27.756 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.577440+0000", 2024-04-19T01:05:27.756 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.756 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.756 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.756 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:27.756 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.577440+0000", 2024-04-19T01:05:27.756 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.756 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.756 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.756 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:27.757 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.577442+0000", 2024-04-19T01:05:27.757 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9970000000000001e-06 2024-04-19T01:05:27.757 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.757 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.757 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:27.757 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.577856+0000", 2024-04-19T01:05:27.757 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00041360300000000002 2024-04-19T01:05:27.757 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.757 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.757 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:27.757 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.577897+0000", 2024-04-19T01:05:27.757 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1715999999999998e-05 2024-04-19T01:05:27.757 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.757 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.758 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:27.758 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.577919+0000", 2024-04-19T01:05:27.758 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1763000000000002e-05 2024-04-19T01:05:27.758 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.758 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.758 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:27.758 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.581237+0000", 2024-04-19T01:05:27.758 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.003317768 2024-04-19T01:05:27.758 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.758 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.759 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:27.759 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.581268+0000", 2024-04-19T01:05:27.759 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1035999999999999e-05 2024-04-19T01:05:27.759 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.759 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.759 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:27.759 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.600981+0000", 2024-04-19T01:05:27.759 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.019712714999999999 2024-04-19T01:05:27.759 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.759 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.759 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:27.759 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.601010+0000", 2024-04-19T01:05:27.759 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9346e-05 2024-04-19T01:05:27.759 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.760 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.760 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.760 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.760 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.760 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as2 107/31 ECSubWrite(tid=235, reqid=client.4604.0:807, at_version=107'105, trim_to=0'0, roll_forward_to=105'100))", 2024-04-19T01:05:27.760 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:17.583279+0000", 2024-04-19T01:05:27.760 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.117530715999999, 2024-04-19T01:05:27.760 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.018263506999999998, 2024-04-19T01:05:27.760 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:27.761 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:27.761 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:27.761 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:27.761 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.761 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:27.761 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.583279+0000", 2024-04-19T01:05:27.761 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.761 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.761 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.761 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:27.761 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.583279+0000", 2024-04-19T01:05:27.761 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.762 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.762 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.762 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:27.762 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.583281+0000", 2024-04-19T01:05:27.762 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.77e-06 2024-04-19T01:05:27.762 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.762 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.762 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:27.762 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.583452+0000", 2024-04-19T01:05:27.762 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000170724 2024-04-19T01:05:27.762 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.762 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.762 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:27.763 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.583495+0000", 2024-04-19T01:05:27.763 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3340000000000002e-05 2024-04-19T01:05:27.763 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.763 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.763 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:27.763 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.583524+0000", 2024-04-19T01:05:27.763 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8992e-05 2024-04-19T01:05:27.763 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.763 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.763 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:27.763 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.599164+0000", 2024-04-19T01:05:27.763 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.015639904999999999 2024-04-19T01:05:27.764 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.764 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.764 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:27.764 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.599172+0000", 2024-04-19T01:05:27.764 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.2330000000000006e-06 2024-04-19T01:05:27.764 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.764 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.765 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:27.765 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.601502+0000", 2024-04-19T01:05:27.765 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0023294269999999998 2024-04-19T01:05:27.765 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.765 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.765 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:27.765 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.601543+0000", 2024-04-19T01:05:27.765 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1115999999999997e-05 2024-04-19T01:05:27.765 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.766 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.766 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.766 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.766 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.766 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.as2 107/31 ECSubWrite(tid=236, reqid=client.4604.0:808, at_version=107'106, trim_to=0'0, roll_forward_to=105'100))", 2024-04-19T01:05:27.766 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:17.585706+0000", 2024-04-19T01:05:27.766 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.115103589, 2024-04-19T01:05:27.766 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.016516552, 2024-04-19T01:05:27.766 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:27.766 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:27.766 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:27.766 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:27.766 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.767 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:27.767 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.585706+0000", 2024-04-19T01:05:27.767 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.767 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.767 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.767 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:27.767 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.585706+0000", 2024-04-19T01:05:27.767 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.767 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.767 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.767 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:27.767 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.585708+0000", 2024-04-19T01:05:27.767 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.553e-06 2024-04-19T01:05:27.768 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.768 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.768 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:27.768 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.585739+0000", 2024-04-19T01:05:27.768 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1426e-05 2024-04-19T01:05:27.768 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.768 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.768 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:27.768 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.585772+0000", 2024-04-19T01:05:27.768 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2836000000000002e-05 2024-04-19T01:05:27.768 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.768 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.768 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:27.768 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.585795+0000", 2024-04-19T01:05:27.768 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2350000000000001e-05 2024-04-19T01:05:27.769 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.769 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.769 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:27.769 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.600750+0000", 2024-04-19T01:05:27.769 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.014955466000000001 2024-04-19T01:05:27.769 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.769 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.769 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:27.769 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.600759+0000", 2024-04-19T01:05:27.769 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.0650000000000005e-06 2024-04-19T01:05:27.769 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.769 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.769 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:27.770 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.602179+0000", 2024-04-19T01:05:27.770 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0014199169999999999 2024-04-19T01:05:27.770 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.770 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.770 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:27.770 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.602223+0000", 2024-04-19T01:05:27.770 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3939000000000001e-05 2024-04-19T01:05:27.770 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.770 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.770 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.770 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.770 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.770 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.2s6 112/98 ECSubWrite(tid=322, reqid=client.4604.0:882, at_version=112'55, trim_to=0'0, roll_forward_to=109'53))", 2024-04-19T01:05:27.770 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:22.578357+0000", 2024-04-19T01:05:27.771 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.1224532480000002, 2024-04-19T01:05:27.771 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.021722352, 2024-04-19T01:05:27.771 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:27.771 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:27.771 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:27.771 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:27.771 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.771 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:27.771 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.578357+0000", 2024-04-19T01:05:27.771 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.771 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.771 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.771 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:27.771 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.578357+0000", 2024-04-19T01:05:27.772 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.772 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.772 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.772 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:27.772 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.578357+0000", 2024-04-19T01:05:27.772 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8400000000000001e-07 2024-04-19T01:05:27.772 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.772 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.772 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:27.772 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.578709+0000", 2024-04-19T01:05:27.772 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00035251899999999998 2024-04-19T01:05:27.772 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.772 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.772 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:27.773 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.578745+0000", 2024-04-19T01:05:27.773 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5784999999999997e-05 2024-04-19T01:05:27.773 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.773 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.773 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:27.773 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.578766+0000", 2024-04-19T01:05:27.773 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0905999999999999e-05 2024-04-19T01:05:27.773 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.773 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.773 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:27.773 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.578891+0000", 2024-04-19T01:05:27.773 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000124425 2024-04-19T01:05:27.773 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.774 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.774 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:27.774 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.578910+0000", 2024-04-19T01:05:27.774 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9284e-05 2024-04-19T01:05:27.774 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.774 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.774 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:27.774 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.600027+0000", 2024-04-19T01:05:27.774 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.021117670000000002 2024-04-19T01:05:27.774 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.774 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.774 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:27.774 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.600079+0000", 2024-04-19T01:05:27.775 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.1579000000000002e-05 2024-04-19T01:05:27.775 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.775 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.775 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.775 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.775 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.775 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.2s6 112/98 ECSubWrite(tid=323, reqid=client.4604.0:883, at_version=112'56, trim_to=0'0, roll_forward_to=109'53))", 2024-04-19T01:05:27.775 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:22.588740+0000", 2024-04-19T01:05:27.775 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.1120696399999996, 2024-04-19T01:05:27.775 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.023085296000000002, 2024-04-19T01:05:27.775 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:27.775 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:27.775 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:27.775 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:27.775 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.776 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:27.776 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.588740+0000", 2024-04-19T01:05:27.776 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.776 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.776 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.776 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:27.776 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.588740+0000", 2024-04-19T01:05:27.776 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.777 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.777 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.777 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:27.777 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.588742+0000", 2024-04-19T01:05:27.781 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9199999999999998e-06 2024-04-19T01:05:27.782 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.782 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.782 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:27.782 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.588862+0000", 2024-04-19T01:05:27.782 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000119633 2024-04-19T01:05:27.782 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.782 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.782 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:27.782 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.588903+0000", 2024-04-19T01:05:27.782 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1185e-05 2024-04-19T01:05:27.782 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.782 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.782 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:27.783 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.588926+0000", 2024-04-19T01:05:27.783 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3354e-05 2024-04-19T01:05:27.783 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.783 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.783 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:27.783 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.591970+0000", 2024-04-19T01:05:27.783 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.003043381 2024-04-19T01:05:27.783 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.783 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.783 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:27.783 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.591996+0000", 2024-04-19T01:05:27.783 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5771000000000001e-05 2024-04-19T01:05:27.783 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.783 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.784 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:27.784 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.611759+0000", 2024-04-19T01:05:27.784 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.019763339000000001 2024-04-19T01:05:27.784 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.784 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.784 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:27.784 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.611826+0000", 2024-04-19T01:05:27.784 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6712999999999996e-05 2024-04-19T01:05:27.784 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.784 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.784 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.784 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.784 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.784 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.2s6 112/98 ECSubWrite(tid=324, reqid=client.4604.0:884, at_version=112'57, trim_to=0'0, roll_forward_to=109'53))", 2024-04-19T01:05:27.784 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:22.595624+0000", 2024-04-19T01:05:27.785 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.1051863390000003, 2024-04-19T01:05:27.785 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.019955461000000001, 2024-04-19T01:05:27.785 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:27.785 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:27.785 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:27.785 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:27.785 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.785 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:27.785 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.595624+0000", 2024-04-19T01:05:27.785 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.785 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.785 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.785 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:27.785 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.595624+0000", 2024-04-19T01:05:27.785 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.786 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.786 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.786 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:27.786 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.595625+0000", 2024-04-19T01:05:27.786 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3939999999999999e-06 2024-04-19T01:05:27.786 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.786 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.786 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:27.786 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.595721+0000", 2024-04-19T01:05:27.786 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5725999999999998e-05 2024-04-19T01:05:27.786 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.786 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.786 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:27.786 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.595747+0000", 2024-04-19T01:05:27.787 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6054000000000002e-05 2024-04-19T01:05:27.787 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.787 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.787 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:27.787 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.595764+0000", 2024-04-19T01:05:27.787 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7601999999999999e-05 2024-04-19T01:05:27.787 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.787 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.787 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:27.787 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.600173+0000", 2024-04-19T01:05:27.787 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0044090630000000004 2024-04-19T01:05:27.787 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.787 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.787 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:27.787 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.600197+0000", 2024-04-19T01:05:27.788 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3045999999999999e-05 2024-04-19T01:05:27.788 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.788 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.788 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:27.788 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.615552+0000", 2024-04-19T01:05:27.788 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.015355381 2024-04-19T01:05:27.788 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.788 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.788 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:27.788 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.615579+0000", 2024-04-19T01:05:27.788 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7195e-05 2024-04-19T01:05:27.788 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.788 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.788 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.788 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.789 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.789 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.1s6 116/115 ECSubWrite(tid=76, reqid=client.4604.0:899, at_version=116'72, trim_to=0'0, roll_forward_to=0'0))", 2024-04-19T01:05:27.789 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:26.578555+0000", 2024-04-19T01:05:27.789 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 1.1222548450000001, 2024-04-19T01:05:27.789 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.039301402999999999, 2024-04-19T01:05:27.789 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:27.789 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:27.789 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:27.789 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:27.789 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.789 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:27.789 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.578555+0000", 2024-04-19T01:05:27.789 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.789 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.789 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.790 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:27.790 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.578555+0000", 2024-04-19T01:05:27.790 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.790 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.790 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.790 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:27.790 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.578562+0000", 2024-04-19T01:05:27.790 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7479999999999996e-06 2024-04-19T01:05:27.790 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.790 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.790 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:27.790 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.579378+0000", 2024-04-19T01:05:27.790 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00081650500000000003 2024-04-19T01:05:27.790 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.790 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.791 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:27.791 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.579403+0000", 2024-04-19T01:05:27.791 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4944999999999999e-05 2024-04-19T01:05:27.791 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.791 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.791 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:27.791 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.579420+0000", 2024-04-19T01:05:27.791 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6956000000000002e-05 2024-04-19T01:05:27.791 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.791 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.791 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:27.791 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.579845+0000", 2024-04-19T01:05:27.792 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00042475800000000001 2024-04-19T01:05:27.792 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.792 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.792 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:27.792 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.579864+0000", 2024-04-19T01:05:27.792 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9372e-05 2024-04-19T01:05:27.793 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.793 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.793 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:27.793 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.617819+0000", 2024-04-19T01:05:27.793 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.037954632000000002 2024-04-19T01:05:27.793 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.793 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.793 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:27.793 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.617857+0000", 2024-04-19T01:05:27.793 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7487000000000002e-05 2024-04-19T01:05:27.793 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.793 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.793 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.794 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.794 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.794 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.1s6 116/115 ECSubWrite(tid=77, reqid=client.4604.0:900, at_version=116'73, trim_to=0'0, roll_forward_to=0'0))", 2024-04-19T01:05:27.794 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:26.594453+0000", 2024-04-19T01:05:27.794 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 1.1063567000000001, 2024-04-19T01:05:27.794 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.023430791999999999, 2024-04-19T01:05:27.795 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:27.795 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:27.795 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:27.795 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:27.795 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.795 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:27.795 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.594453+0000", 2024-04-19T01:05:27.795 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.795 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.795 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.795 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:27.795 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.594453+0000", 2024-04-19T01:05:27.796 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.796 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.796 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.796 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:27.796 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.594455+0000", 2024-04-19T01:05:27.796 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5179999999999999e-06 2024-04-19T01:05:27.796 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.796 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.796 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:27.796 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.594705+0000", 2024-04-19T01:05:27.796 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00025013100000000001 2024-04-19T01:05:27.796 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.796 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.796 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:27.797 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.594738+0000", 2024-04-19T01:05:27.797 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3312000000000002e-05 2024-04-19T01:05:27.797 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.797 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.797 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:27.797 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.594756+0000", 2024-04-19T01:05:27.797 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7555e-05 2024-04-19T01:05:27.797 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.797 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.797 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:27.797 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.600956+0000", 2024-04-19T01:05:27.797 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0062003889999999997 2024-04-19T01:05:27.797 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.797 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.797 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:27.798 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.600965+0000", 2024-04-19T01:05:27.798 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1649999999999995e-06 2024-04-19T01:05:27.798 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.798 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.798 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:27.798 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.617862+0000", 2024-04-19T01:05:27.798 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.016896185000000001 2024-04-19T01:05:27.798 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.798 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.798 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:27.798 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.617884+0000", 2024-04-19T01:05:27.798 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2537000000000001e-05 2024-04-19T01:05:27.798 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.799 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.799 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.799 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.799 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.799 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.1s6 116/115 ECSubWrite(tid=78, reqid=client.4604.0:901, at_version=116'74, trim_to=0'0, roll_forward_to=0'0))", 2024-04-19T01:05:27.799 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:26.598164+0000", 2024-04-19T01:05:27.799 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 1.1026458379999999, 2024-04-19T01:05:27.799 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.020210834, 2024-04-19T01:05:27.799 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:27.799 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:27.799 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:27.799 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:27.799 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.799 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:27.800 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.598164+0000", 2024-04-19T01:05:27.800 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.800 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.800 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.800 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:27.800 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.598164+0000", 2024-04-19T01:05:27.800 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:27.800 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.800 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.800 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:27.800 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.598166+0000", 2024-04-19T01:05:27.800 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4190000000000001e-06 2024-04-19T01:05:27.800 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.801 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.801 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:27.801 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.598497+0000", 2024-04-19T01:05:27.801 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000331825 2024-04-19T01:05:27.801 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.801 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.801 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:27.802 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.598530+0000", 2024-04-19T01:05:27.802 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2670999999999999e-05 2024-04-19T01:05:27.802 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.802 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.802 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:27.802 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.598548+0000", 2024-04-19T01:05:27.802 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7804000000000001e-05 2024-04-19T01:05:27.802 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.802 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.802 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:27.802 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.608051+0000", 2024-04-19T01:05:27.802 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0095030119999999999 2024-04-19T01:05:27.802 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.803 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.803 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:27.803 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.608105+0000", 2024-04-19T01:05:27.803 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4095999999999998e-05 2024-04-19T01:05:27.803 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.803 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.803 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:27.803 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.618333+0000", 2024-04-19T01:05:27.803 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.010228004000000001 2024-04-19T01:05:27.803 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:27.803 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:27.803 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:27.804 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:26.618375+0000", 2024-04-19T01:05:27.804 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2002999999999997e-05 2024-04-19T01:05:27.804 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.804 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.804 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.804 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:27.804 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:27.804 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:05:27.806 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-19T01:05:27.806 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:05:27.804+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:05:27.906 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:05:27.906+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:05:27.974 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:05:27.974 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:05:27.975 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-04-19T01:05:27.977 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:05:27.988 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-19T01:05:28.007 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:05:28.005+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:05:28.106 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:05:28.106 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:05:28.106 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-04-19T01:05:28.106 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-04-19T01:05:28.106 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:05:28.108 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:05:28.108+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:05:28.123 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-19T01:05:28.209 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:05:28.208+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:05:28.248 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:05:28.248 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-04-19T01:05:28.248 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-04-19T01:05:28.248 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-04-19T01:05:28.248 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.248 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4260.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-19T01:05:28.248 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:03:50.620310+0000", 2024-04-19T01:05:28.249 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 97.626559811000007, 2024-04-19T01:05:28.249 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.15782495899999999, 2024-04-19T01:05:28.249 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:28.249 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:28.249 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:28.249 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:28.249 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4260", 2024-04-19T01:05:28.249 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/1127060541", 2024-04-19T01:05:28.249 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1 2024-04-19T01:05:28.249 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.249 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:28.250 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.250 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:28.250 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.620310+0000", 2024-04-19T01:05:28.250 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.250 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.250 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.250 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:28.250 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.620310+0000", 2024-04-19T01:05:28.250 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.250 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.250 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.250 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:28.250 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.620313+0000", 2024-04-19T01:05:28.251 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.757e-06 2024-04-19T01:05:28.251 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.251 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.251 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:28.251 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.620327+0000", 2024-04-19T01:05:28.251 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4029000000000001e-05 2024-04-19T01:05:28.251 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.251 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.251 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:28.251 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.620362+0000", 2024-04-19T01:05:28.251 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5201999999999999e-05 2024-04-19T01:05:28.251 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.251 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.251 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:28.252 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.620383+0000", 2024-04-19T01:05:28.252 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1049000000000001e-05 2024-04-19T01:05:28.252 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.252 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.252 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.252 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.620512+0000", 2024-04-19T01:05:28.252 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012820900000000001 2024-04-19T01:05:28.252 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.252 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.252 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for peered", 2024-04-19T01:05:28.252 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.620542+0000", 2024-04-19T01:05:28.252 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0264999999999999e-05 2024-04-19T01:05:28.252 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.253 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.253 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.253 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.774580+0000", 2024-04-19T01:05:28.253 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.15403776799999999 2024-04-19T01:05:28.253 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.253 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.253 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:28.253 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.775052+0000", 2024-04-19T01:05:28.253 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00047277300000000001 2024-04-19T01:05:28.253 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.253 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.253 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 5", 2024-04-19T01:05:28.253 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.775495+0000", 2024-04-19T01:05:28.254 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00044209400000000003 2024-04-19T01:05:28.254 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.254 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.254 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-04-19T01:05:28.254 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.777608+0000", 2024-04-19T01:05:28.254 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0021133649999999999 2024-04-19T01:05:28.254 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.254 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.254 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-04-19T01:05:28.254 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.777957+0000", 2024-04-19T01:05:28.254 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00034961399999999999 2024-04-19T01:05:28.254 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.255 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.255 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:28.255 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.778062+0000", 2024-04-19T01:05:28.255 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000104576 2024-04-19T01:05:28.255 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.255 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.255 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:28.255 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:03:50.778135+0000", 2024-04-19T01:05:28.255 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3257999999999996e-05 2024-04-19T01:05:28.255 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.255 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.255 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.255 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.256 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.256 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:360 3.8s0 3:11aca45f:::smithi15528762-45:head [append 0~573440] snapc 9=[7] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-04-19T01:05:28.256 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:19.532000+0000", 2024-04-19T01:05:28.256 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 68.714870259999998, 2024-04-19T01:05:28.256 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.042676382999999998, 2024-04-19T01:05:28.256 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:28.256 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:28.256 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:28.256 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:28.256 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:28.256 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:28.256 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 360 2024-04-19T01:05:28.256 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.257 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:28.257 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.257 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:28.257 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.532000+0000", 2024-04-19T01:05:28.257 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.257 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.257 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.257 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:28.257 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.532000+0000", 2024-04-19T01:05:28.257 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.257 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.257 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.257 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:28.257 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.532003+0000", 2024-04-19T01:05:28.258 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3189999999999999e-06 2024-04-19T01:05:28.258 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.258 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.258 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:28.258 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.533430+0000", 2024-04-19T01:05:28.258 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001427069 2024-04-19T01:05:28.258 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.258 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.258 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:28.259 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.533464+0000", 2024-04-19T01:05:28.259 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3708999999999999e-05 2024-04-19T01:05:28.260 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.261 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.261 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:28.261 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.533485+0000", 2024-04-19T01:05:28.261 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0596000000000001e-05 2024-04-19T01:05:28.261 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.261 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.261 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.261 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.535443+0000", 2024-04-19T01:05:28.261 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0019587290000000002 2024-04-19T01:05:28.261 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.261 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.261 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:28.261 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.535618+0000", 2024-04-19T01:05:28.261 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000174518 2024-04-19T01:05:28.262 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.262 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.262 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:28.262 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.537401+0000", 2024-04-19T01:05:28.262 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0017834350000000001 2024-04-19T01:05:28.262 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.262 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.262 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:28.262 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.550350+0000", 2024-04-19T01:05:28.262 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.012948602 2024-04-19T01:05:28.262 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.262 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.262 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:28.262 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.574521+0000", 2024-04-19T01:05:28.262 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.024171392999999999 2024-04-19T01:05:28.263 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.263 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.263 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:28.263 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.574676+0000", 2024-04-19T01:05:28.263 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015501299999999999 2024-04-19T01:05:28.263 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.263 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.263 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.263 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.263 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.263 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:361 3.8s0 3:11aca45f:::smithi15528762-45:head [append 0~425984] snapc 9=[7] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-04-19T01:05:28.263 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:19.540243+0000", 2024-04-19T01:05:28.263 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 68.706626893000006, 2024-04-19T01:05:28.263 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.038864792000000002, 2024-04-19T01:05:28.263 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:28.264 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:28.264 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:28.264 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:28.264 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:28.264 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:28.264 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 361 2024-04-19T01:05:28.264 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.264 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:28.264 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.264 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:28.264 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540243+0000", 2024-04-19T01:05:28.264 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.264 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.264 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.265 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:28.265 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540243+0000", 2024-04-19T01:05:28.265 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.265 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.265 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.265 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:28.265 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540247+0000", 2024-04-19T01:05:28.265 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.506e-06 2024-04-19T01:05:28.266 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.266 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.266 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:28.266 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.541191+0000", 2024-04-19T01:05:28.266 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00094451099999999998 2024-04-19T01:05:28.266 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.266 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.266 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:28.266 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.541227+0000", 2024-04-19T01:05:28.266 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6053000000000002e-05 2024-04-19T01:05:28.266 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.266 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.266 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:28.266 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.541250+0000", 2024-04-19T01:05:28.267 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2543000000000001e-05 2024-04-19T01:05:28.267 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.267 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.267 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.267 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.550582+0000", 2024-04-19T01:05:28.267 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0093325990000000004 2024-04-19T01:05:28.267 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.267 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.267 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:28.267 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.550838+0000", 2024-04-19T01:05:28.267 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00025546199999999997 2024-04-19T01:05:28.267 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.267 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.267 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:28.267 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.552905+0000", 2024-04-19T01:05:28.268 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0020670699999999998 2024-04-19T01:05:28.268 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.268 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.268 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:28.268 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.574896+0000", 2024-04-19T01:05:28.268 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.021991301000000001 2024-04-19T01:05:28.268 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.268 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.268 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:28.268 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.578996+0000", 2024-04-19T01:05:28.268 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0040997849999999999 2024-04-19T01:05:28.268 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.268 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.268 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:28.269 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.579108+0000", 2024-04-19T01:05:28.269 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000111962 2024-04-19T01:05:28.269 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.269 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.269 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.269 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.269 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.269 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.5s3 74/31 ECSubWrite(tid=87, reqid=client.4604.0:543, at_version=74'67, trim_to=0'0, roll_forward_to=72'65))", 2024-04-19T01:05:28.269 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:44.553593+0000", 2024-04-19T01:05:28.269 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 43.693277082999998, 2024-04-19T01:05:28.269 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.044047740000000002, 2024-04-19T01:05:28.269 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:28.269 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:28.269 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:28.269 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:28.270 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.270 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:28.271 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.553593+0000", 2024-04-19T01:05:28.271 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.271 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.271 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.271 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:28.271 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.553593+0000", 2024-04-19T01:05:28.271 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.271 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.271 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.272 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:28.272 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.553594+0000", 2024-04-19T01:05:28.272 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.327e-06 2024-04-19T01:05:28.272 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.272 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.272 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:28.272 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.554148+0000", 2024-04-19T01:05:28.272 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00055343600000000003 2024-04-19T01:05:28.272 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.272 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.272 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:28.273 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.554184+0000", 2024-04-19T01:05:28.273 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5865999999999998e-05 2024-04-19T01:05:28.273 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.273 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.273 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:28.273 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.554211+0000", 2024-04-19T01:05:28.273 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7203999999999999e-05 2024-04-19T01:05:28.273 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.273 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.273 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.273 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.554385+0000", 2024-04-19T01:05:28.273 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000174387 2024-04-19T01:05:28.273 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.273 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.273 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:28.274 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.554418+0000", 2024-04-19T01:05:28.274 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2502999999999997e-05 2024-04-19T01:05:28.274 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.274 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.274 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:28.274 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.597567+0000", 2024-04-19T01:05:28.274 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.043149383999999999 2024-04-19T01:05:28.274 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.274 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.274 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:28.274 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.597641+0000", 2024-04-19T01:05:28.274 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3633000000000005e-05 2024-04-19T01:05:28.274 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.274 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.275 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.275 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.275 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.275 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.5s3 74/31 ECSubWrite(tid=88, reqid=osd.8.0:88, at_version=74'69, trim_to=0'0, roll_forward_to=72'65))", 2024-04-19T01:05:28.275 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:44.559673+0000", 2024-04-19T01:05:28.275 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 43.687197656999999, 2024-04-19T01:05:28.275 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.038043423, 2024-04-19T01:05:28.275 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:28.275 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:28.275 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:28.275 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:28.275 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.275 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:28.275 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.559673+0000", 2024-04-19T01:05:28.276 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.276 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.276 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.276 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:28.276 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.559673+0000", 2024-04-19T01:05:28.276 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.276 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.276 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.276 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:28.276 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.559674+0000", 2024-04-19T01:05:28.276 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.77e-06 2024-04-19T01:05:28.276 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.276 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.276 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:28.277 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.559727+0000", 2024-04-19T01:05:28.277 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2552000000000001e-05 2024-04-19T01:05:28.277 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.277 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.277 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:28.277 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.559756+0000", 2024-04-19T01:05:28.277 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9254000000000001e-05 2024-04-19T01:05:28.277 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.277 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.277 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:28.277 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.559780+0000", 2024-04-19T01:05:28.277 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3856e-05 2024-04-19T01:05:28.277 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.277 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.277 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.278 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.571632+0000", 2024-04-19T01:05:28.278 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.011851733 2024-04-19T01:05:28.278 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.278 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.278 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:28.278 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.571658+0000", 2024-04-19T01:05:28.278 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6568999999999999e-05 2024-04-19T01:05:28.278 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.278 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.278 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:28.278 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.597662+0000", 2024-04-19T01:05:28.278 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.02600421 2024-04-19T01:05:28.278 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.278 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.278 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:28.279 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:44.597716+0000", 2024-04-19T01:05:28.279 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3479000000000001e-05 2024-04-19T01:05:28.279 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.279 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.279 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.279 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.279 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.279 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.5s3 81/31 ECSubWrite(tid=122, reqid=client.4604.0:626, at_version=81'96, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:05:28.279 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:51.593541+0000", 2024-04-19T01:05:28.279 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 36.653329296999999, 2024-04-19T01:05:28.279 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.061054634000000003, 2024-04-19T01:05:28.279 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:28.279 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:28.279 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:28.280 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:28.280 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.280 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:28.280 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.593541+0000", 2024-04-19T01:05:28.280 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.280 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.280 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.280 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:28.280 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.593541+0000", 2024-04-19T01:05:28.280 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.280 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.281 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.281 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:28.281 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.593541+0000", 2024-04-19T01:05:28.281 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9699999999999996e-07 2024-04-19T01:05:28.281 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.281 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.281 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:28.281 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.594429+0000", 2024-04-19T01:05:28.281 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000887318 2024-04-19T01:05:28.281 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.281 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.281 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:28.281 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.594456+0000", 2024-04-19T01:05:28.282 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7509e-05 2024-04-19T01:05:28.282 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.282 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.282 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:28.282 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.594472+0000", 2024-04-19T01:05:28.282 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6107e-05 2024-04-19T01:05:28.282 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.282 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.282 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.282 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.594609+0000", 2024-04-19T01:05:28.282 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013671899999999999 2024-04-19T01:05:28.282 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.282 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.282 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:28.283 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.594628+0000", 2024-04-19T01:05:28.283 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9123e-05 2024-04-19T01:05:28.283 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.283 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.283 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:28.283 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.654485+0000", 2024-04-19T01:05:28.283 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.059857150999999997 2024-04-19T01:05:28.283 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.283 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.283 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:28.283 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.654596+0000", 2024-04-19T01:05:28.283 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011021 2024-04-19T01:05:28.283 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.283 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.283 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.284 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.284 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.284 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.5s3 81/31 ECSubWrite(tid=123, reqid=client.4604.0:627, at_version=81'97, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:05:28.284 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:51.600033+0000", 2024-04-19T01:05:28.284 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 36.646836763000003, 2024-04-19T01:05:28.284 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.054639177999999997, 2024-04-19T01:05:28.284 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:28.284 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:28.284 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:28.284 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:28.284 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.284 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:28.284 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.600033+0000", 2024-04-19T01:05:28.284 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.285 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.285 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.285 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:28.285 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.600033+0000", 2024-04-19T01:05:28.285 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.285 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.285 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.285 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:28.285 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.600036+0000", 2024-04-19T01:05:28.285 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.419e-06 2024-04-19T01:05:28.285 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.285 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.285 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:28.285 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.600416+0000", 2024-04-19T01:05:28.286 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00038017299999999998 2024-04-19T01:05:28.286 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.286 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.286 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:28.286 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.600457+0000", 2024-04-19T01:05:28.286 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1075999999999998e-05 2024-04-19T01:05:28.286 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.286 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.286 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:28.286 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.600480+0000", 2024-04-19T01:05:28.286 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2976000000000002e-05 2024-04-19T01:05:28.286 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.286 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.286 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.287 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.609097+0000", 2024-04-19T01:05:28.287 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.008617408 2024-04-19T01:05:28.287 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.287 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.287 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:28.287 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.609118+0000", 2024-04-19T01:05:28.287 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0136000000000002e-05 2024-04-19T01:05:28.287 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.287 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.287 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:28.287 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.654605+0000", 2024-04-19T01:05:28.287 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.045486985000000001 2024-04-19T01:05:28.287 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.287 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.287 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:28.288 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.654673+0000", 2024-04-19T01:05:28.288 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.8004999999999997e-05 2024-04-19T01:05:28.288 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.288 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.288 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.288 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.288 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.288 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.5s3 81/31 ECSubWrite(tid=124, reqid=client.4604.0:628, at_version=81'98, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:05:28.288 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:51.605750+0000", 2024-04-19T01:05:28.288 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 36.641119729000003, 2024-04-19T01:05:28.288 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.048993015000000001, 2024-04-19T01:05:28.288 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:28.288 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:28.288 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:28.288 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:28.289 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.289 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:28.289 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.605750+0000", 2024-04-19T01:05:28.289 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.289 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.289 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.289 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:28.289 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.605750+0000", 2024-04-19T01:05:28.290 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.290 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.290 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.290 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:28.290 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.605752+0000", 2024-04-19T01:05:28.290 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9870000000000002e-06 2024-04-19T01:05:28.290 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.290 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.290 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:28.290 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.606125+0000", 2024-04-19T01:05:28.290 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00037219999999999999 2024-04-19T01:05:28.290 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.291 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.291 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:28.291 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.606160+0000", 2024-04-19T01:05:28.291 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5534999999999998e-05 2024-04-19T01:05:28.291 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.291 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.292 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:28.292 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.606182+0000", 2024-04-19T01:05:28.292 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1999000000000001e-05 2024-04-19T01:05:28.292 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.292 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.292 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.292 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.618898+0000", 2024-04-19T01:05:28.292 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.012715888 2024-04-19T01:05:28.292 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.292 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.292 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:28.292 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.618930+0000", 2024-04-19T01:05:28.292 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1712999999999999e-05 2024-04-19T01:05:28.292 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.293 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.293 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:28.293 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.654683+0000", 2024-04-19T01:05:28.293 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.035753339000000002 2024-04-19T01:05:28.293 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.293 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.293 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:28.293 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.654743+0000", 2024-04-19T01:05:28.293 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0353999999999998e-05 2024-04-19T01:05:28.293 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.293 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.293 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.293 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.293 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.293 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.5s3 81/31 ECSubWrite(tid=125, reqid=client.4604.0:629, at_version=81'99, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:05:28.294 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:51.609931+0000", 2024-04-19T01:05:28.294 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 36.636938807, 2024-04-19T01:05:28.294 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.044873181999999998, 2024-04-19T01:05:28.294 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:28.294 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:28.294 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:28.294 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:28.294 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.294 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:28.294 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.609931+0000", 2024-04-19T01:05:28.294 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.294 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.294 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.294 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:28.294 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.609931+0000", 2024-04-19T01:05:28.295 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.295 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.295 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.295 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:28.295 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.609933+0000", 2024-04-19T01:05:28.295 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.753e-06 2024-04-19T01:05:28.295 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.295 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.295 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:28.295 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.609982+0000", 2024-04-19T01:05:28.295 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9330999999999999e-05 2024-04-19T01:05:28.295 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.295 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.295 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:28.296 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.610031+0000", 2024-04-19T01:05:28.296 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8718999999999999e-05 2024-04-19T01:05:28.296 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.296 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.296 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:28.296 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.610048+0000", 2024-04-19T01:05:28.296 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6923e-05 2024-04-19T01:05:28.296 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.296 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.296 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.296 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.629621+0000", 2024-04-19T01:05:28.296 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.019572523000000001 2024-04-19T01:05:28.296 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.296 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.297 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:28.297 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.629650+0000", 2024-04-19T01:05:28.297 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9104000000000001e-05 2024-04-19T01:05:28.297 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.297 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.297 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:28.297 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.654749+0000", 2024-04-19T01:05:28.297 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.025098874 2024-04-19T01:05:28.297 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.297 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.297 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:28.297 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.654805+0000", 2024-04-19T01:05:28.297 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.5955000000000002e-05 2024-04-19T01:05:28.297 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.297 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.298 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.298 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.298 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.298 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.5s3 81/31 ECSubWrite(tid=126, reqid=client.4604.0:631, at_version=81'101, trim_to=0'0, roll_forward_to=79'94))", 2024-04-19T01:05:28.298 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:51.618408+0000", 2024-04-19T01:05:28.298 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 36.628462298999999, 2024-04-19T01:05:28.298 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.038972772000000003, 2024-04-19T01:05:28.298 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:28.298 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:28.298 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:28.298 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:28.298 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.298 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:28.298 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.618408+0000", 2024-04-19T01:05:28.299 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.299 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.299 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.299 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:28.299 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.618408+0000", 2024-04-19T01:05:28.299 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.299 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.299 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.299 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:28.299 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.618410+0000", 2024-04-19T01:05:28.299 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.863e-06 2024-04-19T01:05:28.299 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.299 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.299 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:28.299 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.619228+0000", 2024-04-19T01:05:28.300 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00081862099999999995 2024-04-19T01:05:28.300 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.300 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.300 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:28.300 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.619258+0000", 2024-04-19T01:05:28.300 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9805e-05 2024-04-19T01:05:28.300 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.300 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.300 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:28.300 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.619280+0000", 2024-04-19T01:05:28.300 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.158e-05 2024-04-19T01:05:28.300 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.300 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.300 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.300 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.630187+0000", 2024-04-19T01:05:28.301 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.010906911 2024-04-19T01:05:28.301 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.301 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.301 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:28.301 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.630202+0000", 2024-04-19T01:05:28.301 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.56e-05 2024-04-19T01:05:28.301 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.301 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.302 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:28.302 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.657322+0000", 2024-04-19T01:05:28.302 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.027119353999999998 2024-04-19T01:05:28.302 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.302 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.302 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:28.302 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:51.657381+0000", 2024-04-19T01:05:28.302 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9038e-05 2024-04-19T01:05:28.302 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.302 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.302 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.302 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.303 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.303 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:655 3.2s0 3:47ba51fe:::smithi15528762-10:head [append 0~770048] snapc 1d=[1d,1c,1b,14] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-04-19T01:05:28.303 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:56.576580+0000", 2024-04-19T01:05:28.303 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 31.670290452, 2024-04-19T01:05:28.303 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.045487739999999999, 2024-04-19T01:05:28.303 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:28.303 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:28.303 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:28.303 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:28.303 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:28.303 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:28.303 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 655 2024-04-19T01:05:28.303 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.303 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:28.304 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.304 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:28.304 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.576580+0000", 2024-04-19T01:05:28.304 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.304 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.304 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.304 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:28.304 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.576580+0000", 2024-04-19T01:05:28.304 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.304 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.304 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.304 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:28.304 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.576583+0000", 2024-04-19T01:05:28.304 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.055e-06 2024-04-19T01:05:28.304 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.305 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.305 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:28.305 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.577087+0000", 2024-04-19T01:05:28.305 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00050413300000000003 2024-04-19T01:05:28.306 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.306 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.306 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:28.306 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.577115+0000", 2024-04-19T01:05:28.306 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7668e-05 2024-04-19T01:05:28.306 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.306 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.306 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:28.306 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.577128+0000", 2024-04-19T01:05:28.306 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3454e-05 2024-04-19T01:05:28.306 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.306 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.306 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.306 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.577281+0000", 2024-04-19T01:05:28.307 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000152583 2024-04-19T01:05:28.307 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.307 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.307 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:28.307 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.577572+0000", 2024-04-19T01:05:28.307 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00029132099999999998 2024-04-19T01:05:28.307 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.307 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.307 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:28.307 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.580336+0000", 2024-04-19T01:05:28.307 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0027644039999999998 2024-04-19T01:05:28.307 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.308 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.309 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:28.309 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.614100+0000", 2024-04-19T01:05:28.309 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.033763537000000003 2024-04-19T01:05:28.309 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.309 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.309 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:28.310 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.621899+0000", 2024-04-19T01:05:28.310 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0077987109999999998 2024-04-19T01:05:28.310 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.310 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.310 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:28.310 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.622067+0000", 2024-04-19T01:05:28.310 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016887400000000001 2024-04-19T01:05:28.310 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.310 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.310 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.310 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.310 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.310 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.5s3 107/31 ECSubWrite(tid=213, reqid=client.4604.0:811, at_version=107'200, trim_to=0'0, roll_forward_to=106'199))", 2024-04-19T01:05:28.310 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:17.593245+0000", 2024-04-19T01:05:28.311 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.653624806, 2024-04-19T01:05:28.311 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.048862989000000003, 2024-04-19T01:05:28.311 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:28.311 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:28.311 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:28.311 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:28.311 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.311 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:28.311 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.593245+0000", 2024-04-19T01:05:28.311 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.311 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.311 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.311 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:28.312 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.593245+0000", 2024-04-19T01:05:28.312 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.312 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.312 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.312 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:28.312 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.593246+0000", 2024-04-19T01:05:28.312 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.27e-07 2024-04-19T01:05:28.312 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.312 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.312 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:28.312 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.594356+0000", 2024-04-19T01:05:28.312 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011102200000000001 2024-04-19T01:05:28.312 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.312 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.312 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:28.313 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.594385+0000", 2024-04-19T01:05:28.313 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8583000000000001e-05 2024-04-19T01:05:28.313 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.313 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.313 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:28.313 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.594410+0000", 2024-04-19T01:05:28.313 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5199999999999999e-05 2024-04-19T01:05:28.313 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.313 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.313 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.313 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.594549+0000", 2024-04-19T01:05:28.313 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013903400000000001 2024-04-19T01:05:28.313 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.313 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.314 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:28.314 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.594573+0000", 2024-04-19T01:05:28.314 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4428999999999999e-05 2024-04-19T01:05:28.314 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.314 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.314 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:28.314 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.638210+0000", 2024-04-19T01:05:28.314 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.043637189999999999 2024-04-19T01:05:28.314 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.314 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.314 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:28.314 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.642108+0000", 2024-04-19T01:05:28.314 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.003897906 2024-04-19T01:05:28.315 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.315 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.315 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.315 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.315 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.315 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.5s3 107/31 ECSubWrite(tid=214, reqid=client.4604.0:812, at_version=107'201, trim_to=0'0, roll_forward_to=106'199))", 2024-04-19T01:05:28.315 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:17.600427+0000", 2024-04-19T01:05:28.315 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.646443517, 2024-04-19T01:05:28.315 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.041765305000000003, 2024-04-19T01:05:28.315 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:28.315 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:28.315 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:28.315 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:28.315 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.316 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:28.316 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.600427+0000", 2024-04-19T01:05:28.316 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.316 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.316 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.316 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:28.316 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.600427+0000", 2024-04-19T01:05:28.316 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.316 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.316 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.316 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:28.316 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.600429+0000", 2024-04-19T01:05:28.316 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6000000000000001e-06 2024-04-19T01:05:28.316 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.316 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.317 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:28.317 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.601054+0000", 2024-04-19T01:05:28.317 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00062503000000000005 2024-04-19T01:05:28.317 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.317 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.317 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:28.317 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.601069+0000", 2024-04-19T01:05:28.317 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4914e-05 2024-04-19T01:05:28.317 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.317 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.317 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:28.317 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.601079+0000", 2024-04-19T01:05:28.317 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.8980000000000008e-06 2024-04-19T01:05:28.317 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.318 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.318 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.318 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.608141+0000", 2024-04-19T01:05:28.318 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0070621169999999997 2024-04-19T01:05:28.318 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.318 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.318 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:28.318 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.608165+0000", 2024-04-19T01:05:28.318 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3754e-05 2024-04-19T01:05:28.319 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.319 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.319 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:28.319 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.642124+0000", 2024-04-19T01:05:28.319 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.033959325999999998 2024-04-19T01:05:28.319 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.319 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.319 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:28.319 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.642192+0000", 2024-04-19T01:05:28.319 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7665999999999999e-05 2024-04-19T01:05:28.319 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.319 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.320 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.320 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.320 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.320 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:826 3.cs0 3:37e9fe95:::smithi15528762-36:head [append 0~737280] snapc 27=[27,25,24,21] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-19T01:05:28.320 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:17.633212+0000", 2024-04-19T01:05:28.320 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.613658594, 2024-04-19T01:05:28.320 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.052979892000000001, 2024-04-19T01:05:28.320 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:28.320 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:28.321 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:28.321 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:28.321 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:28.321 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:28.321 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 826 2024-04-19T01:05:28.321 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.321 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:28.321 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.321 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:28.321 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.633212+0000", 2024-04-19T01:05:28.321 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.321 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.321 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.321 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:28.322 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.633212+0000", 2024-04-19T01:05:28.322 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.322 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.322 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.322 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:28.322 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.633214+0000", 2024-04-19T01:05:28.322 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7360000000000001e-06 2024-04-19T01:05:28.322 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.322 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.322 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:28.322 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.633876+0000", 2024-04-19T01:05:28.322 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00066133399999999997 2024-04-19T01:05:28.322 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.322 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.322 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:28.323 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.633900+0000", 2024-04-19T01:05:28.323 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4530999999999999e-05 2024-04-19T01:05:28.323 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.323 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.323 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:28.323 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.633913+0000", 2024-04-19T01:05:28.323 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3029e-05 2024-04-19T01:05:28.323 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.323 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.323 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.323 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.634077+0000", 2024-04-19T01:05:28.323 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016385 2024-04-19T01:05:28.323 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.323 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.324 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:28.324 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.634356+0000", 2024-04-19T01:05:28.324 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00027934100000000001 2024-04-19T01:05:28.324 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.324 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.324 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:28.324 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.637628+0000", 2024-04-19T01:05:28.324 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0032716540000000001 2024-04-19T01:05:28.324 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.324 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.324 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:28.324 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.664620+0000", 2024-04-19T01:05:28.324 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.026991780999999999 2024-04-19T01:05:28.324 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.324 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.325 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:28.325 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.686080+0000", 2024-04-19T01:05:28.325 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.021460554 2024-04-19T01:05:28.325 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.325 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.325 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:28.325 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.686191+0000", 2024-04-19T01:05:28.325 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000111082 2024-04-19T01:05:28.325 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.325 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.325 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.325 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.325 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.325 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:827 3.cs0 3:37e9fe95:::smithi15528762-36:head [append 0~770048] snapc 27=[27,25,24,21] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-19T01:05:28.326 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:17.640005+0000", 2024-04-19T01:05:28.326 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.606865472999999, 2024-04-19T01:05:28.326 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.046704448000000003, 2024-04-19T01:05:28.326 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:28.326 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:28.326 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:28.326 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:28.326 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:28.326 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:28.326 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 827 2024-04-19T01:05:28.326 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.326 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:28.326 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.327 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:28.327 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.640005+0000", 2024-04-19T01:05:28.327 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.327 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.327 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.327 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:28.327 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.640005+0000", 2024-04-19T01:05:28.327 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.327 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.327 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.327 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:28.327 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.640008+0000", 2024-04-19T01:05:28.327 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8540000000000001e-06 2024-04-19T01:05:28.327 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.328 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.328 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:28.328 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.640970+0000", 2024-04-19T01:05:28.328 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00096262400000000003 2024-04-19T01:05:28.328 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.328 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.328 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:28.328 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.641004+0000", 2024-04-19T01:05:28.328 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4205000000000002e-05 2024-04-19T01:05:28.328 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.328 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.328 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:28.328 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.641025+0000", 2024-04-19T01:05:28.328 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.056e-05 2024-04-19T01:05:28.329 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.329 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.329 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.329 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.654965+0000", 2024-04-19T01:05:28.329 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.013940368999999999 2024-04-19T01:05:28.329 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.329 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.329 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:28.329 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.655189+0000", 2024-04-19T01:05:28.329 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000223284 2024-04-19T01:05:28.329 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.329 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.329 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:28.330 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.656459+0000", 2024-04-19T01:05:28.330 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0012706289999999999 2024-04-19T01:05:28.330 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.330 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.330 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:28.330 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.675616+0000", 2024-04-19T01:05:28.330 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.019157230000000001 2024-04-19T01:05:28.331 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.331 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.331 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:28.331 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.686606+0000", 2024-04-19T01:05:28.331 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.010989597 2024-04-19T01:05:28.331 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.331 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.331 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:28.331 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.686709+0000", 2024-04-19T01:05:28.331 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000103096 2024-04-19T01:05:28.331 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.331 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.331 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.332 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.332 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.332 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:828 3.cs0 3:37e9fe95:::smithi15528762-36:head [append 0~540672] snapc 27=[27,25,24,21] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-19T01:05:28.332 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:17.644607+0000", 2024-04-19T01:05:28.332 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.602263150000001, 2024-04-19T01:05:28.332 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.042363011999999999, 2024-04-19T01:05:28.332 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:28.332 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:28.332 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:28.332 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:28.332 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:28.332 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:28.332 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 828 2024-04-19T01:05:28.333 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.333 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:28.333 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.333 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:28.333 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.644607+0000", 2024-04-19T01:05:28.333 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.333 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.333 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.333 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:28.333 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.644607+0000", 2024-04-19T01:05:28.333 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.333 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.333 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.334 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:28.334 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.644610+0000", 2024-04-19T01:05:28.334 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1590000000000002e-06 2024-04-19T01:05:28.334 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.334 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.334 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:28.334 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.645352+0000", 2024-04-19T01:05:28.334 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00074229400000000001 2024-04-19T01:05:28.334 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.334 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.334 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:28.334 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.645388+0000", 2024-04-19T01:05:28.334 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5781999999999997e-05 2024-04-19T01:05:28.334 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.335 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.335 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:28.335 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.645409+0000", 2024-04-19T01:05:28.335 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0457000000000001e-05 2024-04-19T01:05:28.335 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.335 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.335 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.335 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.664800+0000", 2024-04-19T01:05:28.335 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.019391289999999999 2024-04-19T01:05:28.335 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.335 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.335 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:28.335 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.664886+0000", 2024-04-19T01:05:28.336 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.6055000000000002e-05 2024-04-19T01:05:28.336 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.336 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.336 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:28.336 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.665616+0000", 2024-04-19T01:05:28.336 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00072983900000000003 2024-04-19T01:05:28.336 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.336 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.336 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:28.336 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.675618+0000", 2024-04-19T01:05:28.336 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.010002499 2024-04-19T01:05:28.336 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.336 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.336 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:28.337 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.686866+0000", 2024-04-19T01:05:28.337 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.011247742 2024-04-19T01:05:28.337 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.337 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.337 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:28.337 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.686970+0000", 2024-04-19T01:05:28.337 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010389499999999999 2024-04-19T01:05:28.337 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.337 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.337 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.338 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.338 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.338 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:829 3.cs0 3:37e9fe95:::smithi15528762-36:head [append 0~245760] snapc 27=[27,25,24,21] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-19T01:05:28.338 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:17.646859+0000", 2024-04-19T01:05:28.338 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.600011546999999, 2024-04-19T01:05:28.338 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.041349916, 2024-04-19T01:05:28.338 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:28.338 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:28.338 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:28.338 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:28.338 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:28.338 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:28.339 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 829 2024-04-19T01:05:28.339 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.339 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:28.339 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.339 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:28.339 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.646859+0000", 2024-04-19T01:05:28.339 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.339 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.339 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.339 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:28.339 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.646859+0000", 2024-04-19T01:05:28.339 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.339 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.339 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.340 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:28.340 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.646861+0000", 2024-04-19T01:05:28.340 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1560000000000002e-06 2024-04-19T01:05:28.340 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.340 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.340 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:28.340 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.646978+0000", 2024-04-19T01:05:28.340 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011715800000000001 2024-04-19T01:05:28.340 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.340 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.340 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:28.340 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.647005+0000", 2024-04-19T01:05:28.340 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7134000000000001e-05 2024-04-19T01:05:28.341 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.341 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.341 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:28.341 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.647020+0000", 2024-04-19T01:05:28.341 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5273000000000001e-05 2024-04-19T01:05:28.341 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.341 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.341 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.342 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.670024+0000", 2024-04-19T01:05:28.342 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.023004053 2024-04-19T01:05:28.342 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.342 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.342 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:28.342 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.670220+0000", 2024-04-19T01:05:28.342 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000195334 2024-04-19T01:05:28.342 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.342 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.342 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:28.342 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.671430+0000", 2024-04-19T01:05:28.342 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001209943 2024-04-19T01:05:28.342 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.343 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.343 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:28.343 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.675927+0000", 2024-04-19T01:05:28.343 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0044970820000000003 2024-04-19T01:05:28.343 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.343 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.343 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:28.343 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.688107+0000", 2024-04-19T01:05:28.343 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.012180671000000001 2024-04-19T01:05:28.343 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.343 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.343 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:28.343 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.688209+0000", 2024-04-19T01:05:28.343 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000101112 2024-04-19T01:05:28.344 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.344 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.344 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.344 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.344 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.344 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:830 3.cs0 3:37e9fe95:::smithi15528762-36:head [setxattr _header (57) in=7b] snapc 27=[27,25,24,21] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-19T01:05:28.344 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:17.647045+0000", 2024-04-19T01:05:28.344 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.599825429999999, 2024-04-19T01:05:28.344 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.041885134999999997, 2024-04-19T01:05:28.344 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:28.345 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:28.345 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:28.345 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:28.345 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:28.345 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:28.345 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 830 2024-04-19T01:05:28.345 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.345 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:28.345 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.345 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:28.345 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.647045+0000", 2024-04-19T01:05:28.345 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.345 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.345 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.345 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:28.346 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.647045+0000", 2024-04-19T01:05:28.346 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.346 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.346 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.346 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:28.346 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.647046+0000", 2024-04-19T01:05:28.346 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.093e-06 2024-04-19T01:05:28.346 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.346 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.346 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:28.346 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.647052+0000", 2024-04-19T01:05:28.346 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2929999999999997e-06 2024-04-19T01:05:28.346 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.346 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.347 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:28.347 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.647070+0000", 2024-04-19T01:05:28.347 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7569000000000001e-05 2024-04-19T01:05:28.347 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.347 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.347 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:28.347 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.647081+0000", 2024-04-19T01:05:28.347 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1208e-05 2024-04-19T01:05:28.347 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.347 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.347 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.347 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.676262+0000", 2024-04-19T01:05:28.347 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.029180615 2024-04-19T01:05:28.347 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.348 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.348 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:28.348 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.676392+0000", 2024-04-19T01:05:28.348 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013023399999999999 2024-04-19T01:05:28.348 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.348 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.348 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:28.348 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.676961+0000", 2024-04-19T01:05:28.348 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00056880099999999996 2024-04-19T01:05:28.348 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.348 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.348 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:28.348 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.677575+0000", 2024-04-19T01:05:28.348 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00061419199999999995 2024-04-19T01:05:28.349 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.349 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.349 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:28.349 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.688845+0000", 2024-04-19T01:05:28.349 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.011270217000000001 2024-04-19T01:05:28.349 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.349 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.349 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:28.349 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.688930+0000", 2024-04-19T01:05:28.349 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4913000000000005e-05 2024-04-19T01:05:28.349 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.349 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.349 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.349 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.350 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.350 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:831 3.cs0 3:37e9fe95:::smithi15528762-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e107)", 2024-04-19T01:05:28.350 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:17.647118+0000", 2024-04-19T01:05:28.350 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.599752645000001, 2024-04-19T01:05:28.350 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.043597384000000003, 2024-04-19T01:05:28.350 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:28.350 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:28.350 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-04-19T01:05:28.350 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:28.350 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:28.350 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:28.350 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 831 2024-04-19T01:05:28.350 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.350 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:28.351 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.351 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:28.351 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.647118+0000", 2024-04-19T01:05:28.351 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.352 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.352 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.352 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:28.352 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.647118+0000", 2024-04-19T01:05:28.352 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.352 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.352 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.352 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:28.352 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.647119+0000", 2024-04-19T01:05:28.352 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0330000000000001e-06 2024-04-19T01:05:28.352 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.352 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.352 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:28.352 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.647127+0000", 2024-04-19T01:05:28.353 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.174e-06 2024-04-19T01:05:28.353 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.353 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.353 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:28.353 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.647146+0000", 2024-04-19T01:05:28.353 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8906000000000002e-05 2024-04-19T01:05:28.353 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.353 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.353 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:28.353 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.647157+0000", 2024-04-19T01:05:28.353 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1056000000000001e-05 2024-04-19T01:05:28.353 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.354 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.354 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.355 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.677273+0000", 2024-04-19T01:05:28.355 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.030116139 2024-04-19T01:05:28.355 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.355 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.355 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-04-19T01:05:28.355 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.677417+0000", 2024-04-19T01:05:28.355 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000144346 2024-04-19T01:05:28.355 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.355 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.355 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.355 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.689296+0000", 2024-04-19T01:05:28.356 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.011878717 2024-04-19T01:05:28.356 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.356 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.356 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:28.356 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.689404+0000", 2024-04-19T01:05:28.356 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000108433 2024-04-19T01:05:28.356 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.357 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.357 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:28.357 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.690715+0000", 2024-04-19T01:05:28.357 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00131058 2024-04-19T01:05:28.357 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.357 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.357 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.357 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.357 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.357 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:883 3.2s0 3:47ba51fe:::smithi15528762-10:head [append 0~425984] snapc 2a=[2a,29,28,25,24] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-04-19T01:05:28.358 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:22.582769+0000", 2024-04-19T01:05:28.358 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.6641010549999997, 2024-04-19T01:05:28.358 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.038710597999999999, 2024-04-19T01:05:28.358 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:28.358 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:28.358 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:28.358 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:28.358 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:28.358 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:28.358 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 883 2024-04-19T01:05:28.358 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.358 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:28.358 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.359 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:28.359 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.582769+0000", 2024-04-19T01:05:28.359 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.359 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.359 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.359 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:28.359 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.582769+0000", 2024-04-19T01:05:28.359 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.359 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.359 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.359 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:28.359 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.582772+0000", 2024-04-19T01:05:28.359 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.728e-06 2024-04-19T01:05:28.359 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.360 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.360 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:28.360 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.583192+0000", 2024-04-19T01:05:28.360 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00041997699999999998 2024-04-19T01:05:28.360 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.360 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.360 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:28.360 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.583220+0000", 2024-04-19T01:05:28.360 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7999000000000001e-05 2024-04-19T01:05:28.360 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.360 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.360 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:28.360 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.583233+0000", 2024-04-19T01:05:28.360 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3563e-05 2024-04-19T01:05:28.361 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.361 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.361 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.361 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.587569+0000", 2024-04-19T01:05:28.361 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0043357530000000003 2024-04-19T01:05:28.361 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.361 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.361 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:28.361 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.587656+0000", 2024-04-19T01:05:28.361 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.6732999999999998e-05 2024-04-19T01:05:28.361 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.361 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.361 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:28.361 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.588414+0000", 2024-04-19T01:05:28.362 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00075859199999999999 2024-04-19T01:05:28.362 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.362 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.362 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:28.362 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.591705+0000", 2024-04-19T01:05:28.362 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.003290173 2024-04-19T01:05:28.362 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.362 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.362 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:28.362 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.621369+0000", 2024-04-19T01:05:28.362 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.029664072 2024-04-19T01:05:28.362 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.362 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.363 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:28.363 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:22.621480+0000", 2024-04-19T01:05:28.363 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011100800000000001 2024-04-19T01:05:28.363 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.364 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.365 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.365 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.365 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.365 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:05:28.367 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:05:28.364+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:05:28.368 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-04-19T01:05:28.467 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:05:28.468+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:05:28.474 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-19T01:05:28.539 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:05:28.540 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:05:28.540 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-04-19T01:05:28.540 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:05:28.554 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-04-19T01:05:28.568 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:05:28.568+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:05:28.609 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [0] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-04-19T01:05:28.609 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-19T01:05:28.609 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-04-19T01:05:28.610 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T01:05:28.615 INFO:tasks.rados.rados.0.smithi155.stdout:365: done (7 left) 2024-04-19T01:05:28.616 INFO:tasks.rados.rados.0.smithi155.stdout:366: done (6 left) 2024-04-19T01:05:28.616 INFO:tasks.rados.rados.0.smithi155.stdout:367: done (5 left) 2024-04-19T01:05:28.616 INFO:tasks.rados.rados.0.smithi155.stdout:369: done (4 left) 2024-04-19T01:05:28.616 INFO:tasks.rados.rados.0.smithi155.stdout:372: done (3 left) 2024-04-19T01:05:28.616 INFO:tasks.rados.rados.0.smithi155.stdout:373: done (2 left) 2024-04-19T01:05:28.616 INFO:tasks.rados.rados.0.smithi155.stdout:375: done (1 left) 2024-04-19T01:05:28.616 INFO:tasks.rados.rados.0.smithi155.stdout:376: done (0 left) 2024-04-19T01:05:28.616 INFO:tasks.rados.rados.0.smithi155.stdout:377: snap_create 2024-04-19T01:05:28.631 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-19T01:05:28.635 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-04-19T01:05:28.636 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T01:05:28.657 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-04-19T01:05:28.657 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T01:05:28.670 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:05:28.666+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:05:28.676 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:05:28.676 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:05:28.676 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-04-19T01:05:28.676 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-04-19T01:05:28.676 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:05:28.678 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-04-19T01:05:28.678 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T01:05:28.690 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-04-19T01:05:28.699 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-04-19T01:05:28.699 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T01:05:28.717 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-04-19T01:05:28.717 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T01:05:28.741 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-04-19T01:05:28.742 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T01:05:28.765 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-04-19T01:05:28.765 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T01:05:28.770 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:05:28.769+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:05:28.793 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-04-19T01:05:28.793 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T01:05:28.811 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2024-04-19T01:05:28.812 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T01:05:28.821 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:05:28.821 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-04-19T01:05:28.821 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-04-19T01:05:28.822 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-04-19T01:05:28.822 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.822 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:173 3.3s0 3:c34529b5:::smithi15528762-31:head [append 0~507904] snapc 5=[5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-19T01:05:28.822 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:08.816172+0000", 2024-04-19T01:05:28.822 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 80.003655108999993, 2024-04-19T01:05:28.822 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.024750307999999999, 2024-04-19T01:05:28.822 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:28.822 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:28.822 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:28.822 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:28.822 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:28.822 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:28.822 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 173 2024-04-19T01:05:28.823 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.823 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:28.823 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.823 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:28.823 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816172+0000", 2024-04-19T01:05:28.823 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.823 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.823 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.823 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:28.823 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816172+0000", 2024-04-19T01:05:28.823 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.823 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.823 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.824 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:28.824 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816175+0000", 2024-04-19T01:05:28.824 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9589999999999997e-06 2024-04-19T01:05:28.824 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.824 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.824 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:28.824 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816766+0000", 2024-04-19T01:05:28.824 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00059020999999999998 2024-04-19T01:05:28.824 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.824 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.824 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:28.824 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816802+0000", 2024-04-19T01:05:28.824 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6140999999999998e-05 2024-04-19T01:05:28.824 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.825 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.825 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:28.825 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816820+0000", 2024-04-19T01:05:28.825 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7771999999999999e-05 2024-04-19T01:05:28.825 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.825 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.825 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.825 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816990+0000", 2024-04-19T01:05:28.825 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016993299999999999 2024-04-19T01:05:28.825 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.825 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.825 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op must wait for map", 2024-04-19T01:05:28.825 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.817016+0000", 2024-04-19T01:05:28.826 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6857000000000001e-05 2024-04-19T01:05:28.826 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.826 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.826 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.826 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.821660+0000", 2024-04-19T01:05:28.826 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0046440520000000001 2024-04-19T01:05:28.826 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.826 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.826 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:28.826 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.821835+0000", 2024-04-19T01:05:28.826 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000174112 2024-04-19T01:05:28.826 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.826 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.827 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:28.827 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.824020+0000", 2024-04-19T01:05:28.827 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.002185523 2024-04-19T01:05:28.827 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.827 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.827 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:28.827 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.839423+0000", 2024-04-19T01:05:28.827 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.015403261999999999 2024-04-19T01:05:28.827 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.827 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.827 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:28.827 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.840758+0000", 2024-04-19T01:05:28.827 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001335145 2024-04-19T01:05:28.827 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.828 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.828 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:28.828 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.840922+0000", 2024-04-19T01:05:28.828 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000163342 2024-04-19T01:05:28.828 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.828 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.828 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.828 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.828 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.828 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:176 3.3s0 3:c34529b5:::smithi15528762-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-19T01:05:28.828 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:08.822193+0000", 2024-04-19T01:05:28.828 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 79.997633645999997, 2024-04-19T01:05:28.828 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.023273202, 2024-04-19T01:05:28.828 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:28.829 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:28.829 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-04-19T01:05:28.829 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:28.829 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:28.829 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:28.829 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 176 2024-04-19T01:05:28.829 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.829 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:28.829 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.829 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:28.829 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822193+0000", 2024-04-19T01:05:28.829 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.829 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.829 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.830 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:28.830 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822193+0000", 2024-04-19T01:05:28.830 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.830 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.830 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.830 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:28.830 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822194+0000", 2024-04-19T01:05:28.830 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4200000000000004e-07 2024-04-19T01:05:28.830 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.830 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.830 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:28.830 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822198+0000", 2024-04-19T01:05:28.830 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.206e-06 2024-04-19T01:05:28.831 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.831 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.831 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:28.831 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822210+0000", 2024-04-19T01:05:28.831 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.217e-05 2024-04-19T01:05:28.831 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.831 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.831 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:28.831 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822220+0000", 2024-04-19T01:05:28.831 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0059e-05 2024-04-19T01:05:28.831 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.831 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.832 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.832 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.842930+0000", 2024-04-19T01:05:28.832 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.020709701 2024-04-19T01:05:28.832 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.832 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.832 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:28.832 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.843170+0000", 2024-04-19T01:05:28.832 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00024026700000000001 2024-04-19T01:05:28.832 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.832 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.832 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:28.832 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.845466+0000", 2024-04-19T01:05:28.833 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0022958570000000001 2024-04-19T01:05:28.833 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.833 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.833 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.833 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.833 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.833 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:343 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [copy-from ver 14] snapc 9=[8,7] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:05:28.833 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.511298+0000", 2024-04-19T01:05:28.833 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 70.308528648000006, 2024-04-19T01:05:28.833 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.043267551000000001, 2024-04-19T01:05:28.834 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:28.834 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:28.834 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:28.834 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:28.834 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:28.834 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:28.834 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 343 2024-04-19T01:05:28.834 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.834 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:28.834 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.834 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:28.834 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511298+0000", 2024-04-19T01:05:28.835 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.835 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.835 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.835 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:28.835 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511298+0000", 2024-04-19T01:05:28.835 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.835 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.835 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.835 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:28.835 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511302+0000", 2024-04-19T01:05:28.835 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1030000000000004e-06 2024-04-19T01:05:28.835 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.835 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.835 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:28.836 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511319+0000", 2024-04-19T01:05:28.836 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7070000000000001e-05 2024-04-19T01:05:28.836 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.836 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.836 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:28.836 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511357+0000", 2024-04-19T01:05:28.836 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8056000000000003e-05 2024-04-19T01:05:28.836 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.836 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.836 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:28.836 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511390+0000", 2024-04-19T01:05:28.836 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3213000000000002e-05 2024-04-19T01:05:28.836 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.837 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.837 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.837 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.516539+0000", 2024-04-19T01:05:28.837 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0051490729999999997 2024-04-19T01:05:28.837 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.837 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.837 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:28.837 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.516748+0000", 2024-04-19T01:05:28.837 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000208099 2024-04-19T01:05:28.837 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.837 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.837 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:28.838 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.541650+0000", 2024-04-19T01:05:28.838 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.024902837000000001 2024-04-19T01:05:28.839 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.839 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.839 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:28.839 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.551197+0000", 2024-04-19T01:05:28.839 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0095463240000000001 2024-04-19T01:05:28.839 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.839 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.839 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:28.839 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.554383+0000", 2024-04-19T01:05:28.839 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0031863830000000001 2024-04-19T01:05:28.840 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.840 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.840 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:28.840 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.554566+0000", 2024-04-19T01:05:28.840 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00018239299999999999 2024-04-19T01:05:28.840 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.840 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.840 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.840 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.840 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.840 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:344 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:05:28.840 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.511452+0000", 2024-04-19T01:05:28.840 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 70.308374397999998, 2024-04-19T01:05:28.841 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.038223957000000003, 2024-04-19T01:05:28.841 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:28.841 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:28.841 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-04-19T01:05:28.841 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:28.841 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:28.841 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:28.841 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 344 2024-04-19T01:05:28.841 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.841 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:28.841 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.842 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:28.842 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511452+0000", 2024-04-19T01:05:28.842 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.842 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.842 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.842 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:28.842 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511452+0000", 2024-04-19T01:05:28.842 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.842 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.842 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.842 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:28.843 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511454+0000", 2024-04-19T01:05:28.843 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.39e-06 2024-04-19T01:05:28.843 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.843 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.843 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:28.843 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511460+0000", 2024-04-19T01:05:28.843 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.46e-06 2024-04-19T01:05:28.843 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.843 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.843 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:28.843 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511476+0000", 2024-04-19T01:05:28.843 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6180000000000001e-05 2024-04-19T01:05:28.843 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.843 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.844 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:28.844 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511487+0000", 2024-04-19T01:05:28.844 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.049e-05 2024-04-19T01:05:28.844 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.844 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.844 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.844 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.517203+0000", 2024-04-19T01:05:28.844 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00571595 2024-04-19T01:05:28.845 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.845 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.845 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for blocked object", 2024-04-19T01:05:28.845 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.517341+0000", 2024-04-19T01:05:28.845 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013835099999999999 2024-04-19T01:05:28.845 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.845 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.845 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.846 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.549188+0000", 2024-04-19T01:05:28.846 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.031846589000000002 2024-04-19T01:05:28.846 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.846 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.846 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:28.846 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.549399+0000", 2024-04-19T01:05:28.846 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021149600000000001 2024-04-19T01:05:28.846 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.846 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.846 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:28.846 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.549676+0000", 2024-04-19T01:05:28.846 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000277051 2024-04-19T01:05:28.846 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.846 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.847 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.847 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.847 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.847 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.8s3 49/31 ECSubWrite(tid=203, reqid=client.4604.0:360, at_version=49'25, trim_to=0'0, roll_forward_to=49'24))", 2024-04-19T01:05:28.847 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:19.539927+0000", 2024-04-19T01:05:28.847 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 69.279899215, 2024-04-19T01:05:28.847 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.027280209, 2024-04-19T01:05:28.847 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:28.847 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:28.847 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:28.847 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:28.847 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.847 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:28.847 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.539927+0000", 2024-04-19T01:05:28.847 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.848 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.848 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.848 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:28.848 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.539927+0000", 2024-04-19T01:05:28.848 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.848 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.848 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.848 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:28.848 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.539929+0000", 2024-04-19T01:05:28.848 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3829999999999999e-06 2024-04-19T01:05:28.848 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.848 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.848 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:28.848 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540190+0000", 2024-04-19T01:05:28.849 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00026120499999999998 2024-04-19T01:05:28.849 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.849 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.849 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:28.849 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540216+0000", 2024-04-19T01:05:28.849 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5573000000000001e-05 2024-04-19T01:05:28.849 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.849 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.849 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:28.849 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540242+0000", 2024-04-19T01:05:28.849 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5939999999999999e-05 2024-04-19T01:05:28.849 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.849 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.849 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.849 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540387+0000", 2024-04-19T01:05:28.850 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000145983 2024-04-19T01:05:28.850 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.850 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.850 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:28.850 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540412+0000", 2024-04-19T01:05:28.850 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.442e-05 2024-04-19T01:05:28.850 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.850 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.850 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:28.850 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.567114+0000", 2024-04-19T01:05:28.850 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.026701643000000001 2024-04-19T01:05:28.850 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.850 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.850 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:28.851 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.567208+0000", 2024-04-19T01:05:28.851 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4061999999999995e-05 2024-04-19T01:05:28.851 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.851 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.851 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.851 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.851 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.851 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.8s3 60/31 ECSubWrite(tid=225, reqid=client.4604.0:433, at_version=60'43, trim_to=0'0, roll_forward_to=59'41))", 2024-04-19T01:05:28.851 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:30.569755+0000", 2024-04-19T01:05:28.851 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 58.250072013, 2024-04-19T01:05:28.851 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.026016885999999999, 2024-04-19T01:05:28.851 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:28.851 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:28.851 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:28.852 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:28.852 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.852 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:28.852 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.569755+0000", 2024-04-19T01:05:28.852 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.852 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.852 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.852 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:28.852 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.569755+0000", 2024-04-19T01:05:28.852 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.852 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.852 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.852 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:28.852 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.569755+0000", 2024-04-19T01:05:28.852 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6599999999999996e-07 2024-04-19T01:05:28.853 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.853 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.853 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:28.854 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.570112+0000", 2024-04-19T01:05:28.854 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00035699100000000002 2024-04-19T01:05:28.854 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.854 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.854 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:28.854 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.570159+0000", 2024-04-19T01:05:28.854 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7312999999999999e-05 2024-04-19T01:05:28.854 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.854 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.854 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:28.854 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.570184+0000", 2024-04-19T01:05:28.854 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4593e-05 2024-04-19T01:05:28.855 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.855 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.855 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.855 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.570362+0000", 2024-04-19T01:05:28.855 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017814400000000001 2024-04-19T01:05:28.855 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.855 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.855 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:28.855 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.570381+0000", 2024-04-19T01:05:28.855 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8369999999999999e-05 2024-04-19T01:05:28.855 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.855 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.855 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:28.855 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.595729+0000", 2024-04-19T01:05:28.855 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.025348186000000002 2024-04-19T01:05:28.856 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.856 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.856 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:28.856 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.595771+0000", 2024-04-19T01:05:28.856 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2722999999999997e-05 2024-04-19T01:05:28.856 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.856 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.856 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.856 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.856 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.856 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.4s4 71/31 ECSubWrite(tid=89, reqid=client.4604.0:518, at_version=71'22, trim_to=0'0, roll_forward_to=52'20))", 2024-04-19T01:05:28.856 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:41.542817+0000", 2024-04-19T01:05:28.856 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 47.277009792999998, 2024-04-19T01:05:28.856 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.068188129, 2024-04-19T01:05:28.857 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:28.857 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:28.857 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:28.857 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:28.857 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.857 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:28.857 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.542817+0000", 2024-04-19T01:05:28.857 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.857 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.857 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.857 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:28.857 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.542817+0000", 2024-04-19T01:05:28.857 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.857 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.858 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.858 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:28.858 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.542819+0000", 2024-04-19T01:05:28.860 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.283e-06 2024-04-19T01:05:28.860 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.860 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.860 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:28.860 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.545764+0000", 2024-04-19T01:05:28.860 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0029444530000000001 2024-04-19T01:05:28.860 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.860 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.860 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:28.860 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.545797+0000", 2024-04-19T01:05:28.861 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3203e-05 2024-04-19T01:05:28.861 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.861 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.861 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:28.861 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.545821+0000", 2024-04-19T01:05:28.861 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4436000000000001e-05 2024-04-19T01:05:28.861 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.861 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.861 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.861 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.545998+0000", 2024-04-19T01:05:28.861 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017682099999999999 2024-04-19T01:05:28.861 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.862 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.862 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:28.862 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.546048+0000", 2024-04-19T01:05:28.862 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0423000000000002e-05 2024-04-19T01:05:28.862 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.862 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.862 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:28.862 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.610960+0000", 2024-04-19T01:05:28.862 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.064912005999999994 2024-04-19T01:05:28.862 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.862 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.862 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:28.862 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.611005+0000", 2024-04-19T01:05:28.863 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4503999999999999e-05 2024-04-19T01:05:28.863 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.863 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.863 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.863 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.863 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.863 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:612 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [append 0~704512] snapc 1b=[1b,19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-04-19T01:05:28.863 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:50.652703+0000", 2024-04-19T01:05:28.863 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 38.167123580000002, 2024-04-19T01:05:28.863 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.035173242, 2024-04-19T01:05:28.863 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:28.863 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:28.863 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:28.863 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:28.864 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:28.864 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:28.864 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 612 2024-04-19T01:05:28.864 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.864 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:28.864 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.864 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:28.864 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.652703+0000", 2024-04-19T01:05:28.864 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.864 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.864 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.864 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:28.864 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.652703+0000", 2024-04-19T01:05:28.864 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.864 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.865 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.865 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:28.865 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.652706+0000", 2024-04-19T01:05:28.865 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4070000000000002e-06 2024-04-19T01:05:28.865 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.865 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.865 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:28.865 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.653472+0000", 2024-04-19T01:05:28.865 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00076519800000000003 2024-04-19T01:05:28.865 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.865 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.865 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:28.865 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.653507+0000", 2024-04-19T01:05:28.865 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5525999999999998e-05 2024-04-19T01:05:28.866 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.866 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.866 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:28.866 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.653525+0000", 2024-04-19T01:05:28.866 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7329e-05 2024-04-19T01:05:28.866 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.866 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.866 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.866 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.653662+0000", 2024-04-19T01:05:28.866 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013767900000000001 2024-04-19T01:05:28.866 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.866 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.866 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op must wait for map", 2024-04-19T01:05:28.866 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.653675+0000", 2024-04-19T01:05:28.867 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2723e-05 2024-04-19T01:05:28.867 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.867 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.867 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.867 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.658036+0000", 2024-04-19T01:05:28.867 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0043615820000000001 2024-04-19T01:05:28.867 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.867 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.867 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:28.867 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.658195+0000", 2024-04-19T01:05:28.867 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000158448 2024-04-19T01:05:28.867 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.868 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.868 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:28.868 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.660614+0000", 2024-04-19T01:05:28.868 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0024192749999999998 2024-04-19T01:05:28.868 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.868 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.869 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:28.869 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.687106+0000", 2024-04-19T01:05:28.869 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.026491879999999999 2024-04-19T01:05:28.869 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.870 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.870 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:28.870 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.687703+0000", 2024-04-19T01:05:28.870 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00059716299999999995 2024-04-19T01:05:28.870 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.870 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.870 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:28.871 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.687876+0000", 2024-04-19T01:05:28.871 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017303199999999999 2024-04-19T01:05:28.871 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.871 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.871 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.871 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.871 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.871 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:613 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [append 0~655360] snapc 1b=[1b,19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-04-19T01:05:28.871 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:50.661555+0000", 2024-04-19T01:05:28.871 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 38.158271343999999, 2024-04-19T01:05:28.871 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.038292557999999997, 2024-04-19T01:05:28.871 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:28.871 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:28.871 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:28.872 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:28.872 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:28.872 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:28.872 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 613 2024-04-19T01:05:28.872 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.872 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:28.872 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.872 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:28.872 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.661555+0000", 2024-04-19T01:05:28.872 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.872 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.872 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.872 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:28.873 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.661555+0000", 2024-04-19T01:05:28.873 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.873 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.873 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.873 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:28.873 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.661559+0000", 2024-04-19T01:05:28.873 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.337e-06 2024-04-19T01:05:28.873 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.873 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.873 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:28.873 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.662402+0000", 2024-04-19T01:05:28.873 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00084358700000000003 2024-04-19T01:05:28.873 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.873 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.874 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:28.874 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.662434+0000", 2024-04-19T01:05:28.874 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2223e-05 2024-04-19T01:05:28.874 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.874 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.874 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:28.874 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.662458+0000", 2024-04-19T01:05:28.874 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3598999999999999e-05 2024-04-19T01:05:28.874 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.874 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.874 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.874 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.673646+0000", 2024-04-19T01:05:28.874 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.011187852999999999 2024-04-19T01:05:28.874 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.875 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.875 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:28.875 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.673820+0000", 2024-04-19T01:05:28.875 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017450199999999999 2024-04-19T01:05:28.875 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.875 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.875 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:28.875 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.675719+0000", 2024-04-19T01:05:28.875 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001898757 2024-04-19T01:05:28.875 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.875 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.875 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:28.875 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.687984+0000", 2024-04-19T01:05:28.875 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.012265009 2024-04-19T01:05:28.876 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.876 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.876 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:28.876 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.699734+0000", 2024-04-19T01:05:28.876 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.011749426 2024-04-19T01:05:28.876 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.876 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.876 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:28.876 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.699848+0000", 2024-04-19T01:05:28.876 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000114265 2024-04-19T01:05:28.876 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.876 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.876 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.876 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.877 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.877 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:614 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [append 0~507904] snapc 1b=[1b,19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-04-19T01:05:28.877 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:50.668341+0000", 2024-04-19T01:05:28.877 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 38.151486108999997, 2024-04-19T01:05:28.877 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.032677876000000002, 2024-04-19T01:05:28.877 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:28.877 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:28.877 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:28.878 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:28.878 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:28.878 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:28.878 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 614 2024-04-19T01:05:28.878 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.878 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:28.878 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.878 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:28.878 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.668341+0000", 2024-04-19T01:05:28.878 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.878 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.879 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.879 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:28.879 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.668341+0000", 2024-04-19T01:05:28.879 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.879 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.879 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.879 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:28.879 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.668343+0000", 2024-04-19T01:05:28.879 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2759999999999999e-06 2024-04-19T01:05:28.879 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.879 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.879 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:28.879 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.668923+0000", 2024-04-19T01:05:28.879 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00058019699999999996 2024-04-19T01:05:28.880 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.880 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.880 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:28.880 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.668956+0000", 2024-04-19T01:05:28.880 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2539000000000001e-05 2024-04-19T01:05:28.880 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.880 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.880 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:28.880 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.668977+0000", 2024-04-19T01:05:28.880 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1903e-05 2024-04-19T01:05:28.880 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.880 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.880 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.880 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.688297+0000", 2024-04-19T01:05:28.881 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.019319106999999999 2024-04-19T01:05:28.881 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.881 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.881 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:28.881 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.688460+0000", 2024-04-19T01:05:28.881 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016338799999999999 2024-04-19T01:05:28.881 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.881 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.881 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:28.881 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.689733+0000", 2024-04-19T01:05:28.881 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001273499 2024-04-19T01:05:28.881 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.881 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.881 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:28.882 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.699996+0000", 2024-04-19T01:05:28.882 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.010263042 2024-04-19T01:05:28.882 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.882 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.882 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:28.882 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.700905+0000", 2024-04-19T01:05:28.882 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00090875999999999999 2024-04-19T01:05:28.882 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.882 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.882 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:28.882 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.701018+0000", 2024-04-19T01:05:28.882 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000113165 2024-04-19T01:05:28.882 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.882 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.883 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.883 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.883 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.883 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:615 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [setxattr _header (57) in=7b] snapc 1b=[1b,19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-04-19T01:05:28.883 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:50.669014+0000", 2024-04-19T01:05:28.883 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 38.150812631000001, 2024-04-19T01:05:28.883 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.032983169999999999, 2024-04-19T01:05:28.883 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:28.883 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:28.883 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:28.883 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:28.883 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:28.883 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:28.883 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 615 2024-04-19T01:05:28.884 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.884 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:28.884 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.884 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:28.884 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669014+0000", 2024-04-19T01:05:28.884 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.884 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.884 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.884 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:28.884 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669014+0000", 2024-04-19T01:05:28.884 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.884 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.884 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.884 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:28.884 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669016+0000", 2024-04-19T01:05:28.885 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3439999999999999e-06 2024-04-19T01:05:28.885 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.885 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.885 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:28.885 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669039+0000", 2024-04-19T01:05:28.885 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3008999999999999e-05 2024-04-19T01:05:28.885 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.885 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.885 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:28.885 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669057+0000", 2024-04-19T01:05:28.885 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.749e-05 2024-04-19T01:05:28.885 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.885 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.885 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:28.885 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669073+0000", 2024-04-19T01:05:28.886 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6492999999999999e-05 2024-04-19T01:05:28.886 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.886 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.886 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.886 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.698263+0000", 2024-04-19T01:05:28.886 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.029189397999999998 2024-04-19T01:05:28.886 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.886 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.886 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:28.886 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.698412+0000", 2024-04-19T01:05:28.886 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014930499999999999 2024-04-19T01:05:28.887 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.887 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.887 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:28.887 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.699113+0000", 2024-04-19T01:05:28.887 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00070141799999999999 2024-04-19T01:05:28.887 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.887 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.887 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:28.887 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.701045+0000", 2024-04-19T01:05:28.887 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001931667 2024-04-19T01:05:28.888 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.888 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.888 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:28.888 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.701847+0000", 2024-04-19T01:05:28.888 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00080155099999999996 2024-04-19T01:05:28.888 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.888 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.888 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:28.888 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.701997+0000", 2024-04-19T01:05:28.888 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000150495 2024-04-19T01:05:28.888 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.888 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.888 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.888 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.889 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.889 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:616 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e80)", 2024-04-19T01:05:28.889 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:50.669097+0000", 2024-04-19T01:05:28.889 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 38.150730095, 2024-04-19T01:05:28.889 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.038907733999999999, 2024-04-19T01:05:28.889 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:28.889 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:28.889 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-04-19T01:05:28.889 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:28.889 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:28.889 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:28.889 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 616 2024-04-19T01:05:28.889 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.889 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:28.890 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.890 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:28.890 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669097+0000", 2024-04-19T01:05:28.890 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.890 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.890 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.890 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:28.890 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669097+0000", 2024-04-19T01:05:28.890 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.890 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.890 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.890 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:28.890 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669098+0000", 2024-04-19T01:05:28.891 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.587e-06 2024-04-19T01:05:28.891 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.891 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.891 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:28.891 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669105+0000", 2024-04-19T01:05:28.891 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3660000000000003e-06 2024-04-19T01:05:28.891 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.891 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.891 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:28.891 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669130+0000", 2024-04-19T01:05:28.891 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4377000000000001e-05 2024-04-19T01:05:28.891 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.891 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.891 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:28.891 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669146+0000", 2024-04-19T01:05:28.892 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6399e-05 2024-04-19T01:05:28.892 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.892 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.892 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.892 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.702705+0000", 2024-04-19T01:05:28.892 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.033558534000000001 2024-04-19T01:05:28.892 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.892 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.892 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:28.892 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.702962+0000", 2024-04-19T01:05:28.892 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00025750700000000003 2024-04-19T01:05:28.892 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.892 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.892 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:28.892 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.708004+0000", 2024-04-19T01:05:28.893 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0050419640000000003 2024-04-19T01:05:28.893 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.893 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.893 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.893 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.893 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.893 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.es6 86/31 ECSubWrite(tid=195, reqid=client.4604.0:653, at_version=86'39, trim_to=0'0, roll_forward_to=84'38))", 2024-04-19T01:05:28.893 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:56.572643+0000", 2024-04-19T01:05:28.894 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 32.247183688, 2024-04-19T01:05:28.894 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.035539842000000002, 2024-04-19T01:05:28.894 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:28.894 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:28.894 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:28.894 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:28.894 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.894 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:28.894 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.572643+0000", 2024-04-19T01:05:28.894 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.894 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.895 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.895 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:28.895 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.572643+0000", 2024-04-19T01:05:28.895 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.895 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.895 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.895 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:28.895 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.572644+0000", 2024-04-19T01:05:28.895 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8999999999999996e-07 2024-04-19T01:05:28.896 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.896 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.896 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:28.896 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.573792+0000", 2024-04-19T01:05:28.896 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011481810000000001 2024-04-19T01:05:28.896 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.896 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.896 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:28.896 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.573818+0000", 2024-04-19T01:05:28.896 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5814000000000001e-05 2024-04-19T01:05:28.896 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.896 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.896 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:28.896 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.573832+0000", 2024-04-19T01:05:28.897 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4056999999999999e-05 2024-04-19T01:05:28.897 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.897 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.897 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.897 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.573964+0000", 2024-04-19T01:05:28.897 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013278399999999999 2024-04-19T01:05:28.897 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.897 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.897 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:28.897 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.573992+0000", 2024-04-19T01:05:28.898 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7250000000000002e-05 2024-04-19T01:05:28.898 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.898 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.898 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:28.898 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.608115+0000", 2024-04-19T01:05:28.898 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.034123819999999999 2024-04-19T01:05:28.898 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.898 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.898 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:28.898 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.608183+0000", 2024-04-19T01:05:28.898 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7346000000000002e-05 2024-04-19T01:05:28.898 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.898 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.898 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.898 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.899 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.899 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:745 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [append 0~573440] snapc 25=[25,24,22,21,1f] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-04-19T01:05:28.899 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:12.601513+0000", 2024-04-19T01:05:28.899 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 16.218314021000001, 2024-04-19T01:05:28.899 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.026388179000000001, 2024-04-19T01:05:28.899 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:28.899 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:28.899 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:28.899 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:28.899 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:28.899 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:28.899 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 745 2024-04-19T01:05:28.899 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.899 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:28.900 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.900 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:28.900 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.601513+0000", 2024-04-19T01:05:28.900 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.900 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.900 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.900 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:28.900 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.601513+0000", 2024-04-19T01:05:28.900 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.900 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.900 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.900 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:28.900 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.601516+0000", 2024-04-19T01:05:28.900 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6370000000000001e-06 2024-04-19T01:05:28.901 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.901 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.901 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:28.901 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.602170+0000", 2024-04-19T01:05:28.901 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00065339399999999996 2024-04-19T01:05:28.901 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.901 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.901 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:28.901 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.602201+0000", 2024-04-19T01:05:28.901 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1356000000000003e-05 2024-04-19T01:05:28.901 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.901 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.901 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:28.901 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.602225+0000", 2024-04-19T01:05:28.902 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4348000000000001e-05 2024-04-19T01:05:28.902 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.902 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.902 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.902 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.602333+0000", 2024-04-19T01:05:28.902 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000108137 2024-04-19T01:05:28.902 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.902 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.902 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:28.902 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.602579+0000", 2024-04-19T01:05:28.902 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00024595500000000001 2024-04-19T01:05:28.902 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.902 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.903 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:28.903 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.604232+0000", 2024-04-19T01:05:28.903 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0016524490000000001 2024-04-19T01:05:28.903 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.903 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.903 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:28.903 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.627421+0000", 2024-04-19T01:05:28.903 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.023188937 2024-04-19T01:05:28.903 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.903 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.903 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:28.903 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.627761+0000", 2024-04-19T01:05:28.903 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00034045499999999998 2024-04-19T01:05:28.903 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.903 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.904 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:28.904 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.627901+0000", 2024-04-19T01:05:28.904 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013951100000000001 2024-04-19T01:05:28.904 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.904 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.904 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.904 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.904 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.904 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:746 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [append 0~622592] snapc 25=[25,24,22,21,1f] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-04-19T01:05:28.904 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:12.610368+0000", 2024-04-19T01:05:28.904 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 16.209458591000001, 2024-04-19T01:05:28.904 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.034973300999999998, 2024-04-19T01:05:28.904 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:28.904 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:28.905 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:28.905 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:28.905 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:28.905 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:28.905 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 746 2024-04-19T01:05:28.905 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.905 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:28.905 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.905 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:28.905 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.610368+0000", 2024-04-19T01:05:28.905 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.905 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.905 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.905 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:28.905 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.610368+0000", 2024-04-19T01:05:28.906 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.906 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.906 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.906 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:28.906 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.610371+0000", 2024-04-19T01:05:28.906 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2830000000000002e-06 2024-04-19T01:05:28.906 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.906 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.907 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:28.907 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.611190+0000", 2024-04-19T01:05:28.907 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000818796 2024-04-19T01:05:28.907 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.907 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.907 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:28.908 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.611225+0000", 2024-04-19T01:05:28.908 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5008000000000001e-05 2024-04-19T01:05:28.908 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.908 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.908 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:28.908 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.611245+0000", 2024-04-19T01:05:28.908 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0231e-05 2024-04-19T01:05:28.908 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.908 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.908 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.908 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.614527+0000", 2024-04-19T01:05:28.908 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0032817599999999999 2024-04-19T01:05:28.908 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.908 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.908 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:28.909 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.614806+0000", 2024-04-19T01:05:28.909 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00027925500000000001 2024-04-19T01:05:28.909 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.909 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.909 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:28.909 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.616838+0000", 2024-04-19T01:05:28.909 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.002031167 2024-04-19T01:05:28.909 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.909 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.909 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:28.909 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.628169+0000", 2024-04-19T01:05:28.909 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.011331831000000001 2024-04-19T01:05:28.909 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.909 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.910 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:28.910 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.645187+0000", 2024-04-19T01:05:28.910 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.017017498999999998 2024-04-19T01:05:28.910 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.910 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.910 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:28.910 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.645341+0000", 2024-04-19T01:05:28.910 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015447099999999999 2024-04-19T01:05:28.910 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.910 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.910 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.910 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.910 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.911 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:747 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [append 0~622592] snapc 25=[25,24,22,21,1f] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-04-19T01:05:28.911 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:12.618378+0000", 2024-04-19T01:05:28.911 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 16.201448658, 2024-04-19T01:05:28.911 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.031200483000000001, 2024-04-19T01:05:28.911 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:28.911 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:28.911 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:28.911 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:28.911 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:28.911 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:28.912 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 747 2024-04-19T01:05:28.912 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.912 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:28.912 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.912 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:28.912 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.618378+0000", 2024-04-19T01:05:28.912 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.912 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.912 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.912 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:28.912 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.618378+0000", 2024-04-19T01:05:28.912 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.912 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.912 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.913 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:28.913 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.618380+0000", 2024-04-19T01:05:28.913 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4779999999999998e-06 2024-04-19T01:05:28.913 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.913 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.913 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:28.913 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.619044+0000", 2024-04-19T01:05:28.913 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00066315199999999997 2024-04-19T01:05:28.913 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.914 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.914 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:28.914 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.619073+0000", 2024-04-19T01:05:28.914 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9363e-05 2024-04-19T01:05:28.914 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.914 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.914 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:28.914 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.619088+0000", 2024-04-19T01:05:28.914 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5526e-05 2024-04-19T01:05:28.914 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.914 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.914 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.914 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.628378+0000", 2024-04-19T01:05:28.915 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0092899550000000008 2024-04-19T01:05:28.915 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.915 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.915 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:28.915 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.628555+0000", 2024-04-19T01:05:28.915 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017693299999999999 2024-04-19T01:05:28.915 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.915 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.915 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:28.915 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.630031+0000", 2024-04-19T01:05:28.915 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0014759720000000001 2024-04-19T01:05:28.915 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.915 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.915 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:28.915 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.645501+0000", 2024-04-19T01:05:28.916 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.015469505999999999 2024-04-19T01:05:28.916 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.916 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.916 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:28.916 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.649469+0000", 2024-04-19T01:05:28.916 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.003968297 2024-04-19T01:05:28.916 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.916 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.916 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:28.916 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.649578+0000", 2024-04-19T01:05:28.916 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000109301 2024-04-19T01:05:28.916 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.916 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.916 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.917 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.917 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.917 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:748 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [append 0~311296] snapc 25=[25,24,22,21,1f] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-04-19T01:05:28.917 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:12.621710+0000", 2024-04-19T01:05:28.917 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 16.198116482, 2024-04-19T01:05:28.917 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.02824397, 2024-04-19T01:05:28.917 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:28.917 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:28.917 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:28.917 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:28.918 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:28.918 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:28.918 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 748 2024-04-19T01:05:28.918 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.918 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:28.918 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.918 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:28.918 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.621710+0000", 2024-04-19T01:05:28.918 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.918 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.919 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.919 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:28.919 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.621710+0000", 2024-04-19T01:05:28.919 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.919 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.919 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.919 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:28.919 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.621711+0000", 2024-04-19T01:05:28.919 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3319999999999999e-06 2024-04-19T01:05:28.919 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.919 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.919 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:28.919 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.622069+0000", 2024-04-19T01:05:28.919 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00035724299999999998 2024-04-19T01:05:28.920 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.920 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.920 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:28.920 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.622094+0000", 2024-04-19T01:05:28.920 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5510000000000001e-05 2024-04-19T01:05:28.920 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.920 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.920 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:28.920 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.622107+0000", 2024-04-19T01:05:28.920 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2989e-05 2024-04-19T01:05:28.920 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.920 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.920 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.921 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.638128+0000", 2024-04-19T01:05:28.921 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.016020319000000002 2024-04-19T01:05:28.921 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.921 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.921 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:28.921 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.638423+0000", 2024-04-19T01:05:28.921 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000295006 2024-04-19T01:05:28.921 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.921 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.921 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:28.921 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.639758+0000", 2024-04-19T01:05:28.921 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0013350949999999999 2024-04-19T01:05:28.921 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.921 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.922 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:28.922 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.645682+0000", 2024-04-19T01:05:28.922 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0059239319999999998 2024-04-19T01:05:28.922 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.922 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.922 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:28.922 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.649864+0000", 2024-04-19T01:05:28.922 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.004182346 2024-04-19T01:05:28.922 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.922 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.922 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:28.922 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.649954+0000", 2024-04-19T01:05:28.922 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.0198000000000006e-05 2024-04-19T01:05:28.922 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.923 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.923 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.923 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.923 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.923 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:749 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [setxattr _header (57) in=7b] snapc 25=[25,24,22,21,1f] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-04-19T01:05:28.923 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:12.622198+0000", 2024-04-19T01:05:28.923 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 16.197628904999998, 2024-04-19T01:05:28.923 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.030651616, 2024-04-19T01:05:28.923 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:28.923 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:28.923 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:28.923 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:28.923 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:28.923 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:28.924 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 749 2024-04-19T01:05:28.924 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.924 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:28.924 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.924 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:28.924 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.622198+0000", 2024-04-19T01:05:28.924 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.924 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.924 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.924 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:28.924 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.622198+0000", 2024-04-19T01:05:28.925 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.925 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.926 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.926 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:28.926 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.622199+0000", 2024-04-19T01:05:28.926 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2800000000000005e-07 2024-04-19T01:05:28.926 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.926 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.926 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:28.926 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.622207+0000", 2024-04-19T01:05:28.926 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.0420000000000003e-06 2024-04-19T01:05:28.926 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.926 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.926 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:28.926 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.622220+0000", 2024-04-19T01:05:28.926 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3709e-05 2024-04-19T01:05:28.926 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.927 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.927 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:28.927 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.622229+0000", 2024-04-19T01:05:28.934 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.6200000000000005e-06 2024-04-19T01:05:28.938 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.938 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.938 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.938 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.646439+0000", 2024-04-19T01:05:28.938 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.024210084 2024-04-19T01:05:28.939 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.939 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.939 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:28.939 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.646693+0000", 2024-04-19T01:05:28.939 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00025368100000000001 2024-04-19T01:05:28.939 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.939 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.939 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:28.939 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.647772+0000", 2024-04-19T01:05:28.939 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0010789479999999999 2024-04-19T01:05:28.939 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.939 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.939 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:28.939 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.648875+0000", 2024-04-19T01:05:28.940 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011036139999999999 2024-04-19T01:05:28.940 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.940 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.940 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:28.940 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.652743+0000", 2024-04-19T01:05:28.940 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0038673510000000002 2024-04-19T01:05:28.940 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.940 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.940 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:28.940 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.652849+0000", 2024-04-19T01:05:28.940 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010663899999999999 2024-04-19T01:05:28.940 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.941 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.941 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.941 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.941 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.941 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:750 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e102)", 2024-04-19T01:05:28.941 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:12.622276+0000", 2024-04-19T01:05:28.941 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 16.197550451000001, 2024-04-19T01:05:28.941 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.033252480000000001, 2024-04-19T01:05:28.941 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:28.941 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:28.941 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-04-19T01:05:28.941 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:28.942 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:28.942 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:28.942 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 750 2024-04-19T01:05:28.942 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.942 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:28.942 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.943 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:28.943 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.622276+0000", 2024-04-19T01:05:28.944 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.944 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.944 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.944 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:28.944 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.622276+0000", 2024-04-19T01:05:28.944 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.944 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.945 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.945 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:28.945 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.622277+0000", 2024-04-19T01:05:28.945 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7000000000000004e-07 2024-04-19T01:05:28.945 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.945 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.945 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:28.945 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.622280+0000", 2024-04-19T01:05:28.945 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5640000000000001e-06 2024-04-19T01:05:28.945 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.945 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.945 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:28.945 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.622294+0000", 2024-04-19T01:05:28.946 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3278000000000001e-05 2024-04-19T01:05:28.946 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.946 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.946 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:28.946 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.622302+0000", 2024-04-19T01:05:28.946 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.3760000000000008e-06 2024-04-19T01:05:28.946 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.946 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.946 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.946 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.648257+0000", 2024-04-19T01:05:28.946 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.025954481000000001 2024-04-19T01:05:28.946 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.947 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.947 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-04-19T01:05:28.947 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.648468+0000", 2024-04-19T01:05:28.947 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021143900000000001 2024-04-19T01:05:28.947 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.947 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.947 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.947 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.653397+0000", 2024-04-19T01:05:28.947 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0049289210000000002 2024-04-19T01:05:28.947 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.947 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.947 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:28.947 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.653608+0000", 2024-04-19T01:05:28.948 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021074700000000001 2024-04-19T01:05:28.948 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.948 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.948 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:28.948 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.655529+0000", 2024-04-19T01:05:28.948 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001921004 2024-04-19T01:05:28.948 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.948 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.948 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.948 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.948 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.948 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.4s4 107/31 ECSubWrite(tid=196, reqid=client.4604.0:819, at_version=107'87, trim_to=0'0, roll_forward_to=102'85))", 2024-04-19T01:05:28.948 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:17.628492+0000", 2024-04-19T01:05:28.949 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 11.191335093999999, 2024-04-19T01:05:28.949 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.033676134000000003, 2024-04-19T01:05:28.949 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:28.949 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:28.949 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:28.949 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:28.949 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.949 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:28.949 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.628492+0000", 2024-04-19T01:05:28.949 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.949 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.949 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.949 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:28.949 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.628492+0000", 2024-04-19T01:05:28.950 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:28.950 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.950 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.950 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:28.950 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.628492+0000", 2024-04-19T01:05:28.950 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9700000000000003e-07 2024-04-19T01:05:28.950 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.950 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.950 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:28.950 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.629345+0000", 2024-04-19T01:05:28.950 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00085334600000000005 2024-04-19T01:05:28.950 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.951 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.951 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:28.951 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.629371+0000", 2024-04-19T01:05:28.951 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6084999999999998e-05 2024-04-19T01:05:28.951 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.951 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.951 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:28.951 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.629388+0000", 2024-04-19T01:05:28.951 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6733000000000001e-05 2024-04-19T01:05:28.951 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.951 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.951 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:28.951 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.629572+0000", 2024-04-19T01:05:28.951 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00018418500000000001 2024-04-19T01:05:28.952 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.952 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.952 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:28.952 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.629604+0000", 2024-04-19T01:05:28.952 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1794000000000001e-05 2024-04-19T01:05:28.952 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.952 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.953 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:28.953 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.662101+0000", 2024-04-19T01:05:28.953 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.032496628999999999 2024-04-19T01:05:28.953 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:28.954 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:28.954 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:28.954 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.662168+0000", 2024-04-19T01:05:28.954 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7064999999999996e-05 2024-04-19T01:05:28.954 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.955 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.955 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.955 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:28.955 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:28.955 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:05:28.956 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2024-04-19T01:05:28.957 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T01:05:28.957 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2024-04-19T01:05:28.957 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T01:05:28.958 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-19T01:05:28.958 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2024-04-19T01:05:28.959 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T01:05:28.959 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2024-04-19T01:05:28.959 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-19T01:05:28.960 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2024-04-19T01:05:28.960 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-19T01:05:28.961 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:05:28.962 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-04-19T01:05:28.962 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2024-04-19T01:05:28.963 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-19T01:05:28.965 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":153,"stamp":"2024-04-19T01:05:28.612912+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":102777376,"num_objects":67,"num_object_clones":23,"num_object_copies":459,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":3,"num_read":544,"num_read_kb":188335,"num_write":1007,"num_write_kb":144708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":16465920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1840,"ondisk_log_size":1840,"up":158,"acting":158,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":138,"num_osds":15,"num_per_pool_osds":15,"num_per_pool_omap_osds":15,"kb":1415577600,"kb_used":681780,"kb_used_data":191836,"kb_used_omap":23,"kb_used_meta":489704,"kb_avail":1414895820,"statfs":{"total":1449551462400,"available":1448853319680,"internally_reserved":0,"allocated":196440064,"data_stored":243298890,"data_compressed":2740708,"data_compressed_allocated":5644288,"data_compressed_original":12181504,"omap_allocated":23925,"internal_metadata":501457547},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7110656,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":12,"num_read_kb":31972,"num_write":37,"num_write_kb":6944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.025341"},"pg_stats":[{"pgid":"3.19","version":"113'148","reported_seq":595,"reported_epoch":116,"state":"active+clean+snaptrim","last_fresh":"2024-04-19T01:05:26.603961+0000","last_change":"2024-04-19T01:05:26.603961+0000","last_active":"2024-04-19T01:05:26.603961+0000","last_peered":"2024-04-19T01:05:26.603961+0000","last_clean":"2024-04-19T01:05:26.603961+0000","last_became_active":"2024-04-19T01:05:26.595940+0000","last_became_peered":"2024-04-19T01:05:26.595940+0000","last_unstale":"2024-04-19T01:05:26.603961+0000","last_undegraded":"2024-04-19T01:05:26.603961+0000","last_fullsized":"2024-04-19T01:05:26.603961+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'132","last_scrub_stamp":"2024-04-19T01:05:07.894862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:07.894862+0000","objects_scrubbed":7,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6004736,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":26,"num_read_kb":11819,"num_write":47,"num_write_kb":7976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11,14,9,4,10,6],"acting":[13,11,14,9,4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"27","length":"1"},{"start":"29","length":"1"}]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-19T01:05:25.585628+0000","last_change":"2024-04-19T01:05:25.585628+0000","last_active":"2024-04-19T01:05:25.585628+0000","last_peered":"2024-04-19T01:05:25.585628+0000","last_clean":"2024-04-19T01:05:25.585628+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-19T01:05:25.585628+0000","last_undegraded":"2024-04-19T01:05:25.585628+0000","last_fullsized":"2024-04-19T01:05:25.585628+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:25.585628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:25.585628+0000","last_clean_scrub_stamp":"2024-04-19T01:05:25.585628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.17","version":"112'27","reported_seq":485,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-19T01:05:26.570520+0000","last_change":"2024-04-19T01:05:26.569477+0000","last_active":"2024-04-19T01:05:26.570520+0000","last_peered":"2024-04-19T01:05:26.570520+0000","last_clean":"2024-04-19T01:05:26.570520+0000","last_became_active":"2024-04-19T01:05:26.562785+0000","last_became_peered":"2024-04-19T01:05:26.562785+0000","last_unstale":"2024-04-19T01:05:26.570520+0000","last_undegraded":"2024-04-19T01:05:26.570520+0000","last_fullsized":"2024-04-19T01:05:26.570520+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":5,"last_scrub":"107'23","last_scrub_stamp":"2024-04-19T01:05:18.361237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:18.361237+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0056852430000000004,"stat_sum":{"num_bytes":1540096,"num_objects":1,"num_object_clones":1,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":9027,"num_write":15,"num_write_kb":1504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1540096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,7,5,11,6,9],"acting":[14,13,7,5,11,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"}]},{"pgid":"3.16","version":"0'0","reported_seq":439,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-19T01:05:26.570459+0000","last_change":"2024-04-19T01:05:26.570386+0000","last_active":"2024-04-19T01:05:26.570459+0000","last_peered":"2024-04-19T01:05:26.570459+0000","last_clean":"2024-04-19T01:05:26.570459+0000","last_became_active":"2024-04-19T01:05:26.562883+0000","last_became_peered":"2024-04-19T01:05:26.562883+0000","last_unstale":"2024-04-19T01:05:26.570459+0000","last_undegraded":"2024-04-19T01:05:26.570459+0000","last_fullsized":"2024-04-19T01:05:26.570459+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:19.329813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:19.329813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00590596,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7,15,8,11,1,10],"acting":[14,7,15,8,11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"}]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-19T01:05:25.585628+0000","last_change":"2024-04-19T01:05:25.585628+0000","last_active":"2024-04-19T01:05:25.585628+0000","last_peered":"2024-04-19T01:05:25.585628+0000","last_clean":"2024-04-19T01:05:25.585628+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-19T01:05:25.585628+0000","last_undegraded":"2024-04-19T01:05:25.585628+0000","last_fullsized":"2024-04-19T01:05:25.585628+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:25.585628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:25.585628+0000","last_clean_scrub_stamp":"2024-04-19T01:05:25.585628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.14","version":"112'91","reported_seq":541,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-19T01:05:27.620951+0000","last_change":"2024-04-19T01:05:27.620951+0000","last_active":"2024-04-19T01:05:27.620951+0000","last_peered":"2024-04-19T01:05:27.620951+0000","last_clean":"2024-04-19T01:05:27.620951+0000","last_became_active":"2024-04-19T01:05:26.582876+0000","last_became_peered":"2024-04-19T01:05:26.582876+0000","last_unstale":"2024-04-19T01:05:27.620951+0000","last_undegraded":"2024-04-19T01:05:27.620951+0000","last_fullsized":"2024-04-19T01:05:27.620951+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'81","last_scrub_stamp":"2024-04-19T01:05:10.447179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:10.447179+0000","objects_scrubbed":5,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:33.414281+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0022728409999999998,"stat_sum":{"num_bytes":2678784,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":6703,"num_write":12,"num_write_kb":4432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,13,14,3,15,1,5],"acting":[10,13,14,3,15,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":182,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-19T01:05:23.696114+0000","last_change":"2024-04-19T01:03:53.939441+0000","last_active":"2024-04-19T01:05:23.696114+0000","last_peered":"2024-04-19T01:05:23.696114+0000","last_clean":"2024-04-19T01:05:23.696114+0000","last_became_active":"2024-04-19T01:03:53.938774+0000","last_became_peered":"2024-04-19T01:03:53.938774+0000","last_unstale":"2024-04-19T01:05:23.696114+0000","last_undegraded":"2024-04-19T01:05:23.696114+0000","last_fullsized":"2024-04-19T01:05:23.696114+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":439,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-19T01:05:26.558556+0000","last_change":"2024-04-19T01:05:26.560053+0000","last_active":"2024-04-19T01:05:26.558556+0000","last_peered":"2024-04-19T01:05:26.558556+0000","last_clean":"2024-04-19T01:05:26.558556+0000","last_became_active":"2024-04-19T01:05:26.554043+0000","last_became_peered":"2024-04-19T01:05:26.554043+0000","last_unstale":"2024-04-19T01:05:26.558556+0000","last_undegraded":"2024-04-19T01:05:26.558556+0000","last_fullsized":"2024-04-19T01:05:26.558556+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:19.329813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:19.329813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0039587559999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7,15,8,11,1,10],"acting":[14,7,15,8,11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":195,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-19T01:05:26.537572+0000","last_change":"2024-04-19T01:05:11.353464+0000","last_active":"2024-04-19T01:05:26.537572+0000","last_peered":"2024-04-19T01:05:26.537572+0000","last_clean":"2024-04-19T01:05:26.537572+0000","last_became_active":"2024-04-19T01:03:53.735549+0000","last_became_peered":"2024-04-19T01:03:53.735549+0000","last_unstale":"2024-04-19T01:05:26.537572+0000","last_undegraded":"2024-04-19T01:05:26.537572+0000","last_fullsized":"2024-04-19T01:05:26.537572+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:11.353213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:11.353213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:29.090848+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13],"acting":[14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.7","version":"116'29","reported_seq":488,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-19T01:05:26.575446+0000","last_change":"2024-04-19T01:05:26.577223+0000","last_active":"2024-04-19T01:05:26.575446+0000","last_peered":"2024-04-19T01:05:26.575446+0000","last_clean":"2024-04-19T01:05:26.575446+0000","last_became_active":"2024-04-19T01:05:26.561696+0000","last_became_peered":"2024-04-19T01:05:26.561696+0000","last_unstale":"2024-04-19T01:05:26.575446+0000","last_undegraded":"2024-04-19T01:05:26.575446+0000","last_fullsized":"2024-04-19T01:05:26.575446+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'23","last_scrub_stamp":"2024-04-19T01:05:18.361237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:18.361237+0000","objects_scrubbed":2,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.014013222000000001,"stat_sum":{"num_bytes":1540096,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":9026,"num_write":14,"num_write_kb":1504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1540096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,7,5,11,6,9],"acting":[14,13,7,5,11,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":195,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-19T01:05:26.527172+0000","last_change":"2024-04-19T01:05:21.106153+0000","last_active":"2024-04-19T01:05:26.527172+0000","last_peered":"2024-04-19T01:05:26.527172+0000","last_clean":"2024-04-19T01:05:26.527172+0000","last_became_active":"2024-04-19T01:03:53.771887+0000","last_became_peered":"2024-04-19T01:03:53.771887+0000","last_unstale":"2024-04-19T01:05:26.527172+0000","last_undegraded":"2024-04-19T01:05:26.527172+0000","last_fullsized":"2024-04-19T01:05:26.527172+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:21.105806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:21.105806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:40.011866+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.4","version":"116'93","reported_seq":544,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-19T01:05:27.623510+0000","last_change":"2024-04-19T01:05:27.626442+0000","last_active":"2024-04-19T01:05:27.623510+0000","last_peered":"2024-04-19T01:05:27.623510+0000","last_clean":"2024-04-19T01:05:27.623510+0000","last_became_active":"2024-04-19T01:05:26.560752+0000","last_became_peered":"2024-04-19T01:05:26.560752+0000","last_unstale":"2024-04-19T01:05:27.623510+0000","last_undegraded":"2024-04-19T01:05:27.623510+0000","last_fullsized":"2024-04-19T01:05:27.623510+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'81","last_scrub_stamp":"2024-04-19T01:05:10.447179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:10.447179+0000","objects_scrubbed":5,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.007379114,"stat_sum":{"num_bytes":2678784,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":19,"num_read_kb":6702,"num_write":12,"num_write_kb":4432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,13,14,3,15,1,5],"acting":[10,13,14,3,15,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":189,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-19T01:05:23.596492+0000","last_change":"2024-04-19T01:05:10.593158+0000","last_active":"2024-04-19T01:05:23.596492+0000","last_peered":"2024-04-19T01:05:23.596492+0000","last_clean":"2024-04-19T01:05:23.596492+0000","last_became_active":"2024-04-19T01:03:53.716040+0000","last_became_peered":"2024-04-19T01:03:53.716040+0000","last_unstale":"2024-04-19T01:05:23.596492+0000","last_undegraded":"2024-04-19T01:05:23.596492+0000","last_fullsized":"2024-04-19T01:05:23.596492+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:10.592906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:10.592906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:39.747266+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"113'215","reported_seq":636,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-19T01:05:23.699235+0000","last_change":"2024-04-19T01:05:23.699235+0000","last_active":"2024-04-19T01:05:23.699235+0000","last_peered":"2024-04-19T01:05:23.699235+0000","last_clean":"2024-04-19T01:05:23.699235+0000","last_became_active":"2024-04-19T01:04:03.237886+0000","last_became_peered":"2024-04-19T01:04:03.237886+0000","last_unstale":"2024-04-19T01:05:23.699235+0000","last_undegraded":"2024-04-19T01:05:23.699235+0000","last_fullsized":"2024-04-19T01:05:23.699235+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'212","last_scrub_stamp":"2024-04-19T01:05:20.309170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:20.309170+0000","objects_scrubbed":9,"log_size":215,"log_dups_size":0,"ondisk_log_size":215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:44.520034+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.003136588,"stat_sum":{"num_bytes":13647872,"num_objects":10,"num_object_clones":3,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":26975,"num_write":146,"num_write_kb":22416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,14,11,10,2,5],"acting":[8,6,14,11,10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"1f","length":"1"},{"start":"21","length":"2"},{"start":"26","length":"2"},{"start":"29","length":"1"}]},{"pgid":"3.3","version":"109'103","reported_seq":546,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-19T01:05:23.725917+0000","last_change":"2024-04-19T01:05:23.726200+0000","last_active":"2024-04-19T01:05:23.725917+0000","last_peered":"2024-04-19T01:05:23.725917+0000","last_clean":"2024-04-19T01:05:23.725917+0000","last_became_active":"2024-04-19T01:05:09.556011+0000","last_became_peered":"2024-04-19T01:05:09.556011+0000","last_unstale":"2024-04-19T01:05:23.725917+0000","last_undegraded":"2024-04-19T01:05:23.725917+0000","last_fullsized":"2024-04-19T01:05:23.725917+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051587499999999995,"stat_sum":{"num_bytes":7176192,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":16881,"num_write":101,"num_write_kb":9712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":2932736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2,3,1,4,12],"acting":[15,8,2,3,1,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"1f","length":"1"},{"start":"21","length":"2"},{"start":"26","length":"2"},{"start":"29","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":190,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-19T01:05:23.696000+0000","last_change":"2024-04-19T01:05:08.432764+0000","last_active":"2024-04-19T01:05:23.696000+0000","last_peered":"2024-04-19T01:05:23.696000+0000","last_clean":"2024-04-19T01:05:23.696000+0000","last_became_active":"2024-04-19T01:03:54.011460+0000","last_became_peered":"2024-04-19T01:03:54.011460+0000","last_unstale":"2024-04-19T01:05:23.696000+0000","last_undegraded":"2024-04-19T01:05:23.696000+0000","last_fullsized":"2024-04-19T01:05:23.696000+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:08.432519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:08.432519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:10.143460+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.0","version":"109'122","reported_seq":535,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-19T01:05:23.713862+0000","last_change":"2024-04-19T01:05:23.714244+0000","last_active":"2024-04-19T01:05:23.713862+0000","last_peered":"2024-04-19T01:05:23.713862+0000","last_clean":"2024-04-19T01:05:23.713862+0000","last_became_active":"2024-04-19T01:04:03.322503+0000","last_became_peered":"2024-04-19T01:04:03.322503+0000","last_unstale":"2024-04-19T01:05:23.713862+0000","last_undegraded":"2024-04-19T01:05:23.713862+0000","last_fullsized":"2024-04-19T01:05:23.713862+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'96","last_scrub_stamp":"2024-04-19T01:05:06.447000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:06.447000+0000","objects_scrubbed":5,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:10.812950+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.00069242700000000002,"stat_sum":{"num_bytes":9191424,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":50268,"num_write":39,"num_write_kb":10960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4,3,10,11,6],"acting":[1,9,4,3,10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1f","length":"1"},{"start":"21","length":"2"},{"start":"26","length":"2"},{"start":"29","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":180,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-19T01:05:23.725340+0000","last_change":"2024-04-19T01:03:53.650434+0000","last_active":"2024-04-19T01:05:23.725340+0000","last_peered":"2024-04-19T01:05:23.725340+0000","last_clean":"2024-04-19T01:05:23.725340+0000","last_became_active":"2024-04-19T01:03:53.649606+0000","last_became_peered":"2024-04-19T01:03:53.649606+0000","last_unstale":"2024-04-19T01:05:23.725340+0000","last_undegraded":"2024-04-19T01:05:23.725340+0000","last_fullsized":"2024-04-19T01:05:23.725340+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.1","version":"112'70","reported_seq":468,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-19T01:05:23.596993+0000","last_change":"2024-04-19T01:05:23.597555+0000","last_active":"2024-04-19T01:05:23.596993+0000","last_peered":"2024-04-19T01:05:23.596993+0000","last_clean":"2024-04-19T01:05:23.596993+0000","last_became_active":"2024-04-19T01:04:03.323727+0000","last_became_peered":"2024-04-19T01:04:03.323727+0000","last_unstale":"2024-04-19T01:05:23.596993+0000","last_undegraded":"2024-04-19T01:05:23.596993+0000","last_fullsized":"2024-04-19T01:05:23.596993+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:23.830712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087053800000000002,"stat_sum":{"num_bytes":802816,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":14,"num_read_kb":2020,"num_write":53,"num_write_kb":4992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,11,9,1,7],"acting":[6,4,3,11,9,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1f","length":"1"},{"start":"21","length":"2"},{"start":"26","length":"2"},{"start":"29","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":182,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-19T01:05:23.695582+0000","last_change":"2024-04-19T01:03:53.760075+0000","last_active":"2024-04-19T01:05:23.695582+0000","last_peered":"2024-04-19T01:05:23.695582+0000","last_clean":"2024-04-19T01:05:23.695582+0000","last_became_active":"2024-04-19T01:03:53.759326+0000","last_became_peered":"2024-04-19T01:03:53.759326+0000","last_unstale":"2024-04-19T01:05:23.695582+0000","last_undegraded":"2024-04-19T01:05:23.695582+0000","last_fullsized":"2024-04-19T01:05:23.695582+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.2","version":"113'61","reported_seq":494,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-19T01:05:23.706187+0000","last_change":"2024-04-19T01:05:23.707487+0000","last_active":"2024-04-19T01:05:23.706187+0000","last_peered":"2024-04-19T01:05:23.706187+0000","last_clean":"2024-04-19T01:05:23.706187+0000","last_became_active":"2024-04-19T01:05:09.553946+0000","last_became_peered":"2024-04-19T01:05:09.553946+0000","last_unstale":"2024-04-19T01:05:23.706187+0000","last_undegraded":"2024-04-19T01:05:23.706187+0000","last_fullsized":"2024-04-19T01:05:23.706187+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:28.701354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0083663990000000001,"stat_sum":{"num_bytes":6963200,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":13,"num_read_kb":534,"num_write":22,"num_write_kb":7136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":2588672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,13,4,8,3,7],"acting":[11,5,13,4,8,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"1f","length":"1"},{"start":"21","length":"2"},{"start":"26","length":"2"},{"start":"29","length":"1"}]},{"pgid":"2.3","version":"0'0","reported_seq":195,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-19T01:05:23.698979+0000","last_change":"2024-04-19T01:05:07.549099+0000","last_active":"2024-04-19T01:05:23.698979+0000","last_peered":"2024-04-19T01:05:23.698979+0000","last_clean":"2024-04-19T01:05:23.698979+0000","last_became_active":"2024-04-19T01:05:06.529945+0000","last_became_peered":"2024-04-19T01:05:06.529945+0000","last_unstale":"2024-04-19T01:05:23.698979+0000","last_undegraded":"2024-04-19T01:05:23.698979+0000","last_fullsized":"2024-04-19T01:05:23.698979+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:07.548635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:07.548635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:16.001957+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"25'157","reported_seq":392,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-19T01:05:23.698895+0000","last_change":"2024-04-19T01:05:11.601754+0000","last_active":"2024-04-19T01:05:23.698895+0000","last_peered":"2024-04-19T01:05:23.698895+0000","last_clean":"2024-04-19T01:05:23.698895+0000","last_became_active":"2024-04-19T01:03:50.774078+0000","last_became_peered":"2024-04-19T01:03:50.774078+0000","last_unstale":"2024-04-19T01:05:23.698895+0000","last_undegraded":"2024-04-19T01:05:23.698895+0000","last_fullsized":"2024-04-19T01:05:23.698895+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'157","last_scrub_stamp":"2024-04-19T01:05:11.601630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:49.580851+0000","last_clean_scrub_stamp":"2024-04-19T01:05:11.601630+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:12.040402+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.8","version":"112'67","reported_seq":517,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-19T01:05:23.699460+0000","last_change":"2024-04-19T01:05:23.699801+0000","last_active":"2024-04-19T01:05:23.699460+0000","last_peered":"2024-04-19T01:05:23.699460+0000","last_clean":"2024-04-19T01:05:23.699460+0000","last_became_active":"2024-04-19T01:05:09.549364+0000","last_became_peered":"2024-04-19T01:05:09.549364+0000","last_unstale":"2024-04-19T01:05:23.699460+0000","last_undegraded":"2024-04-19T01:05:23.699460+0000","last_fullsized":"2024-04-19T01:05:23.699460+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'65","last_scrub_stamp":"2024-04-19T01:05:19.632517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:19.632517+0000","objects_scrubbed":4,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:22.743666+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0.00081038900000000001,"stat_sum":{"num_bytes":3112960,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":1671,"num_write":63,"num_write_kb":5664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":3112960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,1,15,14,6,5],"acting":[11,10,1,15,14,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"1f","length":"1"},{"start":"21","length":"2"},{"start":"26","length":"2"},{"start":"29","length":"1"}]},{"pgid":"3.9","version":"113'148","reported_seq":596,"reported_epoch":116,"state":"active+clean+snaptrim","last_fresh":"2024-04-19T01:05:26.569937+0000","last_change":"2024-04-19T01:05:26.569937+0000","last_active":"2024-04-19T01:05:26.569937+0000","last_peered":"2024-04-19T01:05:26.569937+0000","last_clean":"2024-04-19T01:05:26.569937+0000","last_became_active":"2024-04-19T01:05:26.556542+0000","last_became_peered":"2024-04-19T01:05:26.556542+0000","last_unstale":"2024-04-19T01:05:26.569937+0000","last_undegraded":"2024-04-19T01:05:26.569937+0000","last_fullsized":"2024-04-19T01:05:26.569937+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'132","last_scrub_stamp":"2024-04-19T01:05:07.894862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:07.894862+0000","objects_scrubbed":7,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.001169034,"stat_sum":{"num_bytes":6004736,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":11818,"num_write":47,"num_write_kb":7976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11,14,9,4,10,6],"acting":[13,11,14,9,4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"27","length":"1"},{"start":"29","length":"1"}]},{"pgid":"3.a","version":"112'112","reported_seq":522,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-19T01:05:23.581372+0000","last_change":"2024-04-19T01:05:23.581881+0000","last_active":"2024-04-19T01:05:23.581372+0000","last_peered":"2024-04-19T01:05:23.581372+0000","last_clean":"2024-04-19T01:05:23.581372+0000","last_became_active":"2024-04-19T01:04:03.323177+0000","last_became_peered":"2024-04-19T01:04:03.323177+0000","last_unstale":"2024-04-19T01:05:23.581372+0000","last_undegraded":"2024-04-19T01:05:23.581372+0000","last_fullsized":"2024-04-19T01:05:23.581372+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:23.585660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070403299999999996,"stat_sum":{"num_bytes":8126464,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":32,"num_read_kb":5821,"num_write":37,"num_write_kb":15312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,7,3,13,1,8],"acting":[12,4,7,3,13,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"1f","length":"1"},{"start":"21","length":"2"},{"start":"26","length":"2"},{"start":"29","length":"1"}]},{"pgid":"3.b","version":"105'56","reported_seq":491,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-19T01:05:27.621160+0000","last_change":"2024-04-19T01:05:27.622998+0000","last_active":"2024-04-19T01:05:27.621160+0000","last_peered":"2024-04-19T01:05:27.621160+0000","last_clean":"2024-04-19T01:05:27.621160+0000","last_became_active":"2024-04-19T01:04:02.781569+0000","last_became_peered":"2024-04-19T01:04:02.781569+0000","last_unstale":"2024-04-19T01:05:27.621160+0000","last_undegraded":"2024-04-19T01:05:27.621160+0000","last_fullsized":"2024-04-19T01:05:27.621160+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048062399999999998,"stat_sum":{"num_bytes":5095424,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":8182,"num_write":39,"num_write_kb":4976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11,15,1,4,14],"acting":[10,6,11,15,1,4,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"}]},{"pgid":"3.c","version":"112'82","reported_seq":484,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-19T01:05:23.700084+0000","last_change":"2024-04-19T01:05:23.700451+0000","last_active":"2024-04-19T01:05:23.700084+0000","last_peered":"2024-04-19T01:05:23.700084+0000","last_clean":"2024-04-19T01:05:23.700084+0000","last_became_active":"2024-04-19T01:05:09.550415+0000","last_became_peered":"2024-04-19T01:05:09.550415+0000","last_unstale":"2024-04-19T01:05:23.700084+0000","last_undegraded":"2024-04-19T01:05:23.700084+0000","last_fullsized":"2024-04-19T01:05:23.700084+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012176260000000001,"stat_sum":{"num_bytes":2883584,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":2454,"num_write":43,"num_write_kb":4144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,12,1,9,2,6],"acting":[11,10,12,1,9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"1f","length":"1"},{"start":"21","length":"2"},{"start":"26","length":"2"},{"start":"29","length":"1"}]},{"pgid":"3.d","version":"109'98","reported_seq":529,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-19T01:05:23.581648+0000","last_change":"2024-04-19T01:05:23.582154+0000","last_active":"2024-04-19T01:05:23.581648+0000","last_peered":"2024-04-19T01:05:23.581648+0000","last_clean":"2024-04-19T01:05:23.581648+0000","last_became_active":"2024-04-19T01:05:09.548237+0000","last_became_peered":"2024-04-19T01:05:09.548237+0000","last_unstale":"2024-04-19T01:05:23.581648+0000","last_undegraded":"2024-04-19T01:05:23.581648+0000","last_fullsized":"2024-04-19T01:05:23.581648+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079345399999999997,"stat_sum":{"num_bytes":9125888,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":4329,"num_write":35,"num_write_kb":10000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":4751360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,9,1,8,2,6],"acting":[12,4,9,1,8,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"1f","length":"1"},{"start":"21","length":"2"},{"start":"26","length":"2"},{"start":"29","length":"1"}]},{"pgid":"3.e","version":"115'57","reported_seq":491,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-19T01:05:26.533336+0000","last_change":"2024-04-19T01:05:25.651797+0000","last_active":"2024-04-19T01:05:26.533336+0000","last_peered":"2024-04-19T01:05:26.533336+0000","last_clean":"2024-04-19T01:05:26.533336+0000","last_became_active":"2024-04-19T01:04:03.238386+0000","last_became_peered":"2024-04-19T01:04:03.238386+0000","last_unstale":"2024-04-19T01:05:26.533336+0000","last_undegraded":"2024-04-19T01:05:26.533336+0000","last_fullsized":"2024-04-19T01:05:26.533336+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'45","last_scrub_stamp":"2024-04-19T01:05:05.862139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:05.862139+0000","objects_scrubbed":3,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:07.258148+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.047169318000000002,"stat_sum":{"num_bytes":2277376,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":22,"num_read_kb":8695,"num_write":17,"num_write_kb":5088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11,5,4,2,9,15],"acting":[13,11,5,4,2,9,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"}]},{"pgid":"3.f","version":"117'104","reported_seq":520,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-19T01:05:27.628745+0000","last_change":"2024-04-19T01:05:27.630203+0000","last_active":"2024-04-19T01:05:27.628745+0000","last_peered":"2024-04-19T01:05:27.628745+0000","last_clean":"2024-04-19T01:05:27.628745+0000","last_became_active":"2024-04-19T01:05:09.542698+0000","last_became_peered":"2024-04-19T01:05:09.542698+0000","last_unstale":"2024-04-19T01:05:27.628745+0000","last_undegraded":"2024-04-19T01:05:27.628745+0000","last_fullsized":"2024-04-19T01:05:27.628745+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'81","last_scrub_stamp":"2024-04-19T01:05:18.498676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:18.498676+0000","objects_scrubbed":4,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:34.128207+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.012032268,"stat_sum":{"num_bytes":13336576,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":21,"num_read_kb":5292,"num_write":62,"num_write_kb":13024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,13,5,11,7,1,2],"acting":[10,13,5,11,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"}]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-19T01:05:25.585628+0000","last_change":"2024-04-19T01:05:25.585628+0000","last_active":"2024-04-19T01:05:25.585628+0000","last_peered":"2024-04-19T01:05:25.585628+0000","last_clean":"2024-04-19T01:05:25.585628+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-19T01:05:25.585628+0000","last_undegraded":"2024-04-19T01:05:25.585628+0000","last_fullsized":"2024-04-19T01:05:25.585628+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:25.585628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:25.585628+0000","last_clean_scrub_stamp":"2024-04-19T01:05:25.585628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-19T01:05:25.585628+0000","last_change":"2024-04-19T01:05:25.585628+0000","last_active":"2024-04-19T01:05:25.585628+0000","last_peered":"2024-04-19T01:05:25.585628+0000","last_clean":"2024-04-19T01:05:25.585628+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-19T01:05:25.585628+0000","last_undegraded":"2024-04-19T01:05:25.585628+0000","last_fullsized":"2024-04-19T01:05:25.585628+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:25.585628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:25.585628+0000","last_clean_scrub_stamp":"2024-04-19T01:05:25.585628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-19T01:05:25.585628+0000","last_change":"2024-04-19T01:05:25.585628+0000","last_active":"2024-04-19T01:05:25.585628+0000","last_peered":"2024-04-19T01:05:25.585628+0000","last_clean":"2024-04-19T01:05:25.585628+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-19T01:05:25.585628+0000","last_undegraded":"2024-04-19T01:05:25.585628+0000","last_fullsized":"2024-04-19T01:05:25.585628+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:25.585628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:25.585628+0000","last_clean_scrub_stamp":"2024-04-19T01:05:25.585628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-19T01:05:25.585628+0000","last_change":"2024-04-19T01:05:25.585628+0000","last_active":"2024-04-19T01:05:25.585628+0000","last_peered":"2024-04-19T01:05:25.585628+0000","last_clean":"2024-04-19T01:05:25.585628+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-19T01:05:25.585628+0000","last_undegraded":"2024-04-19T01:05:25.585628+0000","last_fullsized":"2024-04-19T01:05:25.585628+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:25.585628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:25.585628+0000","last_clean_scrub_stamp":"2024-04-19T01:05:25.585628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":102187008,"num_objects":65,"num_object_clones":23,"num_object_copies":455,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":3,"num_read":408,"num_read_kb":188217,"num_write":804,"num_write_kb":141248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":16465920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":182677504,"data_stored":234041344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1683,"ondisk_log_size":1683,"up":140,"acting":140,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":13},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":15546,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":14,"up_from":24,"seq":103079215127,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46972,"kb_used_data":11632,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94324868,"statfs":{"total":96636764160,"available":96588664832,"internally_reserved":0,"allocated":11911168,"data_stored":12975125,"data_compressed":184121,"data_compressed_allocated":376832,"data_compressed_original":753664,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,5,6,7,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":13,"up_from":24,"seq":103079215128,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49132,"kb_used_data":13792,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94322708,"statfs":{"total":96636764160,"available":96586452992,"internally_reserved":0,"allocated":14123008,"data_stored":17320981,"data_compressed":184121,"data_compressed_allocated":376832,"data_compressed_original":753664,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":9,"up_from":24,"seq":103079215129,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46256,"kb_used_data":10916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94325584,"statfs":{"total":96636764160,"available":96589398016,"internally_reserved":0,"allocated":11177984,"data_stored":14609429,"data_compressed":184121,"data_compressed_allocated":376832,"data_compressed_original":753664,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[1,2,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313240,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47812,"kb_used_data":16624,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324028,"statfs":{"total":96636764160,"available":96587804672,"internally_reserved":0,"allocated":17022976,"data_stored":21728248,"data_compressed":188205,"data_compressed_allocated":385024,"data_compressed_original":770048,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280533,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46840,"kb_used_data":15660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325000,"statfs":{"total":96636764160,"available":96588800000,"internally_reserved":0,"allocated":16035840,"data_stored":19878650,"data_compressed":178008,"data_compressed_allocated":364544,"data_compressed_original":729088,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,3,4,5,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280535,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47536,"kb_used_data":16356,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324304,"statfs":{"total":96636764160,"available":96588087296,"internally_reserved":0,"allocated":16748544,"data_stored":20759290,"data_compressed":178008,"data_compressed_allocated":364544,"data_compressed_original":729088,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,6,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280534,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37488,"kb_used_data":10464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334352,"statfs":{"total":96636764160,"available":96598376448,"internally_reserved":0,"allocated":10715136,"data_stored":13503331,"data_compressed":180048,"data_compressed_allocated":368640,"data_compressed_original":737280,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[2,4,5,6,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280534,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47300,"kb_used_data":16116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324540,"statfs":{"total":96636764160,"available":96588328960,"internally_reserved":0,"allocated":16502784,"data_stored":20687715,"data_compressed":180048,"data_compressed_allocated":368640,"data_compressed_original":737280,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280534,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43592,"kb_used_data":12408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328248,"statfs":{"total":96636764160,"available":96592125952,"internally_reserved":0,"allocated":12705792,"data_stored":15056259,"data_compressed":187821,"data_compressed_allocated":401408,"data_compressed_original":1249280,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,6,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280535,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43460,"kb_used_data":12276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328380,"statfs":{"total":96636764160,"available":96592261120,"internally_reserved":0,"allocated":12570624,"data_stored":15514467,"data_compressed":180048,"data_compressed_allocated":368640,"data_compressed_original":737280,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,5,6,7,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280533,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67652,"kb_used_data":18420,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94304188,"statfs":{"total":96636764160,"available":96567488512,"internally_reserved":0,"allocated":18862080,"data_stored":23915907,"data_compressed":187821,"data_compressed_allocated":401408,"data_compressed_original":1249280,"omap_allocated":1596,"internal_metadata":50395588},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280533,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34972,"kb_used_data":7948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336868,"statfs":{"total":96636764160,"available":96600952832,"internally_reserved":0,"allocated":8138752,"data_stored":10259299,"data_compressed":180048,"data_compressed_allocated":368640,"data_compressed_original":737280,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7,8,9,10,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280534,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33924,"kb_used_data":6900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337916,"statfs":{"total":96636764160,"available":96602025984,"internally_reserved":0,"allocated":7065600,"data_stored":7482211,"data_compressed":180048,"data_compressed_allocated":368640,"data_compressed_original":737280,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":24,"seq":103079215127,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48472,"kb_used_data":13132,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94323368,"statfs":{"total":96636764160,"available":96587128832,"internally_reserved":0,"allocated":13447168,"data_stored":17206293,"data_compressed":184121,"data_compressed_allocated":376832,"data_compressed_original":753664,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":0,"seq":103079215128,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":24,"seq":103079215127,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40372,"kb_used_data":9192,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331468,"statfs":{"total":96636764160,"available":96595423232,"internally_reserved":0,"allocated":9412608,"data_stored":12401685,"data_compressed":184121,"data_compressed_allocated":376832,"data_compressed_original":753664,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5787648,"data_stored":6684672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":15450112,"data_stored":19775488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12111872,"data_stored":16191488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":9404416,"data_stored":12509184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":15192064,"data_stored":19693568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":11341824,"data_stored":13533184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":14737408,"data_stored":18894848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8675328,"data_stored":11984896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":11259904,"data_stored":14520320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":10133504,"data_stored":13885440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":15663104,"data_stored":20692992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":17498112,"data_stored":21483520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":6828032,"data_stored":9265152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":12787712,"data_stored":16306176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":10051584,"data_stored":12132352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":5754880,"data_stored":6488064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T01:05:28.967 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2024-04-19T01:05:28.968 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 36 2024-04-19T01:05:28.968 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2024-04-19T01:05:28.968 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-19T01:05:28.969 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:05:28.953+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:05:29.057 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:05:29.054+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:05:29.083 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:29.083 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:29.097 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-19T01:05:29.161 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:05:29.160+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:05:29.212 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:29.212 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:29.225 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-19T01:05:29.274 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:05:29.273+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:05:29.343 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:29.344 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:29.357 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-19T01:05:29.399 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:05:29.397+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:05:29.462 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:05:29.460+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:05:29.516 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:29.516 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:29.530 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-19T01:05:29.563 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:05:29.564+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:05:29.618 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2024-04-19T01:05:29.618 INFO:teuthology.orchestra.run.smithi033.stderr:set pool 3 pg_num to 36 2024-04-19T01:05:29.627 INFO:tasks.rados.rados.0.smithi155.stdout:377: done (0 left) 2024-04-19T01:05:29.627 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 32 current snap is 44 2024-04-19T01:05:29.627 INFO:tasks.rados.rados.0.smithi155.stdout:378: seq_num 148 ranges {3145728=81920} 2024-04-19T01:05:29.653 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:29.654 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:29.664 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:05:29.664+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:05:29.666 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-19T01:05:29.672 INFO:tasks.rados.rados.0.smithi155.stdout:378: writing smithi15528762-32 from 3145728 to 3227648 tid 1 2024-04-19T01:05:29.672 INFO:tasks.rados.rados.0.smithi155.stdout:379: snap_create 2024-04-19T01:05:29.688 INFO:tasks.rados.rados.0.smithi155.stdout:378: finishing write tid 1 to smithi15528762-32 2024-04-19T01:05:29.688 INFO:tasks.rados.rados.0.smithi155.stdout:378: finishing write tid 2 to smithi15528762-32 2024-04-19T01:05:29.693 INFO:tasks.rados.rados.0.smithi155.stdout:378: finishing write tid 3 to smithi15528762-32 2024-04-19T01:05:29.693 INFO:tasks.rados.rados.0.smithi155.stdout:378: oid 32 updating version 0 to 154 2024-04-19T01:05:29.693 INFO:tasks.rados.rados.0.smithi155.stdout:378: oid 32 updating version 154 to 155 2024-04-19T01:05:29.693 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 32 v 155 (ObjNum 148 snap 44 seq_num 148) dirty exists 2024-04-19T01:05:29.693 INFO:tasks.rados.rados.0.smithi155.stdout:378: left oid 32 (ObjNum 148 snap 44 seq_num 148) 2024-04-19T01:05:29.765 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:05:29.764+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:05:29.835 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:29.835 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:29.849 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-19T01:05:29.866 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:05:29.866+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:05:29.967 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:05:29.965+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:05:29.969 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:29.969 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:29.984 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-19T01:05:30.067 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:05:30.066+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:05:30.152 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:30.152 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:30.167 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-19T01:05:30.168 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:05:30.168+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:05:30.269 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:05:30.268+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:05:30.292 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:30.292 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:30.306 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-04-19T01:05:30.371 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:05:30.366+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:05:30.425 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:30.425 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:30.443 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-04-19T01:05:30.472 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:05:30.470+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:05:30.568 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:30.568 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:30.573 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:05:30.569+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:05:30.583 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-04-19T01:05:30.632 INFO:tasks.rados.rados.0.smithi155.stdout:378: done (1 left) 2024-04-19T01:05:30.633 INFO:tasks.rados.rados.0.smithi155.stdout:379: done (0 left) 2024-04-19T01:05:30.633 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 1 current snap is 45 2024-04-19T01:05:30.633 INFO:tasks.rados.rados.0.smithi155.stdout:380: seq_num 149 ranges {0=98304} 2024-04-19T01:05:30.634 INFO:tasks.rados.rados.0.smithi155.stdout:380: writing smithi15528762-1 from 0 to 98304 tid 1 2024-04-19T01:05:30.634 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 36 current snap is 45 2024-04-19T01:05:30.634 INFO:tasks.rados.rados.0.smithi155.stdout:381: seq_num 150 ranges {2883584=114688} 2024-04-19T01:05:30.674 INFO:tasks.rados.rados.0.smithi155.stdout:381: writing smithi15528762-36 from 2883584 to 2998272 tid 1 2024-04-19T01:05:30.674 INFO:tasks.rados.rados.0.smithi155.stdout:382: snap_create 2024-04-19T01:05:30.674 INFO:tasks.rados.rados.0.smithi155.stdout:380: finishing write tid 1 to smithi15528762-1 2024-04-19T01:05:30.674 INFO:tasks.rados.rados.0.smithi155.stdout:380: finishing write tid 2 to smithi15528762-1 2024-04-19T01:05:30.674 INFO:tasks.rados.rados.0.smithi155.stdout:380: finishing write tid 3 to smithi15528762-1 2024-04-19T01:05:30.674 INFO:tasks.rados.rados.0.smithi155.stdout:380: oid 1 updating version 0 to 108 2024-04-19T01:05:30.674 INFO:tasks.rados.rados.0.smithi155.stdout:380: oid 1 version 108 is already newer than 107 2024-04-19T01:05:30.674 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 1 v 108 (ObjNum 149 snap 45 seq_num 149) dirty exists 2024-04-19T01:05:30.674 INFO:tasks.rados.rados.0.smithi155.stdout:380: left oid 1 (ObjNum 149 snap 45 seq_num 149) 2024-04-19T01:05:30.675 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:05:30.672+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:05:30.686 INFO:tasks.rados.rados.0.smithi155.stdout:381: finishing write tid 1 to smithi15528762-36 2024-04-19T01:05:30.687 INFO:tasks.rados.rados.0.smithi155.stdout:381: finishing write tid 2 to smithi15528762-36 2024-04-19T01:05:30.689 INFO:tasks.rados.rados.0.smithi155.stdout:381: finishing write tid 3 to smithi15528762-36 2024-04-19T01:05:30.690 INFO:tasks.rados.rados.0.smithi155.stdout:381: oid 36 updating version 0 to 88 2024-04-19T01:05:30.690 INFO:tasks.rados.rados.0.smithi155.stdout:381: oid 36 updating version 88 to 89 2024-04-19T01:05:30.690 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 36 v 89 (ObjNum 150 snap 45 seq_num 150) dirty exists 2024-04-19T01:05:30.690 INFO:tasks.rados.rados.0.smithi155.stdout:381: left oid 36 (ObjNum 150 snap 45 seq_num 150) 2024-04-19T01:05:30.748 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:30.748 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:30.763 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-19T01:05:30.775 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:05:30.772+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:05:30.877 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:05:30.878+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:05:30.892 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:30.892 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:30.906 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-19T01:05:30.971 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-19T01:05:31.018 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:05:31.017+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:05:31.028 INFO:teuthology.orchestra.run.smithi110.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:31.028 INFO:teuthology.orchestra.run.smithi110.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:31.042 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-19T01:05:31.078 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-04-19T01:05:31.076+0000 7f43c311e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25349) UID: 0 2024-04-19T01:05:31.131 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-04-19T01:05:31.131 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T01:05:31.151 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-04-19T01:05:31.151 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T01:05:31.163 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:05:31.164 INFO:teuthology.orchestra.run.smithi110.stdout: "size": 20, 2024-04-19T01:05:31.164 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 600, 2024-04-19T01:05:31.164 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [ 2024-04-19T01:05:31.164 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.164 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:683 3.0s0 3:07ce2446:::smithi15528762-30:head [append 0~458752] snapc 1f=[1f,1e,1c] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-19T01:05:31.164 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:01.581733+0000", 2024-04-19T01:05:31.164 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 29.579720232, 2024-04-19T01:05:31.164 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.046044901999999999, 2024-04-19T01:05:31.164 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:31.165 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:31.165 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:31.165 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:31.165 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:31.165 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:31.165 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 683 2024-04-19T01:05:31.165 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.165 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:31.165 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.165 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:31.165 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.581733+0000", 2024-04-19T01:05:31.165 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.166 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.166 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.166 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:31.166 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.581733+0000", 2024-04-19T01:05:31.166 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.166 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.166 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.166 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:31.166 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.581737+0000", 2024-04-19T01:05:31.166 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.3699999999999999e-06 2024-04-19T01:05:31.166 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.166 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.167 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:31.167 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.582569+0000", 2024-04-19T01:05:31.167 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00083236400000000004 2024-04-19T01:05:31.167 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.167 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.167 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:31.167 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.582606+0000", 2024-04-19T01:05:31.167 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.6992000000000001e-05 2024-04-19T01:05:31.167 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.167 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.167 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:31.167 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.582623+0000", 2024-04-19T01:05:31.168 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7088999999999999e-05 2024-04-19T01:05:31.168 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.168 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.168 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.168 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.582774+0000", 2024-04-19T01:05:31.168 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015108299999999999 2024-04-19T01:05:31.168 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.168 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.168 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "op must wait for map", 2024-04-19T01:05:31.168 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.582792+0000", 2024-04-19T01:05:31.168 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7336000000000001e-05 2024-04-19T01:05:31.169 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.169 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.169 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.169 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.592365+0000", 2024-04-19T01:05:31.169 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0095734059999999996 2024-04-19T01:05:31.169 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.169 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.169 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:31.169 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.592530+0000", 2024-04-19T01:05:31.169 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000165169 2024-04-19T01:05:31.169 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.169 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.169 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:31.170 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.594498+0000", 2024-04-19T01:05:31.170 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0019679279999999999 2024-04-19T01:05:31.170 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.170 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.170 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:31.170 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.626973+0000", 2024-04-19T01:05:31.170 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.032474513000000003 2024-04-19T01:05:31.170 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.170 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.170 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:31.170 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.627604+0000", 2024-04-19T01:05:31.170 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00063180400000000002 2024-04-19T01:05:31.171 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.171 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.171 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:31.171 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.627778+0000", 2024-04-19T01:05:31.171 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00017384799999999999 2024-04-19T01:05:31.171 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.171 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.171 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.171 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.171 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.172 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:684 3.0s0 3:07ce2446:::smithi15528762-30:head [append 0~507904] snapc 1f=[1f,1e,1c] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-19T01:05:31.172 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:01.587330+0000", 2024-04-19T01:05:31.172 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 29.574123553, 2024-04-19T01:05:31.172 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.041423123999999999, 2024-04-19T01:05:31.172 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:31.172 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:31.172 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:31.172 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:31.172 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:31.172 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:31.172 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 684 2024-04-19T01:05:31.172 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.172 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:31.173 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.173 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:31.173 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.587330+0000", 2024-04-19T01:05:31.173 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.173 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.173 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.173 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:31.173 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.587330+0000", 2024-04-19T01:05:31.173 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.173 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.173 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.173 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:31.174 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.587333+0000", 2024-04-19T01:05:31.174 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.1030000000000002e-06 2024-04-19T01:05:31.174 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.174 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.174 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:31.174 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.588112+0000", 2024-04-19T01:05:31.174 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00077844999999999998 2024-04-19T01:05:31.174 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.174 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.174 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:31.174 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.588139+0000", 2024-04-19T01:05:31.174 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.7616000000000001e-05 2024-04-19T01:05:31.175 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.175 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.175 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:31.175 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.588155+0000", 2024-04-19T01:05:31.175 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5432000000000001e-05 2024-04-19T01:05:31.175 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.175 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.175 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.175 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.588288+0000", 2024-04-19T01:05:31.175 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00013369400000000001 2024-04-19T01:05:31.175 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.175 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.175 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:05:31.176 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.588300+0000", 2024-04-19T01:05:31.176 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.2053e-05 2024-04-19T01:05:31.176 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.176 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.176 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.176 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.604734+0000", 2024-04-19T01:05:31.176 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.016433758999999999 2024-04-19T01:05:31.176 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.176 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.176 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:31.176 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.604928+0000", 2024-04-19T01:05:31.176 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00019333100000000001 2024-04-19T01:05:31.177 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.177 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.177 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:31.177 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.606556+0000", 2024-04-19T01:05:31.177 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0016286670000000001 2024-04-19T01:05:31.177 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.177 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.177 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:31.177 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.626976+0000", 2024-04-19T01:05:31.177 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.020420062999999999 2024-04-19T01:05:31.177 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.177 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.177 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:31.178 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.628638+0000", 2024-04-19T01:05:31.178 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001661495 2024-04-19T01:05:31.178 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.178 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.178 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:31.178 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.628753+0000", 2024-04-19T01:05:31.178 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000115461 2024-04-19T01:05:31.178 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.178 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.178 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.178 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.178 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.178 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:685 3.0s0 3:07ce2446:::smithi15528762-30:head [append 0~507904] snapc 1f=[1f,1e,1c] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-19T01:05:31.179 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:01.592396+0000", 2024-04-19T01:05:31.179 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 29.569058042999998, 2024-04-19T01:05:31.179 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.050032994999999997, 2024-04-19T01:05:31.179 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:31.179 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:31.179 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:31.179 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:31.179 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:31.179 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:31.179 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 685 2024-04-19T01:05:31.179 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.179 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:31.180 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.180 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:31.180 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.592396+0000", 2024-04-19T01:05:31.180 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.180 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.180 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.180 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:31.180 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.592396+0000", 2024-04-19T01:05:31.180 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.180 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.180 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.180 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:31.181 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.592398+0000", 2024-04-19T01:05:31.181 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2359999999999999e-06 2024-04-19T01:05:31.181 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.181 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.181 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:31.181 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.593152+0000", 2024-04-19T01:05:31.181 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00075385199999999995 2024-04-19T01:05:31.181 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.181 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.181 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:31.181 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.593176+0000", 2024-04-19T01:05:31.181 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4771999999999999e-05 2024-04-19T01:05:31.182 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.182 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.182 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:31.182 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.593192+0000", 2024-04-19T01:05:31.182 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5509e-05 2024-04-19T01:05:31.182 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.182 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.182 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.182 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.616030+0000", 2024-04-19T01:05:31.182 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.022837803 2024-04-19T01:05:31.182 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.182 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.182 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:31.183 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.616273+0000", 2024-04-19T01:05:31.183 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00024351900000000001 2024-04-19T01:05:31.183 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.183 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.183 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:31.183 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.617970+0000", 2024-04-19T01:05:31.183 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001696674 2024-04-19T01:05:31.183 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.183 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.183 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:31.183 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.628051+0000", 2024-04-19T01:05:31.183 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.010080505 2024-04-19T01:05:31.184 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.184 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.184 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:31.184 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.642308+0000", 2024-04-19T01:05:31.184 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.014257801000000001 2024-04-19T01:05:31.184 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.184 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.184 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:31.184 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.642429+0000", 2024-04-19T01:05:31.184 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000120324 2024-04-19T01:05:31.184 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.184 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.185 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.185 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.185 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.185 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:686 3.0s0 3:07ce2446:::smithi15528762-30:head [append 0~458752] snapc 1f=[1f,1e,1c] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-19T01:05:31.185 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:01.597293+0000", 2024-04-19T01:05:31.185 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 29.564160249, 2024-04-19T01:05:31.185 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.046212640999999999, 2024-04-19T01:05:31.185 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:31.185 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:31.185 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:31.185 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:31.185 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:31.185 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:31.186 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 686 2024-04-19T01:05:31.186 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.186 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:31.186 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.186 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:31.186 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.597293+0000", 2024-04-19T01:05:31.186 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.186 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.186 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.186 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:31.186 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.597293+0000", 2024-04-19T01:05:31.186 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.187 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.187 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.187 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:31.187 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.597296+0000", 2024-04-19T01:05:31.187 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.7499999999999999e-06 2024-04-19T01:05:31.187 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.187 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.187 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:31.187 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598220+0000", 2024-04-19T01:05:31.187 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00092368400000000003 2024-04-19T01:05:31.187 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.187 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.188 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:31.188 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598258+0000", 2024-04-19T01:05:31.188 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.7694000000000002e-05 2024-04-19T01:05:31.188 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.188 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.188 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:31.188 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598280+0000", 2024-04-19T01:05:31.188 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2626e-05 2024-04-19T01:05:31.188 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.188 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.188 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.188 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.629327+0000", 2024-04-19T01:05:31.188 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.03104641 2024-04-19T01:05:31.189 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.189 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.189 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:31.189 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.629495+0000", 2024-04-19T01:05:31.189 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00016847499999999999 2024-04-19T01:05:31.189 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.189 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.189 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:31.189 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.630802+0000", 2024-04-19T01:05:31.189 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0013073620000000001 2024-04-19T01:05:31.189 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.189 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.189 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:31.190 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.642744+0000", 2024-04-19T01:05:31.190 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.011941396 2024-04-19T01:05:31.190 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.190 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.190 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:31.190 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.643393+0000", 2024-04-19T01:05:31.190 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00064892900000000002 2024-04-19T01:05:31.190 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.190 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.190 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:31.190 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.643506+0000", 2024-04-19T01:05:31.190 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000113315 2024-04-19T01:05:31.191 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.191 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.191 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.191 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.191 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.191 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:687 3.0s0 3:07ce2446:::smithi15528762-30:head [append 0~81920] snapc 1f=[1f,1e,1c] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-19T01:05:31.191 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:01.598323+0000", 2024-04-19T01:05:31.191 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 29.563130687000001, 2024-04-19T01:05:31.191 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.048342318000000002, 2024-04-19T01:05:31.191 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:31.191 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:31.191 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:31.191 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:31.192 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:31.192 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:31.192 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 687 2024-04-19T01:05:31.192 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.192 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:31.192 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.192 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:31.192 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598323+0000", 2024-04-19T01:05:31.192 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.192 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.192 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.192 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:31.192 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598323+0000", 2024-04-19T01:05:31.193 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.193 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.193 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.193 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:31.193 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598325+0000", 2024-04-19T01:05:31.193 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.897e-06 2024-04-19T01:05:31.193 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.193 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.193 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:31.193 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598490+0000", 2024-04-19T01:05:31.193 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000164928 2024-04-19T01:05:31.193 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.193 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.194 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:31.194 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598525+0000", 2024-04-19T01:05:31.194 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.5342999999999997e-05 2024-04-19T01:05:31.194 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.194 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.194 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:31.194 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598545+0000", 2024-04-19T01:05:31.194 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0125000000000001e-05 2024-04-19T01:05:31.194 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.194 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.194 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.194 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.638867+0000", 2024-04-19T01:05:31.195 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.040321362999999999 2024-04-19T01:05:31.195 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.195 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.195 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:31.195 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.639029+0000", 2024-04-19T01:05:31.195 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00016188899999999999 2024-04-19T01:05:31.195 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.196 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.196 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:31.196 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.640093+0000", 2024-04-19T01:05:31.196 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001064307 2024-04-19T01:05:31.196 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.196 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.196 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:31.196 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.643531+0000", 2024-04-19T01:05:31.196 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0034386059999999999 2024-04-19T01:05:31.196 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.196 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.196 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:31.197 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.646439+0000", 2024-04-19T01:05:31.197 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0029075770000000002 2024-04-19T01:05:31.197 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.197 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.197 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:31.197 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.646665+0000", 2024-04-19T01:05:31.197 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000226283 2024-04-19T01:05:31.197 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.197 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.197 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.197 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.197 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.198 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:688 3.0s0 3:07ce2446:::smithi15528762-30:head [setxattr _header (57) in=7b] snapc 1f=[1f,1e,1c] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-19T01:05:31.198 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:01.598570+0000", 2024-04-19T01:05:31.198 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 29.562883488000001, 2024-04-19T01:05:31.198 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.049224946999999998, 2024-04-19T01:05:31.198 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:31.198 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:31.198 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:31.198 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:31.198 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:31.198 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:31.198 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 688 2024-04-19T01:05:31.198 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.199 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:31.199 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.199 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:31.199 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598570+0000", 2024-04-19T01:05:31.199 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.199 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.199 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.199 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:31.199 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598570+0000", 2024-04-19T01:05:31.199 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.199 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.199 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.199 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:31.200 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598572+0000", 2024-04-19T01:05:31.200 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6929999999999999e-06 2024-04-19T01:05:31.200 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.200 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.200 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:31.200 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598580+0000", 2024-04-19T01:05:31.200 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.4190000000000002e-06 2024-04-19T01:05:31.200 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.201 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.201 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:31.201 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598602+0000", 2024-04-19T01:05:31.201 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2133e-05 2024-04-19T01:05:31.201 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.201 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.201 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:31.201 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598617+0000", 2024-04-19T01:05:31.201 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.4576000000000001e-05 2024-04-19T01:05:31.201 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.201 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.201 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.201 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.644017+0000", 2024-04-19T01:05:31.202 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.045400082000000001 2024-04-19T01:05:31.202 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.202 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.202 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:31.202 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.644184+0000", 2024-04-19T01:05:31.202 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00016675500000000001 2024-04-19T01:05:31.202 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.202 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.202 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:31.202 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.644907+0000", 2024-04-19T01:05:31.202 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00072305000000000004 2024-04-19T01:05:31.202 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.203 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.203 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:31.203 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.647032+0000", 2024-04-19T01:05:31.203 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0021255940000000002 2024-04-19T01:05:31.203 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.203 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.203 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:31.206 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.647630+0000", 2024-04-19T01:05:31.206 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00059702599999999996 2024-04-19T01:05:31.206 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.206 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.206 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:31.206 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.647795+0000", 2024-04-19T01:05:31.206 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00016561899999999999 2024-04-19T01:05:31.206 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.206 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.207 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.207 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.207 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.207 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:689 3.0s0 3:07ce2446:::smithi15528762-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e91)", 2024-04-19T01:05:31.207 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:01.598635+0000", 2024-04-19T01:05:31.207 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 29.562818980999999, 2024-04-19T01:05:31.207 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.052904010000000001, 2024-04-19T01:05:31.207 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:31.207 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:31.207 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:05:31.207 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:31.207 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:31.208 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:31.208 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 689 2024-04-19T01:05:31.208 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.208 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:31.208 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.208 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:31.208 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598635+0000", 2024-04-19T01:05:31.208 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.208 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.208 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.208 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:31.208 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598635+0000", 2024-04-19T01:05:31.208 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.209 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.209 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.209 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:31.209 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598636+0000", 2024-04-19T01:05:31.209 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.516e-06 2024-04-19T01:05:31.209 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.209 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.209 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:31.209 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598642+0000", 2024-04-19T01:05:31.209 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.8100000000000003e-06 2024-04-19T01:05:31.209 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.209 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.210 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:31.210 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598664+0000", 2024-04-19T01:05:31.210 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1702e-05 2024-04-19T01:05:31.210 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.210 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.210 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:31.210 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.598679+0000", 2024-04-19T01:05:31.210 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5705999999999998e-05 2024-04-19T01:05:31.210 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.210 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.210 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.210 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.645587+0000", 2024-04-19T01:05:31.210 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.046907881999999998 2024-04-19T01:05:31.211 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.211 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.211 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting for rw locks", 2024-04-19T01:05:31.211 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.645846+0000", 2024-04-19T01:05:31.211 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00025909400000000002 2024-04-19T01:05:31.211 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.211 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.211 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.211 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.648563+0000", 2024-04-19T01:05:31.211 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0027163840000000001 2024-04-19T01:05:31.211 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.211 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.211 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:31.212 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.648857+0000", 2024-04-19T01:05:31.212 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00029466900000000001 2024-04-19T01:05:31.212 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.212 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.212 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:31.212 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.651539+0000", 2024-04-19T01:05:31.212 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0026812469999999999 2024-04-19T01:05:31.212 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.212 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.212 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.212 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.212 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.213 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:742 3.0s0 3:07ce2446:::smithi15528762-30:head [read 0~5718016,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-04-19T01:05:31.213 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:10.543083+0000", 2024-04-19T01:05:31.213 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 20.618370984999999, 2024-04-19T01:05:31.213 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.036955597, 2024-04-19T01:05:31.213 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:31.213 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:31.213 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:05:31.213 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:31.213 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:31.213 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:31.213 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 742 2024-04-19T01:05:31.213 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.213 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:31.214 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.214 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:31.214 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:10.543083+0000", 2024-04-19T01:05:31.214 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.214 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.214 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.214 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:31.214 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:10.543083+0000", 2024-04-19T01:05:31.214 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.214 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.214 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.214 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:31.215 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:10.543084+0000", 2024-04-19T01:05:31.215 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6869999999999999e-06 2024-04-19T01:05:31.215 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.215 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.215 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:31.215 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:10.543094+0000", 2024-04-19T01:05:31.215 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.7229999999999999e-06 2024-04-19T01:05:31.215 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.215 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.215 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:31.215 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:10.543115+0000", 2024-04-19T01:05:31.215 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0689000000000001e-05 2024-04-19T01:05:31.216 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.216 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.216 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:31.216 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:10.543131+0000", 2024-04-19T01:05:31.216 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6103000000000002e-05 2024-04-19T01:05:31.216 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.216 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.216 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.216 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:10.548504+0000", 2024-04-19T01:05:31.216 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.005373488 2024-04-19T01:05:31.216 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.216 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.216 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:31.217 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:10.548769+0000", 2024-04-19T01:05:31.217 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00026449299999999998 2024-04-19T01:05:31.217 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.217 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.217 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:31.217 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:10.580038+0000", 2024-04-19T01:05:31.217 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.031269414000000002 2024-04-19T01:05:31.217 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.217 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.217 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.217 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.217 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.218 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:752 3.0s0 3:07ce2446:::smithi15528762-30:head [append 0~753664] snapc 25=[25,24,22,21,1f] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-04-19T01:05:31.218 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:12.672965+0000", 2024-04-19T01:05:31.218 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 18.488488218000001, 2024-04-19T01:05:31.218 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.050758237999999997, 2024-04-19T01:05:31.218 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:31.218 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:31.218 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:31.218 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:31.218 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:31.218 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:31.218 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 752 2024-04-19T01:05:31.218 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.219 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:31.219 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.219 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:31.219 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.672965+0000", 2024-04-19T01:05:31.219 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.219 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.219 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.219 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:31.219 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.672965+0000", 2024-04-19T01:05:31.219 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.219 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.219 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.219 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:31.220 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.672968+0000", 2024-04-19T01:05:31.220 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.01e-06 2024-04-19T01:05:31.220 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.220 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.220 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:31.220 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.674437+0000", 2024-04-19T01:05:31.220 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001468949 2024-04-19T01:05:31.220 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.220 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.220 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:31.220 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.674481+0000", 2024-04-19T01:05:31.220 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.3424999999999998e-05 2024-04-19T01:05:31.221 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.221 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.221 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:31.221 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.674503+0000", 2024-04-19T01:05:31.221 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2331999999999999e-05 2024-04-19T01:05:31.221 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.221 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.221 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.221 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.682908+0000", 2024-04-19T01:05:31.221 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0084046859999999998 2024-04-19T01:05:31.221 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.221 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.221 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:31.222 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.683086+0000", 2024-04-19T01:05:31.222 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000177923 2024-04-19T01:05:31.222 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.222 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.222 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:31.222 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.685404+0000", 2024-04-19T01:05:31.222 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.002317843 2024-04-19T01:05:31.222 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.222 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.222 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:31.222 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.715273+0000", 2024-04-19T01:05:31.222 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.029869252999999998 2024-04-19T01:05:31.223 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.223 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.223 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:31.223 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.723523+0000", 2024-04-19T01:05:31.223 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0082503760000000002 2024-04-19T01:05:31.223 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.223 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.223 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:31.223 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.723724+0000", 2024-04-19T01:05:31.223 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00020044099999999999 2024-04-19T01:05:31.223 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.223 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.224 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.224 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.224 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.224 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:753 3.0s0 3:07ce2446:::smithi15528762-30:head [append 0~475136] snapc 25=[25,24,22,21,1f] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-04-19T01:05:31.224 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:12.675795+0000", 2024-04-19T01:05:31.224 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 18.485658763, 2024-04-19T01:05:31.224 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.049103989000000001, 2024-04-19T01:05:31.224 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:31.224 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:31.224 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:31.224 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:31.224 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:31.224 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:31.225 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 753 2024-04-19T01:05:31.225 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.225 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:31.225 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.225 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:31.225 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.675795+0000", 2024-04-19T01:05:31.225 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.225 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.225 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.225 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:31.225 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.675795+0000", 2024-04-19T01:05:31.225 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.226 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.226 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.226 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:31.226 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.675797+0000", 2024-04-19T01:05:31.226 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1500000000000002e-06 2024-04-19T01:05:31.226 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.226 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.226 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:31.226 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.676682+0000", 2024-04-19T01:05:31.226 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00088471899999999996 2024-04-19T01:05:31.226 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.226 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.227 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:31.227 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.676714+0000", 2024-04-19T01:05:31.227 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.1918999999999997e-05 2024-04-19T01:05:31.227 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.227 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.227 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:31.227 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.676732+0000", 2024-04-19T01:05:31.227 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.8196e-05 2024-04-19T01:05:31.227 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.227 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.227 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.227 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.701230+0000", 2024-04-19T01:05:31.228 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.024498220000000001 2024-04-19T01:05:31.228 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.228 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.228 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:31.228 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.701507+0000", 2024-04-19T01:05:31.228 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00027720000000000002 2024-04-19T01:05:31.228 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.228 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.228 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:31.228 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.703138+0000", 2024-04-19T01:05:31.228 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001630657 2024-04-19T01:05:31.228 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.228 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.229 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:31.229 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.724037+0000", 2024-04-19T01:05:31.229 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.020898856 2024-04-19T01:05:31.229 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.229 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.229 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:31.229 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.724688+0000", 2024-04-19T01:05:31.229 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00065100200000000003 2024-04-19T01:05:31.229 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.229 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.229 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:31.229 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.724899+0000", 2024-04-19T01:05:31.230 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00021107 2024-04-19T01:05:31.230 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.230 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.230 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.230 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.230 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.230 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:754 3.0s0 3:07ce2446:::smithi15528762-30:head [append 0~327680] snapc 25=[25,24,22,21,1f] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-04-19T01:05:31.230 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:12.677636+0000", 2024-04-19T01:05:31.230 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 18.483817302999999, 2024-04-19T01:05:31.230 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.054923912999999998, 2024-04-19T01:05:31.230 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:31.230 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:31.231 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:31.231 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:31.231 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:31.231 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:31.231 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 754 2024-04-19T01:05:31.231 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.231 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:31.231 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.231 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:31.231 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.677636+0000", 2024-04-19T01:05:31.231 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.232 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.232 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.232 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:31.232 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.677636+0000", 2024-04-19T01:05:31.232 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.232 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.232 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.232 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:31.232 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.677639+0000", 2024-04-19T01:05:31.232 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2189999999999998e-06 2024-04-19T01:05:31.232 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.233 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.233 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:31.233 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.678252+0000", 2024-04-19T01:05:31.233 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00061363499999999996 2024-04-19T01:05:31.233 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.233 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.233 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:31.233 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.678284+0000", 2024-04-19T01:05:31.233 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.1933000000000001e-05 2024-04-19T01:05:31.233 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.233 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.233 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:31.233 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.678303+0000", 2024-04-19T01:05:31.234 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.8785999999999999e-05 2024-04-19T01:05:31.234 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.234 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.234 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.234 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.715372+0000", 2024-04-19T01:05:31.234 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.037068893999999998 2024-04-19T01:05:31.234 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.234 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.234 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:31.234 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.715528+0000", 2024-04-19T01:05:31.234 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000156615 2024-04-19T01:05:31.234 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.235 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.235 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:31.235 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.716690+0000", 2024-04-19T01:05:31.235 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0011620020000000001 2024-04-19T01:05:31.235 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.235 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.235 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:31.235 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.724267+0000", 2024-04-19T01:05:31.235 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0075768379999999998 2024-04-19T01:05:31.235 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.235 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.235 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:31.236 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.732473+0000", 2024-04-19T01:05:31.236 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0082060359999999999 2024-04-19T01:05:31.236 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.236 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.236 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:31.236 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.732560+0000", 2024-04-19T01:05:31.236 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.6954999999999997e-05 2024-04-19T01:05:31.236 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.236 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.236 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.236 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.236 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.236 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:755 3.0s0 3:07ce2446:::smithi15528762-30:head [setxattr _header (57) in=7b] snapc 25=[25,24,22,21,1f] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-04-19T01:05:31.237 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:12.678330+0000", 2024-04-19T01:05:31.237 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 18.483123903999999, 2024-04-19T01:05:31.237 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.054932598999999999, 2024-04-19T01:05:31.237 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:31.237 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:31.237 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:31.237 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:31.237 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:31.237 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:31.237 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 755 2024-04-19T01:05:31.237 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.237 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:31.238 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.238 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:31.238 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.678330+0000", 2024-04-19T01:05:31.238 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.238 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.238 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.238 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:31.238 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.678330+0000", 2024-04-19T01:05:31.238 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.238 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.238 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.238 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:31.239 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.678331+0000", 2024-04-19T01:05:31.239 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7129999999999999e-06 2024-04-19T01:05:31.239 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.239 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.239 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:31.239 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.678341+0000", 2024-04-19T01:05:31.239 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.2459999999999992e-06 2024-04-19T01:05:31.239 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.239 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.239 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:31.239 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.678360+0000", 2024-04-19T01:05:31.239 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.8839999999999999e-05 2024-04-19T01:05:31.239 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.240 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.240 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:31.240 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.678374+0000", 2024-04-19T01:05:31.240 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.4466e-05 2024-04-19T01:05:31.240 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.240 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.240 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.240 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.725559+0000", 2024-04-19T01:05:31.241 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.047184798 2024-04-19T01:05:31.241 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.241 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.241 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:31.241 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.725794+0000", 2024-04-19T01:05:31.241 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00023519200000000001 2024-04-19T01:05:31.241 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.241 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.241 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:31.241 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.726914+0000", 2024-04-19T01:05:31.241 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0011197710000000001 2024-04-19T01:05:31.241 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.242 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.242 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:31.242 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.731904+0000", 2024-04-19T01:05:31.242 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0049900509999999997 2024-04-19T01:05:31.242 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.242 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.242 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:31.242 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.733105+0000", 2024-04-19T01:05:31.242 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001200872 2024-04-19T01:05:31.242 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.242 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.242 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:31.243 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.733262+0000", 2024-04-19T01:05:31.243 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015765000000000001 2024-04-19T01:05:31.243 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.243 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.243 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.243 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.243 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.243 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:756 3.0s0 3:07ce2446:::smithi15528762-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e102)", 2024-04-19T01:05:31.243 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:12.678397+0000", 2024-04-19T01:05:31.244 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 18.483057003999999, 2024-04-19T01:05:31.244 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.060086422, 2024-04-19T01:05:31.244 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:31.244 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:31.244 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:05:31.244 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:31.244 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:31.244 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:31.244 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 756 2024-04-19T01:05:31.244 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.244 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:31.244 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.245 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:31.245 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.678397+0000", 2024-04-19T01:05:31.245 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.245 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.245 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.245 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:31.245 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.678397+0000", 2024-04-19T01:05:31.245 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.245 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.245 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.245 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:31.245 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.678398+0000", 2024-04-19T01:05:31.245 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.4270000000000001e-06 2024-04-19T01:05:31.246 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.246 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.246 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:31.246 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.678404+0000", 2024-04-19T01:05:31.246 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.3330000000000002e-06 2024-04-19T01:05:31.246 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.246 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.246 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:31.246 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.678427+0000", 2024-04-19T01:05:31.246 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2373000000000001e-05 2024-04-19T01:05:31.246 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.246 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.247 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:31.247 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.678441+0000", 2024-04-19T01:05:31.247 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.4163e-05 2024-04-19T01:05:31.247 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.247 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.247 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.247 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.727558+0000", 2024-04-19T01:05:31.247 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.04911679 2024-04-19T01:05:31.247 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.247 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.247 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting for rw locks", 2024-04-19T01:05:31.247 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.727766+0000", 2024-04-19T01:05:31.247 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00020851800000000001 2024-04-19T01:05:31.248 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.248 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.248 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.248 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.733330+0000", 2024-04-19T01:05:31.248 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0055636649999999998 2024-04-19T01:05:31.248 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.248 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.248 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:31.248 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.733431+0000", 2024-04-19T01:05:31.248 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00010116 2024-04-19T01:05:31.248 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.248 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.249 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:31.249 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.738483+0000", 2024-04-19T01:05:31.249 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0050519930000000003 2024-04-19T01:05:31.249 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.249 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.249 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.249 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.249 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.249 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.ds3 102/98 ECSubWrite(tid=209, reqid=client.4604.0:759, at_version=102'79, trim_to=0'0, roll_forward_to=0'0))", 2024-04-19T01:05:31.249 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:12.700882+0000", 2024-04-19T01:05:31.249 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 18.460572024000001, 2024-04-19T01:05:31.249 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.038819439999999997, 2024-04-19T01:05:31.249 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:31.250 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:31.250 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:31.250 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:31.250 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.250 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:31.250 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.700882+0000", 2024-04-19T01:05:31.250 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.250 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.251 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.251 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:31.251 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.700882+0000", 2024-04-19T01:05:31.251 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.251 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.251 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.251 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:31.251 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.700883+0000", 2024-04-19T01:05:31.251 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.2139999999999999e-06 2024-04-19T01:05:31.251 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.251 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.251 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:31.251 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.701119+0000", 2024-04-19T01:05:31.252 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00023612199999999999 2024-04-19T01:05:31.252 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.252 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.252 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:31.252 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.701150+0000", 2024-04-19T01:05:31.252 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.0929000000000002e-05 2024-04-19T01:05:31.252 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.252 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.252 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:31.252 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.701173+0000", 2024-04-19T01:05:31.252 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2702e-05 2024-04-19T01:05:31.252 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.253 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.253 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.253 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.708364+0000", 2024-04-19T01:05:31.253 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0071914129999999998 2024-04-19T01:05:31.253 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.253 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.253 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:31.253 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.708396+0000", 2024-04-19T01:05:31.253 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.1755999999999999e-05 2024-04-19T01:05:31.253 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.253 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.253 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:31.254 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.739646+0000", 2024-04-19T01:05:31.254 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.031250003999999998 2024-04-19T01:05:31.254 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.254 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.254 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:31.254 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.739701+0000", 2024-04-19T01:05:31.254 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.5300000000000002e-05 2024-04-19T01:05:31.254 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.254 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.254 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.254 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.254 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.254 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:766 3.0s0 3:0e44b6f6:::smithi15528762-26:head [append 0~0] snapc 25=[25,24,22,21,1f] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-04-19T01:05:31.255 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:12.703390+0000", 2024-04-19T01:05:31.255 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 18.458063457000002, 2024-04-19T01:05:31.255 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.038681488999999999, 2024-04-19T01:05:31.255 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:31.255 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:31.255 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:31.255 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:31.255 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:31.255 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:31.255 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 766 2024-04-19T01:05:31.255 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.255 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:31.256 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.256 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:31.256 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.703390+0000", 2024-04-19T01:05:31.256 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.256 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.256 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.256 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:31.256 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.703390+0000", 2024-04-19T01:05:31.256 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.256 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.256 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.256 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:31.256 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.703392+0000", 2024-04-19T01:05:31.257 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.2929999999999999e-06 2024-04-19T01:05:31.257 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.257 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.257 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:31.257 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.703398+0000", 2024-04-19T01:05:31.257 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.1890000000000004e-06 2024-04-19T01:05:31.257 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.257 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.257 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:31.257 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.703413+0000", 2024-04-19T01:05:31.257 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5233e-05 2024-04-19T01:05:31.258 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.258 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.258 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:31.258 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.703420+0000", 2024-04-19T01:05:31.258 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.3300000000000001e-06 2024-04-19T01:05:31.258 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.258 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.258 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.258 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.738735+0000", 2024-04-19T01:05:31.258 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.035314421999999998 2024-04-19T01:05:31.259 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.259 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.259 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:31.259 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.738903+0000", 2024-04-19T01:05:31.259 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000168331 2024-04-19T01:05:31.259 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.259 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.259 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:31.259 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.739721+0000", 2024-04-19T01:05:31.259 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00081775100000000002 2024-04-19T01:05:31.259 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.259 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.260 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:31.260 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.741693+0000", 2024-04-19T01:05:31.260 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001971815 2024-04-19T01:05:31.260 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.260 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.260 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:31.260 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.741980+0000", 2024-04-19T01:05:31.260 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00028753200000000001 2024-04-19T01:05:31.260 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.260 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.260 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:31.260 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.742072+0000", 2024-04-19T01:05:31.261 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.1593000000000002e-05 2024-04-19T01:05:31.261 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.261 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.261 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.261 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.261 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.261 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.as5 107/31 ECSubWrite(tid=232, reqid=client.4604.0:804, at_version=107'102, trim_to=0'0, roll_forward_to=105'100))", 2024-04-19T01:05:31.261 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:17.556866+0000", 2024-04-19T01:05:31.261 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 13.604588089, 2024-04-19T01:05:31.261 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.038386942, 2024-04-19T01:05:31.261 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:31.261 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:31.262 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:31.262 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:31.262 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.262 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:31.262 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.556866+0000", 2024-04-19T01:05:31.262 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.262 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.262 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.262 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:31.262 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.556866+0000", 2024-04-19T01:05:31.262 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.262 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.262 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.263 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:31.263 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.556866+0000", 2024-04-19T01:05:31.263 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.8999999999999997e-07 2024-04-19T01:05:31.263 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.263 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.263 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:31.263 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.557310+0000", 2024-04-19T01:05:31.263 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00044435200000000002 2024-04-19T01:05:31.263 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.263 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.263 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:31.264 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.557344+0000", 2024-04-19T01:05:31.264 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.3989000000000003e-05 2024-04-19T01:05:31.264 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.264 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.264 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:31.264 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.557369+0000", 2024-04-19T01:05:31.264 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5018999999999999e-05 2024-04-19T01:05:31.264 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.264 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.264 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.264 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.557531+0000", 2024-04-19T01:05:31.264 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000161729 2024-04-19T01:05:31.265 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.265 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.265 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:31.265 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.557560+0000", 2024-04-19T01:05:31.265 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.8339000000000001e-05 2024-04-19T01:05:31.265 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.265 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.265 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:31.265 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.595132+0000", 2024-04-19T01:05:31.265 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.037572350999999997 2024-04-19T01:05:31.265 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.265 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.265 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:31.266 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.595253+0000", 2024-04-19T01:05:31.266 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000120673 2024-04-19T01:05:31.266 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.266 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.266 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.266 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.266 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.266 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.4s5 107/31 ECSubWrite(tid=196, reqid=client.4604.0:819, at_version=107'87, trim_to=0'0, roll_forward_to=102'85))", 2024-04-19T01:05:31.266 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:17.629622+0000", 2024-04-19T01:05:31.266 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 13.531831760999999, 2024-04-19T01:05:31.266 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.055609981000000003, 2024-04-19T01:05:31.266 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:31.267 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:31.267 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:31.267 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:31.267 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.267 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:31.267 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.629622+0000", 2024-04-19T01:05:31.267 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.267 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.267 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.267 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:31.267 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.629622+0000", 2024-04-19T01:05:31.267 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.267 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.268 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.268 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:31.268 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.629623+0000", 2024-04-19T01:05:31.268 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.8999999999999996e-07 2024-04-19T01:05:31.268 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.268 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.268 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:31.268 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.630537+0000", 2024-04-19T01:05:31.268 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00091434199999999995 2024-04-19T01:05:31.268 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.268 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.268 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:31.269 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.630579+0000", 2024-04-19T01:05:31.269 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.1554999999999998e-05 2024-04-19T01:05:31.269 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.269 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.269 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:31.269 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.630603+0000", 2024-04-19T01:05:31.269 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4266000000000001e-05 2024-04-19T01:05:31.269 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.269 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.269 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.269 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.630734+0000", 2024-04-19T01:05:31.269 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00013133200000000001 2024-04-19T01:05:31.269 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.270 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.270 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:31.270 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.630751+0000", 2024-04-19T01:05:31.270 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6569999999999999e-05 2024-04-19T01:05:31.270 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.270 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.270 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:31.270 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.685167+0000", 2024-04-19T01:05:31.270 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.054416348000000003 2024-04-19T01:05:31.270 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.270 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.270 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:31.271 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.685232+0000", 2024-04-19T01:05:31.271 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.4877999999999998e-05 2024-04-19T01:05:31.271 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.271 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.271 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.271 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.271 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.271 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.cs3 107/98 ECSubWrite(tid=298, reqid=client.4604.0:826, at_version=107'74, trim_to=0'0, roll_forward_to=105'72))", 2024-04-19T01:05:31.271 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:17.636797+0000", 2024-04-19T01:05:31.271 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 13.524656867999999, 2024-04-19T01:05:31.271 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.048337954000000002, 2024-04-19T01:05:31.271 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:31.271 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:31.272 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:31.272 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:31.272 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.272 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:31.272 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.636797+0000", 2024-04-19T01:05:31.272 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.272 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.272 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.272 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:31.272 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.636797+0000", 2024-04-19T01:05:31.272 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.272 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.273 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.273 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:31.273 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.636798+0000", 2024-04-19T01:05:31.273 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.26e-06 2024-04-19T01:05:31.273 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.273 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.273 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:31.273 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.637571+0000", 2024-04-19T01:05:31.273 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00077305899999999999 2024-04-19T01:05:31.273 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.273 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.273 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:31.273 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.637609+0000", 2024-04-19T01:05:31.274 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.8315000000000002e-05 2024-04-19T01:05:31.274 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.274 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.274 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:31.274 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.637632+0000", 2024-04-19T01:05:31.274 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2422999999999999e-05 2024-04-19T01:05:31.274 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.274 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.274 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.274 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.637791+0000", 2024-04-19T01:05:31.274 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015879499999999999 2024-04-19T01:05:31.274 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.275 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.275 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:31.275 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.637817+0000", 2024-04-19T01:05:31.275 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6716e-05 2024-04-19T01:05:31.275 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.275 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.275 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:31.275 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.685037+0000", 2024-04-19T01:05:31.275 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.047219868999999998 2024-04-19T01:05:31.275 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.275 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.275 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:31.275 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.685135+0000", 2024-04-19T01:05:31.276 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.7516999999999995e-05 2024-04-19T01:05:31.276 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.276 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.276 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.276 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.276 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.276 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.1s5 116/115 ECSubWrite(tid=75, reqid=client.4604.0:898, at_version=116'71, trim_to=0'0, roll_forward_to=0'0))", 2024-04-19T01:05:31.276 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:26.559275+0000", 2024-04-19T01:05:31.276 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 4.6021788109999999, 2024-04-19T01:05:31.276 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.064635593000000005, 2024-04-19T01:05:31.276 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:31.276 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:31.277 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:31.277 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:31.277 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.277 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:31.277 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.559275+0000", 2024-04-19T01:05:31.277 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.277 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.277 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.277 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:31.277 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.559275+0000", 2024-04-19T01:05:31.277 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.277 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.278 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.278 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:31.278 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.559277+0000", 2024-04-19T01:05:31.278 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7150000000000001e-06 2024-04-19T01:05:31.278 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.278 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.278 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:31.278 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.559613+0000", 2024-04-19T01:05:31.278 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00033684200000000001 2024-04-19T01:05:31.278 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.278 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.279 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:31.279 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.559643+0000", 2024-04-19T01:05:31.279 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.9587999999999999e-05 2024-04-19T01:05:31.279 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.279 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.279 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:31.279 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.559665+0000", 2024-04-19T01:05:31.279 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2331e-05 2024-04-19T01:05:31.279 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.279 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.279 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.279 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.559815+0000", 2024-04-19T01:05:31.279 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00014967600000000001 2024-04-19T01:05:31.280 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.280 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.280 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:31.280 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.559843+0000", 2024-04-19T01:05:31.280 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.7722999999999999e-05 2024-04-19T01:05:31.280 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.280 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.280 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:31.280 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.623857+0000", 2024-04-19T01:05:31.280 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.064014617999999995 2024-04-19T01:05:31.280 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.280 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.280 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:31.281 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.623910+0000", 2024-04-19T01:05:31.281 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.3100000000000003e-05 2024-04-19T01:05:31.281 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.281 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.281 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.281 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.281 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.281 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.1s5 116/115 ECSubWrite(tid=76, reqid=client.4604.0:899, at_version=116'72, trim_to=0'0, roll_forward_to=0'0))", 2024-04-19T01:05:31.281 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:26.574423+0000", 2024-04-19T01:05:31.281 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 4.5870309440000003, 2024-04-19T01:05:31.281 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.049678958000000002, 2024-04-19T01:05:31.281 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:31.281 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:31.282 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:31.282 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:31.282 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.282 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:31.282 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.574423+0000", 2024-04-19T01:05:31.282 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.282 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.282 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.282 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:31.282 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.574423+0000", 2024-04-19T01:05:31.282 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.282 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.283 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.284 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:31.284 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.574424+0000", 2024-04-19T01:05:31.284 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.615e-06 2024-04-19T01:05:31.284 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.285 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.285 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:31.285 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.574687+0000", 2024-04-19T01:05:31.285 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000262546 2024-04-19T01:05:31.285 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.285 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.285 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:31.285 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.574722+0000", 2024-04-19T01:05:31.285 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.5586000000000001e-05 2024-04-19T01:05:31.285 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.285 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.285 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:31.285 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.574742+0000", 2024-04-19T01:05:31.286 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9652000000000001e-05 2024-04-19T01:05:31.286 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.286 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.286 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.286 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.575788+0000", 2024-04-19T01:05:31.286 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0010459689999999999 2024-04-19T01:05:31.286 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.286 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.286 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:31.286 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.575813+0000", 2024-04-19T01:05:31.286 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4839999999999999e-05 2024-04-19T01:05:31.286 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.287 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.287 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:31.287 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.624066+0000", 2024-04-19T01:05:31.287 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.048252947999999997 2024-04-19T01:05:31.287 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.287 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.287 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:31.287 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.624102+0000", 2024-04-19T01:05:31.287 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.5802e-05 2024-04-19T01:05:31.287 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.287 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.287 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.287 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.288 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.288 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:05:31.290 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-04-19T01:05:31.291 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T01:05:31.291 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-04-19T01:05:31.291 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T01:05:31.292 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-19T01:05:31.292 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-04-19T01:05:31.293 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T01:05:31.293 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-04-19T01:05:31.293 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T01:05:31.294 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-04-19T01:05:31.294 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T01:05:31.294 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-04-19T01:05:31.294 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T01:05:31.295 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2024-04-19T01:05:31.295 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T01:05:31.296 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:05:31.286+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:05:31.314 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2024-04-19T01:05:31.314 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T01:05:31.336 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2024-04-19T01:05:31.337 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T01:05:31.359 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2024-04-19T01:05:31.360 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T01:05:31.379 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2024-04-19T01:05:31.379 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-19T01:05:31.390 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:05:31.390+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:05:31.400 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2024-04-19T01:05:31.400 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-19T01:05:31.423 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2024-04-19T01:05:31.424 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-19T01:05:31.425 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:05:31.425 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [], 2024-04-19T01:05:31.425 INFO:teuthology.orchestra.run.smithi110.stdout: "num_ops": 0 2024-04-19T01:05:31.425 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:05:31.439 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-19T01:05:31.446 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2024-04-19T01:05:31.446 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-19T01:05:31.510 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:05:31.509+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:05:31.558 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:05:31.559 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [], 2024-04-19T01:05:31.559 INFO:teuthology.orchestra.run.smithi110.stdout: "complaint_time": 30, 2024-04-19T01:05:31.559 INFO:teuthology.orchestra.run.smithi110.stdout: "num_blocked_ops": 0 2024-04-19T01:05:31.559 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:05:31.572 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-19T01:05:31.623 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:05:31.624+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:05:31.644 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-19T01:05:31.656 INFO:tasks.rados.rados.0.smithi155.stdout:380: done (2 left) 2024-04-19T01:05:31.657 INFO:tasks.rados.rados.0.smithi155.stdout:381: done (1 left) 2024-04-19T01:05:31.657 INFO:tasks.rados.rados.0.smithi155.stdout:382: done (0 left) 2024-04-19T01:05:31.657 INFO:tasks.rados.rados.0.smithi155.stdout:383: read oid 11 snap -1 2024-04-19T01:05:31.657 INFO:tasks.rados.rados.0.smithi155.stdout:383: expect (ObjNum 139 snap 39 seq_num 139) 2024-04-19T01:05:31.657 INFO:tasks.rados.rados.0.smithi155.stdout:384: snap_remove snap 44 2024-04-19T01:05:31.695 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:05:31.693+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:05:31.708 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:05:31.708 INFO:teuthology.orchestra.run.smithi110.stdout: "size": 20, 2024-04-19T01:05:31.709 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 600, 2024-04-19T01:05:31.709 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [ 2024-04-19T01:05:31.709 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.709 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs2 43/31 ECSubWrite(tid=47, reqid=client.4604.0:263, at_version=43'8, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:05:31.709 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.995028+0000", 2024-04-19T01:05:31.709 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 77.713383070000006, 2024-04-19T01:05:31.709 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.038142937000000002, 2024-04-19T01:05:31.709 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:31.709 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:31.709 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:31.709 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:31.709 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.709 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:31.709 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995028+0000", 2024-04-19T01:05:31.710 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.710 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.710 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.710 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:31.710 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995028+0000", 2024-04-19T01:05:31.710 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.710 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.710 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.710 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:31.710 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995029+0000", 2024-04-19T01:05:31.710 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.47e-07 2024-04-19T01:05:31.710 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.710 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.711 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:31.711 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995397+0000", 2024-04-19T01:05:31.711 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00036862599999999999 2024-04-19T01:05:31.711 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.711 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.711 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:31.711 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995437+0000", 2024-04-19T01:05:31.711 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.9564999999999998e-05 2024-04-19T01:05:31.711 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.711 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.711 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:31.711 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995461+0000", 2024-04-19T01:05:31.711 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4261999999999999e-05 2024-04-19T01:05:31.711 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.711 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.712 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.712 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995661+0000", 2024-04-19T01:05:31.712 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00019937500000000001 2024-04-19T01:05:31.712 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.712 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.712 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:31.712 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.995689+0000", 2024-04-19T01:05:31.712 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.8282000000000001e-05 2024-04-19T01:05:31.712 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.712 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.712 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:31.712 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.033059+0000", 2024-04-19T01:05:31.712 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.037370288000000002 2024-04-19T01:05:31.712 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.713 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.713 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:31.713 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.033171+0000", 2024-04-19T01:05:31.713 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000111692 2024-04-19T01:05:31.713 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.713 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.713 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.713 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.713 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.713 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs2 43/31 ECSubWrite(tid=48, reqid=client.4604.0:264, at_version=43'9, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:05:31.713 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.005925+0000", 2024-04-19T01:05:31.713 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 77.702486414000006, 2024-04-19T01:05:31.713 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.028007243000000001, 2024-04-19T01:05:31.713 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:31.713 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:31.714 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:31.714 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:31.714 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.714 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:31.714 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.005925+0000", 2024-04-19T01:05:31.714 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.714 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.714 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.714 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:31.714 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.005925+0000", 2024-04-19T01:05:31.714 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.714 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.714 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.714 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:31.715 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.005926+0000", 2024-04-19T01:05:31.715 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9700000000000002e-06 2024-04-19T01:05:31.715 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.715 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.715 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:31.715 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006164+0000", 2024-04-19T01:05:31.715 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00023784199999999999 2024-04-19T01:05:31.715 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.715 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.715 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:31.715 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006203+0000", 2024-04-19T01:05:31.715 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.9171999999999997e-05 2024-04-19T01:05:31.715 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.715 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.715 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:31.716 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006235+0000", 2024-04-19T01:05:31.716 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.1937999999999998e-05 2024-04-19T01:05:31.716 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.716 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.716 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.716 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.015013+0000", 2024-04-19T01:05:31.716 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0087775909999999995 2024-04-19T01:05:31.716 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.716 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.716 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:31.716 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.015043+0000", 2024-04-19T01:05:31.716 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.9739000000000001e-05 2024-04-19T01:05:31.716 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.716 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.717 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:31.717 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.033841+0000", 2024-04-19T01:05:31.717 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.018797861999999999 2024-04-19T01:05:31.717 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.717 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.717 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:31.717 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.033932+0000", 2024-04-19T01:05:31.717 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.1129000000000001e-05 2024-04-19T01:05:31.718 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.718 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.718 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.718 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.718 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.718 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs2 43/31 ECSubWrite(tid=49, reqid=client.4604.0:265, at_version=43'10, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:05:31.719 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.006295+0000", 2024-04-19T01:05:31.719 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 77.702115644000003, 2024-04-19T01:05:31.719 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.02899026, 2024-04-19T01:05:31.719 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:31.719 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:31.719 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:31.719 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:31.719 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.719 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:31.719 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006295+0000", 2024-04-19T01:05:31.719 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.719 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.719 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.719 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:31.719 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006295+0000", 2024-04-19T01:05:31.720 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.720 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.720 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.720 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:31.720 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006296+0000", 2024-04-19T01:05:31.720 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.1230000000000001e-06 2024-04-19T01:05:31.720 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.720 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.720 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:31.720 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006329+0000", 2024-04-19T01:05:31.720 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.2465000000000002e-05 2024-04-19T01:05:31.721 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.721 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.721 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:31.721 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006362+0000", 2024-04-19T01:05:31.721 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.3118999999999999e-05 2024-04-19T01:05:31.721 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.722 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.722 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:31.722 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.006401+0000", 2024-04-19T01:05:31.722 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.9347999999999997e-05 2024-04-19T01:05:31.722 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.722 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.722 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.722 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.033284+0000", 2024-04-19T01:05:31.722 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.026882553999999999 2024-04-19T01:05:31.722 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.722 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.722 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:31.723 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.033308+0000", 2024-04-19T01:05:31.723 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4295999999999999e-05 2024-04-19T01:05:31.723 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.723 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.723 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:31.723 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.035213+0000", 2024-04-19T01:05:31.723 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0019049749999999999 2024-04-19T01:05:31.723 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.723 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.723 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:31.723 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.035286+0000", 2024-04-19T01:05:31.723 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.2379999999999995e-05 2024-04-19T01:05:31.723 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.723 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.723 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.724 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.724 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.724 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs2 48/31 ECSubWrite(tid=76, reqid=client.4604.0:345, at_version=48'22, trim_to=0'0, roll_forward_to=47'19))", 2024-04-19T01:05:31.724 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.538229+0000", 2024-04-19T01:05:31.724 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 73.170181436999997, 2024-04-19T01:05:31.724 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.041621538999999999, 2024-04-19T01:05:31.724 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:31.724 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:31.724 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:31.724 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:31.724 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.724 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:31.724 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538229+0000", 2024-04-19T01:05:31.724 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.725 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.725 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.725 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:31.725 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538229+0000", 2024-04-19T01:05:31.725 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.725 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.725 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.725 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:31.725 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538232+0000", 2024-04-19T01:05:31.725 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1730000000000002e-06 2024-04-19T01:05:31.725 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.725 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.726 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:31.726 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538714+0000", 2024-04-19T01:05:31.726 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00048223299999999999 2024-04-19T01:05:31.726 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.726 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.726 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:31.726 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538755+0000", 2024-04-19T01:05:31.726 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.1044999999999998e-05 2024-04-19T01:05:31.726 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.726 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.726 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:31.726 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.538779+0000", 2024-04-19T01:05:31.726 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3828999999999998e-05 2024-04-19T01:05:31.727 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.727 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.727 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.727 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.539007+0000", 2024-04-19T01:05:31.727 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00022818400000000001 2024-04-19T01:05:31.727 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.727 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.727 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:31.727 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.539033+0000", 2024-04-19T01:05:31.727 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6455999999999999e-05 2024-04-19T01:05:31.727 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.727 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.727 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:31.727 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.579777+0000", 2024-04-19T01:05:31.727 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.040743133000000001 2024-04-19T01:05:31.728 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.728 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.728 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:31.728 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.579851+0000", 2024-04-19T01:05:31.728 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.4486000000000005e-05 2024-04-19T01:05:31.728 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.728 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.728 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.728 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.728 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.728 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs2 48/31 ECSubWrite(tid=78, reqid=client.4604.0:346, at_version=48'23, trim_to=0'0, roll_forward_to=48'21))", 2024-04-19T01:05:31.728 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.549034+0000", 2024-04-19T01:05:31.728 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 73.159377394000003, 2024-04-19T01:05:31.729 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.030907911999999999, 2024-04-19T01:05:31.729 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:31.729 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:31.729 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:31.729 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:31.729 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.729 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:31.729 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.549034+0000", 2024-04-19T01:05:31.729 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.729 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.729 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.729 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:31.729 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.549034+0000", 2024-04-19T01:05:31.729 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.729 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.730 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.730 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:31.730 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.549035+0000", 2024-04-19T01:05:31.730 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9429999999999999e-06 2024-04-19T01:05:31.730 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.730 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.730 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:31.730 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.549509+0000", 2024-04-19T01:05:31.730 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00047399999999999997 2024-04-19T01:05:31.730 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.730 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.730 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:31.730 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.549541+0000", 2024-04-19T01:05:31.730 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.1843e-05 2024-04-19T01:05:31.731 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.731 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.731 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:31.731 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.549563+0000", 2024-04-19T01:05:31.731 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2045e-05 2024-04-19T01:05:31.731 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.731 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.731 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.732 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.558487+0000", 2024-04-19T01:05:31.732 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0089238470000000004 2024-04-19T01:05:31.732 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.732 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.732 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:31.732 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.558508+0000", 2024-04-19T01:05:31.732 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1236000000000001e-05 2024-04-19T01:05:31.732 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.732 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.733 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:31.733 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.579870+0000", 2024-04-19T01:05:31.733 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.021361601000000001 2024-04-19T01:05:31.733 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.733 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.733 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:31.733 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.579941+0000", 2024-04-19T01:05:31.733 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.1396999999999994e-05 2024-04-19T01:05:31.733 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.733 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.733 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.733 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.733 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.733 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.fs2 48/31 ECSubWrite(tid=79, reqid=client.4604.0:347, at_version=48'24, trim_to=0'0, roll_forward_to=48'21))", 2024-04-19T01:05:31.733 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:18.555772+0000", 2024-04-19T01:05:31.734 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 73.152638568, 2024-04-19T01:05:31.734 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.024949988999999999, 2024-04-19T01:05:31.734 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:31.734 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:31.734 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:31.734 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:31.734 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.734 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:31.734 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.555772+0000", 2024-04-19T01:05:31.734 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.734 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.734 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.734 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:31.734 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.555772+0000", 2024-04-19T01:05:31.734 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.735 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.735 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.735 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:31.735 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.555774+0000", 2024-04-19T01:05:31.735 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.734e-06 2024-04-19T01:05:31.735 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.735 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.735 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:31.735 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.555841+0000", 2024-04-19T01:05:31.735 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.6434999999999997e-05 2024-04-19T01:05:31.735 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.735 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.735 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:31.735 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.555877+0000", 2024-04-19T01:05:31.736 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.6544000000000001e-05 2024-04-19T01:05:31.736 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.736 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.736 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:31.736 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.555895+0000", 2024-04-19T01:05:31.736 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7558999999999999e-05 2024-04-19T01:05:31.736 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.736 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.736 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.736 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.576874+0000", 2024-04-19T01:05:31.736 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.020979564999999999 2024-04-19T01:05:31.736 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.736 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.736 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:31.736 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.576898+0000", 2024-04-19T01:05:31.737 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3787000000000001e-05 2024-04-19T01:05:31.737 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.737 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.737 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:31.737 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.580616+0000", 2024-04-19T01:05:31.737 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0037177289999999999 2024-04-19T01:05:31.737 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.737 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.737 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:31.737 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:18.580722+0000", 2024-04-19T01:05:31.737 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000106636 2024-04-19T01:05:31.737 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.737 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.737 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.737 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.738 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.738 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.5s6 59/31 ECSubWrite(tid=66, reqid=client.4604.0:423, at_version=59'51, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:05:31.738 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:29.540851+0000", 2024-04-19T01:05:31.738 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 62.167559902999997, 2024-04-19T01:05:31.738 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.030615177, 2024-04-19T01:05:31.738 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:31.738 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:31.738 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:31.738 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:31.738 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.738 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:31.738 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.540851+0000", 2024-04-19T01:05:31.738 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.738 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.739 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.739 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:31.739 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.540851+0000", 2024-04-19T01:05:31.739 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.739 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.739 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.739 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:31.739 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.540852+0000", 2024-04-19T01:05:31.739 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.6400000000000002e-07 2024-04-19T01:05:31.739 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.739 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.739 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:31.739 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.541398+0000", 2024-04-19T01:05:31.739 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00054620600000000004 2024-04-19T01:05:31.739 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.740 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.740 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:31.740 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.541433+0000", 2024-04-19T01:05:31.740 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.4853e-05 2024-04-19T01:05:31.740 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.740 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.740 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:31.740 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.541457+0000", 2024-04-19T01:05:31.740 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4641999999999999e-05 2024-04-19T01:05:31.740 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.740 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.740 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.740 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.541639+0000", 2024-04-19T01:05:31.740 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00018139100000000001 2024-04-19T01:05:31.740 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.741 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.741 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:31.741 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.541667+0000", 2024-04-19T01:05:31.741 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.7943e-05 2024-04-19T01:05:31.741 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.741 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.741 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:31.741 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.571413+0000", 2024-04-19T01:05:31.741 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0297462 2024-04-19T01:05:31.741 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.742 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.742 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:31.742 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:29.571466+0000", 2024-04-19T01:05:31.742 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.3378000000000003e-05 2024-04-19T01:05:31.742 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.742 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.743 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.743 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.743 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.743 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.8s6 60/31 ECSubWrite(tid=225, reqid=client.4604.0:433, at_version=60'43, trim_to=0'0, roll_forward_to=59'41))", 2024-04-19T01:05:31.743 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:30.569927+0000", 2024-04-19T01:05:31.743 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 61.138483862999998, 2024-04-19T01:05:31.743 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.027044315999999999, 2024-04-19T01:05:31.743 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:31.743 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:31.743 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:31.743 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:31.743 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.743 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:31.743 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.569927+0000", 2024-04-19T01:05:31.744 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.744 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.744 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.744 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:31.744 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.569927+0000", 2024-04-19T01:05:31.744 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.744 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.744 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.744 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:31.744 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.569927+0000", 2024-04-19T01:05:31.744 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.34e-07 2024-04-19T01:05:31.744 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.744 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.744 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:31.744 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.570430+0000", 2024-04-19T01:05:31.745 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00050269400000000004 2024-04-19T01:05:31.745 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.745 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.745 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:31.745 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.570465+0000", 2024-04-19T01:05:31.745 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.4898999999999999e-05 2024-04-19T01:05:31.745 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.745 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.745 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:31.745 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.570490+0000", 2024-04-19T01:05:31.745 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4805999999999999e-05 2024-04-19T01:05:31.745 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.745 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.745 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.745 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.570639+0000", 2024-04-19T01:05:31.746 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00014890899999999999 2024-04-19T01:05:31.746 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.746 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.746 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:31.746 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.570665+0000", 2024-04-19T01:05:31.746 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5825000000000001e-05 2024-04-19T01:05:31.746 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.746 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.746 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:31.746 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.596871+0000", 2024-04-19T01:05:31.746 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.026206291999999999 2024-04-19T01:05:31.746 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.746 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.746 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:31.747 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:30.596971+0000", 2024-04-19T01:05:31.747 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000100457 2024-04-19T01:05:31.747 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.747 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.747 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.747 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.747 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.747 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.5s6 65/31 ECSubWrite(tid=75, reqid=client.4604.0:486, at_version=65'56, trim_to=0'0, roll_forward_to=59'54))", 2024-04-19T01:05:31.747 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:35.660833+0000", 2024-04-19T01:05:31.747 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 56.047577709000002, 2024-04-19T01:05:31.747 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.026740791999999999, 2024-04-19T01:05:31.747 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:31.747 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:31.747 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:31.747 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:31.748 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.748 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:31.748 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.660833+0000", 2024-04-19T01:05:31.748 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.748 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.748 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.748 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:31.748 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.660833+0000", 2024-04-19T01:05:31.748 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.748 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.748 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.748 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:31.748 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.660834+0000", 2024-04-19T01:05:31.748 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.7299999999999995e-07 2024-04-19T01:05:31.748 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.749 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.749 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:31.749 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.661764+0000", 2024-04-19T01:05:31.749 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00092978500000000003 2024-04-19T01:05:31.749 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.749 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.749 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:31.749 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.661794+0000", 2024-04-19T01:05:31.749 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.021e-05 2024-04-19T01:05:31.749 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.749 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.749 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:31.749 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.661810+0000", 2024-04-19T01:05:31.749 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6082e-05 2024-04-19T01:05:31.749 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.750 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.750 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.750 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.662014+0000", 2024-04-19T01:05:31.750 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000203625 2024-04-19T01:05:31.750 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.750 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.750 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:31.750 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.662030+0000", 2024-04-19T01:05:31.750 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6293000000000001e-05 2024-04-19T01:05:31.750 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.750 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.750 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:31.750 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.687509+0000", 2024-04-19T01:05:31.752 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.025479485999999999 2024-04-19T01:05:31.752 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.752 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.752 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:31.752 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:35.687574+0000", 2024-04-19T01:05:31.752 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.4638e-05 2024-04-19T01:05:31.752 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.752 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.752 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.752 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.752 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.752 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.5s6 74/31 ECSubWrite(tid=88, reqid=osd.8.0:88, at_version=74'69, trim_to=0'0, roll_forward_to=72'65))", 2024-04-19T01:05:31.752 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:44.559798+0000", 2024-04-19T01:05:31.753 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 47.148612878999998, 2024-04-19T01:05:31.753 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.024864188999999998, 2024-04-19T01:05:31.753 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:31.753 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:31.753 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:31.753 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:31.753 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.753 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:31.753 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.559798+0000", 2024-04-19T01:05:31.753 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.753 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.753 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.753 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:31.753 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.559798+0000", 2024-04-19T01:05:31.754 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.754 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.754 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.754 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:31.754 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.559800+0000", 2024-04-19T01:05:31.754 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.88e-06 2024-04-19T01:05:31.754 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.754 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.754 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:31.754 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.559851+0000", 2024-04-19T01:05:31.754 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.0739000000000003e-05 2024-04-19T01:05:31.754 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.754 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.754 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:31.755 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.559882+0000", 2024-04-19T01:05:31.755 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.1038999999999999e-05 2024-04-19T01:05:31.755 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.755 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.755 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:31.755 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.559908+0000", 2024-04-19T01:05:31.755 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6460000000000001e-05 2024-04-19T01:05:31.755 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.755 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.755 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.755 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.564678+0000", 2024-04-19T01:05:31.755 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0047699079999999998 2024-04-19T01:05:31.755 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.755 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.756 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:31.756 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.564694+0000", 2024-04-19T01:05:31.756 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6056e-05 2024-04-19T01:05:31.756 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.756 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.756 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:31.756 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.584573+0000", 2024-04-19T01:05:31.756 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.019878939000000002 2024-04-19T01:05:31.756 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.756 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.756 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:31.756 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.584662+0000", 2024-04-19T01:05:31.756 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.9167999999999997e-05 2024-04-19T01:05:31.756 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.757 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.757 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.757 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.757 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.757 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.5s6 74/31 ECSubWrite(tid=89, reqid=client.4604.0:544, at_version=74'70, trim_to=0'0, roll_forward_to=72'65))", 2024-04-19T01:05:31.757 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:44.566121+0000", 2024-04-19T01:05:31.757 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 47.142290373000002, 2024-04-19T01:05:31.757 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.026764973000000001, 2024-04-19T01:05:31.757 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:31.757 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:31.757 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:31.757 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:31.757 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.757 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:31.758 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.566121+0000", 2024-04-19T01:05:31.758 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.758 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.758 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.758 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:31.758 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.566121+0000", 2024-04-19T01:05:31.758 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.758 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.758 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.758 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:31.758 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.566123+0000", 2024-04-19T01:05:31.758 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0770000000000002e-06 2024-04-19T01:05:31.758 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.758 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.759 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:31.759 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.566618+0000", 2024-04-19T01:05:31.759 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000495296 2024-04-19T01:05:31.759 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.759 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.759 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:31.759 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.566651+0000", 2024-04-19T01:05:31.759 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.2795e-05 2024-04-19T01:05:31.759 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.759 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.759 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:31.759 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.566673+0000", 2024-04-19T01:05:31.759 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1926999999999999e-05 2024-04-19T01:05:31.759 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.759 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.760 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.760 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.566823+0000", 2024-04-19T01:05:31.760 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015024999999999999 2024-04-19T01:05:31.760 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.760 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.760 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:31.760 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.566852+0000", 2024-04-19T01:05:31.760 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.9493e-05 2024-04-19T01:05:31.760 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.760 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.760 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:31.760 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.592770+0000", 2024-04-19T01:05:31.760 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.025917557000000001 2024-04-19T01:05:31.760 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.761 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.761 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:31.761 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.592886+0000", 2024-04-19T01:05:31.761 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000115578 2024-04-19T01:05:31.761 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.761 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.761 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.761 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.761 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.761 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.es2 86/31 ECSubWrite(tid=195, reqid=client.4604.0:653, at_version=86'39, trim_to=0'0, roll_forward_to=84'38))", 2024-04-19T01:05:31.761 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:56.572591+0000", 2024-04-19T01:05:31.761 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 35.13582023, 2024-04-19T01:05:31.761 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.030973436999999999, 2024-04-19T01:05:31.762 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:31.762 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:31.762 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:31.762 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:31.762 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.762 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:31.762 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.572591+0000", 2024-04-19T01:05:31.762 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.762 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.762 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.763 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:31.763 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.572591+0000", 2024-04-19T01:05:31.763 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.763 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.763 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.763 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:31.763 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.572592+0000", 2024-04-19T01:05:31.763 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.004e-06 2024-04-19T01:05:31.763 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.763 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.763 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:31.763 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.573633+0000", 2024-04-19T01:05:31.763 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0010417340000000001 2024-04-19T01:05:31.763 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.764 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.764 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:31.764 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.573675+0000", 2024-04-19T01:05:31.764 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.1347999999999998e-05 2024-04-19T01:05:31.764 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.764 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.764 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:31.764 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.573704+0000", 2024-04-19T01:05:31.764 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.8781000000000001e-05 2024-04-19T01:05:31.764 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.764 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.764 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.764 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.573872+0000", 2024-04-19T01:05:31.764 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00016797900000000001 2024-04-19T01:05:31.764 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.765 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.765 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:31.765 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.573888+0000", 2024-04-19T01:05:31.765 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6164e-05 2024-04-19T01:05:31.765 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.765 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.765 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:31.765 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.603491+0000", 2024-04-19T01:05:31.765 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.029603401000000001 2024-04-19T01:05:31.765 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.765 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.765 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:31.765 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.603564+0000", 2024-04-19T01:05:31.765 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.3026000000000002e-05 2024-04-19T01:05:31.766 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.766 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.766 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.766 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.766 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.766 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.2s1 86/31 ECSubWrite(tid=265, reqid=client.4604.0:655, at_version=86'35, trim_to=0'0, roll_forward_to=86'33))", 2024-04-19T01:05:31.766 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:56.580658+0000", 2024-04-19T01:05:31.766 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 35.127752934, 2024-04-19T01:05:31.766 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.030122023000000001, 2024-04-19T01:05:31.766 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:31.766 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:31.766 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:31.766 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:31.766 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.767 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:31.767 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.580658+0000", 2024-04-19T01:05:31.767 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.767 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.767 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.767 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:31.767 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.580658+0000", 2024-04-19T01:05:31.767 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.767 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.767 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.767 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:31.767 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.580660+0000", 2024-04-19T01:05:31.767 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.8169999999999999e-06 2024-04-19T01:05:31.767 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.767 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.768 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:31.768 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.581247+0000", 2024-04-19T01:05:31.768 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00058683699999999999 2024-04-19T01:05:31.768 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.768 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.768 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:31.768 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.581276+0000", 2024-04-19T01:05:31.768 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.9349999999999999e-05 2024-04-19T01:05:31.768 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.768 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.768 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:31.768 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.581297+0000", 2024-04-19T01:05:31.768 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1027e-05 2024-04-19T01:05:31.769 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.769 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.769 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.769 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.581466+0000", 2024-04-19T01:05:31.769 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00016856799999999999 2024-04-19T01:05:31.769 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.769 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.769 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:31.769 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.581495+0000", 2024-04-19T01:05:31.769 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.959e-05 2024-04-19T01:05:31.769 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.769 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.769 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:31.769 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.610709+0000", 2024-04-19T01:05:31.769 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.029214107 2024-04-19T01:05:31.770 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.770 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.770 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:31.770 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.610780+0000", 2024-04-19T01:05:31.770 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.0727000000000002e-05 2024-04-19T01:05:31.770 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.770 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.770 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.770 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.770 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.770 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.5s6 107/31 ECSubWrite(tid=213, reqid=client.4604.0:811, at_version=107'200, trim_to=0'0, roll_forward_to=106'199))", 2024-04-19T01:05:31.770 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:17.592673+0000", 2024-04-19T01:05:31.770 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 14.115738249, 2024-04-19T01:05:31.770 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.048811757999999997, 2024-04-19T01:05:31.771 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:31.771 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:31.771 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:31.771 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:31.771 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.771 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:31.771 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.592673+0000", 2024-04-19T01:05:31.771 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.771 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.771 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.771 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:31.771 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.592673+0000", 2024-04-19T01:05:31.771 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.771 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.772 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.772 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:31.772 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.592673+0000", 2024-04-19T01:05:31.772 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.6000000000000003e-07 2024-04-19T01:05:31.772 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.772 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.772 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:31.772 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.593319+0000", 2024-04-19T01:05:31.773 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00064555000000000005 2024-04-19T01:05:31.773 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.773 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.773 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:31.773 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.593364+0000", 2024-04-19T01:05:31.773 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.5305000000000001e-05 2024-04-19T01:05:31.773 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.773 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.773 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:31.773 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.593390+0000", 2024-04-19T01:05:31.773 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6169e-05 2024-04-19T01:05:31.773 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.773 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.773 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.773 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.593516+0000", 2024-04-19T01:05:31.774 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0001261 2024-04-19T01:05:31.774 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.774 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.774 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:31.774 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.593536+0000", 2024-04-19T01:05:31.774 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9548999999999999e-05 2024-04-19T01:05:31.774 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.774 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.774 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:31.774 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.641400+0000", 2024-04-19T01:05:31.774 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.047864064999999997 2024-04-19T01:05:31.774 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.774 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.775 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:31.775 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.641484+0000", 2024-04-19T01:05:31.775 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.4359999999999999e-05 2024-04-19T01:05:31.775 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.775 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.775 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.775 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.775 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.775 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.5s6 107/31 ECSubWrite(tid=214, reqid=client.4604.0:812, at_version=107'201, trim_to=0'0, roll_forward_to=106'199))", 2024-04-19T01:05:31.775 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:17.599682+0000", 2024-04-19T01:05:31.775 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 14.108729414000001, 2024-04-19T01:05:31.775 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.041863655, 2024-04-19T01:05:31.775 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:31.775 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:31.775 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:31.776 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:31.776 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.776 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:31.776 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.599682+0000", 2024-04-19T01:05:31.776 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.776 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.776 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.776 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:31.776 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.599682+0000", 2024-04-19T01:05:31.776 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.776 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.777 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.777 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:31.777 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.599684+0000", 2024-04-19T01:05:31.777 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0959999999999999e-06 2024-04-19T01:05:31.777 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.777 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.777 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:31.777 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.600168+0000", 2024-04-19T01:05:31.777 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000484492 2024-04-19T01:05:31.777 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.777 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.777 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:31.777 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.600213+0000", 2024-04-19T01:05:31.777 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.4697999999999998e-05 2024-04-19T01:05:31.778 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.778 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.778 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:31.778 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.600237+0000", 2024-04-19T01:05:31.778 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.41e-05 2024-04-19T01:05:31.778 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.778 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.778 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.778 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.606659+0000", 2024-04-19T01:05:31.778 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0064220930000000002 2024-04-19T01:05:31.778 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.778 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.778 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:31.778 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.606679+0000", 2024-04-19T01:05:31.779 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0035e-05 2024-04-19T01:05:31.779 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.779 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.779 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:31.779 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.641492+0000", 2024-04-19T01:05:31.779 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.034813114999999999 2024-04-19T01:05:31.779 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.779 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.779 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:31.780 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.641545+0000", 2024-04-19T01:05:31.780 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.3025999999999997e-05 2024-04-19T01:05:31.780 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.780 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.780 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.780 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.780 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.780 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.5s6 107/31 ECSubWrite(tid=215, reqid=client.4604.0:813, at_version=107'202, trim_to=0'0, roll_forward_to=106'199))", 2024-04-19T01:05:31.780 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:17.606184+0000", 2024-04-19T01:05:31.780 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 14.10222652, 2024-04-19T01:05:31.780 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.035422502000000002, 2024-04-19T01:05:31.780 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:31.780 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:31.781 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:31.781 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:31.781 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.781 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:31.781 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.606184+0000", 2024-04-19T01:05:31.781 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.781 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.781 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.781 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:31.781 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.606184+0000", 2024-04-19T01:05:31.781 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.781 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.781 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.781 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:31.782 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.606186+0000", 2024-04-19T01:05:31.782 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.753e-06 2024-04-19T01:05:31.782 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.782 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.782 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:31.782 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.606582+0000", 2024-04-19T01:05:31.782 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000395447 2024-04-19T01:05:31.782 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.783 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.783 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:31.783 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.606618+0000", 2024-04-19T01:05:31.783 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.6179e-05 2024-04-19T01:05:31.783 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.783 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.783 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:31.783 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.606665+0000", 2024-04-19T01:05:31.783 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.7008999999999997e-05 2024-04-19T01:05:31.783 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.783 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.783 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.783 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.622904+0000", 2024-04-19T01:05:31.783 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.016239388 2024-04-19T01:05:31.784 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.784 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.784 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:31.784 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.622937+0000", 2024-04-19T01:05:31.784 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.2579000000000001e-05 2024-04-19T01:05:31.784 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.784 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.784 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:31.784 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.641552+0000", 2024-04-19T01:05:31.784 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.018615352000000002 2024-04-19T01:05:31.784 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.784 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.784 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:31.784 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.641607+0000", 2024-04-19T01:05:31.785 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.4795e-05 2024-04-19T01:05:31.785 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.785 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.785 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.785 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.785 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.785 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.5s6 107/31 ECSubWrite(tid=216, reqid=client.4604.0:814, at_version=107'203, trim_to=0'0, roll_forward_to=106'199))", 2024-04-19T01:05:31.785 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:17.610716+0000", 2024-04-19T01:05:31.785 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 14.097694800999999, 2024-04-19T01:05:31.785 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.032034477999999998, 2024-04-19T01:05:31.785 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:31.785 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:31.785 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:31.785 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:31.786 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.786 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:31.786 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.610716+0000", 2024-04-19T01:05:31.786 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.786 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.786 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.786 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:31.786 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.610716+0000", 2024-04-19T01:05:31.786 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.786 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.786 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.786 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:31.786 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.610718+0000", 2024-04-19T01:05:31.787 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2929999999999999e-06 2024-04-19T01:05:31.787 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.787 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.787 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:31.787 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.610978+0000", 2024-04-19T01:05:31.787 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00025916200000000001 2024-04-19T01:05:31.787 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.787 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.787 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:31.787 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.611010+0000", 2024-04-19T01:05:31.787 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.2563e-05 2024-04-19T01:05:31.787 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.787 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.787 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:31.788 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.611026+0000", 2024-04-19T01:05:31.788 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5988999999999999e-05 2024-04-19T01:05:31.788 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.788 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.788 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.788 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.632278+0000", 2024-04-19T01:05:31.788 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.021251563000000001 2024-04-19T01:05:31.788 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.788 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.788 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:31.788 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.632293+0000", 2024-04-19T01:05:31.788 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5203e-05 2024-04-19T01:05:31.788 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.788 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.789 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:31.789 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.642673+0000", 2024-04-19T01:05:31.789 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.010380557 2024-04-19T01:05:31.789 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.789 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.789 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:31.789 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.642751+0000", 2024-04-19T01:05:31.789 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.7148000000000001e-05 2024-04-19T01:05:31.789 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.789 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.789 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.789 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.789 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.789 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.5s6 107/31 ECSubWrite(tid=217, reqid=client.4604.0:815, at_version=107'204, trim_to=0'0, roll_forward_to=106'199))", 2024-04-19T01:05:31.789 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:17.613483+0000", 2024-04-19T01:05:31.790 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 14.094928105999999, 2024-04-19T01:05:31.790 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.029331546999999999, 2024-04-19T01:05:31.790 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:31.790 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:31.790 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:31.790 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:31.790 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.790 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:31.790 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.613483+0000", 2024-04-19T01:05:31.790 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.790 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.790 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.790 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:31.791 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.613483+0000", 2024-04-19T01:05:31.791 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.791 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.791 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.791 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:31.791 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.613485+0000", 2024-04-19T01:05:31.791 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9300000000000002e-06 2024-04-19T01:05:31.791 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.791 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.791 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:31.791 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.613529+0000", 2024-04-19T01:05:31.791 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.3922999999999999e-05 2024-04-19T01:05:31.791 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.791 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.791 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:31.792 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.613577+0000", 2024-04-19T01:05:31.792 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.8668000000000002e-05 2024-04-19T01:05:31.792 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.792 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.792 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:31.792 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.613602+0000", 2024-04-19T01:05:31.792 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4369e-05 2024-04-19T01:05:31.792 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.792 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.792 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.792 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.640856+0000", 2024-04-19T01:05:31.792 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.027254787999999999 2024-04-19T01:05:31.792 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.792 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.793 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:31.793 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.640884+0000", 2024-04-19T01:05:31.793 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.7569999999999999e-05 2024-04-19T01:05:31.793 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.793 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.793 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:31.793 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.642758+0000", 2024-04-19T01:05:31.793 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0018743119999999999 2024-04-19T01:05:31.793 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.793 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.793 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:31.793 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.642814+0000", 2024-04-19T01:05:31.793 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.5986999999999998e-05 2024-04-19T01:05:31.793 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.794 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.794 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.794 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.794 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.794 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.2s1 112/98 ECSubWrite(tid=323, reqid=client.4604.0:883, at_version=112'56, trim_to=0'0, roll_forward_to=109'53))", 2024-04-19T01:05:31.794 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:22.588030+0000", 2024-04-19T01:05:31.794 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 9.1203806440000008, 2024-04-19T01:05:31.794 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.032294231999999999, 2024-04-19T01:05:31.794 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:31.794 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:31.794 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:31.794 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:31.794 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.794 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:31.795 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:22.588030+0000", 2024-04-19T01:05:31.795 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.795 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.795 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.795 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:31.795 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:22.588030+0000", 2024-04-19T01:05:31.795 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.795 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.795 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.795 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:31.795 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:22.588032+0000", 2024-04-19T01:05:31.795 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1330000000000002e-06 2024-04-19T01:05:31.795 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.795 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.795 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:31.796 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:22.588315+0000", 2024-04-19T01:05:31.796 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00028298599999999999 2024-04-19T01:05:31.796 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.796 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.796 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:31.796 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:22.588355+0000", 2024-04-19T01:05:31.796 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.0018999999999997e-05 2024-04-19T01:05:31.796 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.796 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.796 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:31.796 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:22.588378+0000", 2024-04-19T01:05:31.796 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2633000000000002e-05 2024-04-19T01:05:31.796 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.796 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.796 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.797 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:22.593358+0000", 2024-04-19T01:05:31.797 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0049802989999999997 2024-04-19T01:05:31.797 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.797 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.797 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:31.797 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:22.593391+0000", 2024-04-19T01:05:31.797 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.2702e-05 2024-04-19T01:05:31.797 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.797 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.797 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:31.797 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:22.620244+0000", 2024-04-19T01:05:31.797 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.026853416000000001 2024-04-19T01:05:31.797 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.798 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.798 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:31.798 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:22.620325+0000", 2024-04-19T01:05:31.798 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.0043999999999995e-05 2024-04-19T01:05:31.798 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.798 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.798 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.798 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.798 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.798 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.2s1 112/98 ECSubWrite(tid=324, reqid=client.4604.0:884, at_version=112'57, trim_to=0'0, roll_forward_to=109'53))", 2024-04-19T01:05:31.798 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:22.594880+0000", 2024-04-19T01:05:31.798 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 9.1135311699999999, 2024-04-19T01:05:31.798 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.025532056000000001, 2024-04-19T01:05:31.798 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:31.798 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:31.799 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:31.799 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:31.799 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.799 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:31.799 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:22.594880+0000", 2024-04-19T01:05:31.799 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.799 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.799 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.799 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:31.799 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:22.594880+0000", 2024-04-19T01:05:31.799 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:31.799 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.799 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.799 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:31.800 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:22.594882+0000", 2024-04-19T01:05:31.800 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.063e-06 2024-04-19T01:05:31.800 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.800 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.800 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:31.800 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:22.595102+0000", 2024-04-19T01:05:31.800 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000220471 2024-04-19T01:05:31.800 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.800 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.800 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:31.800 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:22.595142+0000", 2024-04-19T01:05:31.800 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.9752000000000002e-05 2024-04-19T01:05:31.800 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.800 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.800 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:31.801 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:22.595165+0000", 2024-04-19T01:05:31.801 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3142999999999998e-05 2024-04-19T01:05:31.801 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.801 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.801 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:31.801 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:22.600938+0000", 2024-04-19T01:05:31.801 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0057728270000000003 2024-04-19T01:05:31.801 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.801 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.801 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:31.801 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:22.600969+0000", 2024-04-19T01:05:31.801 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.0595000000000001e-05 2024-04-19T01:05:31.801 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.801 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.802 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:31.802 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:22.620340+0000", 2024-04-19T01:05:31.802 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.01937181 2024-04-19T01:05:31.802 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:31.802 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:31.802 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:31.802 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:22.620412+0000", 2024-04-19T01:05:31.802 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.1395000000000003e-05 2024-04-19T01:05:31.802 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.802 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.802 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.803 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:31.803 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:31.803 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:05:31.804 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-19T01:05:31.805 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:05:31.804+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:05:31.909 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:05:31.902+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:05:31.977 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:05:31.978 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [], 2024-04-19T01:05:31.978 INFO:teuthology.orchestra.run.smithi110.stdout: "num_ops": 0 2024-04-19T01:05:31.978 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:05:31.992 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-19T01:05:32.006 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:05:32.004+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:05:32.107 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:05:32.104+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:05:32.108 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:05:32.108 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [], 2024-04-19T01:05:32.109 INFO:teuthology.orchestra.run.smithi110.stdout: "complaint_time": 30, 2024-04-19T01:05:32.109 INFO:teuthology.orchestra.run.smithi110.stdout: "num_blocked_ops": 0 2024-04-19T01:05:32.109 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:05:32.123 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-19T01:05:32.212 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-19T01:05:32.212+0000 7fe03411a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25039) UID: 0 2024-04-19T01:05:32.297 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:05:32.297 INFO:teuthology.orchestra.run.smithi110.stdout: "size": 20, 2024-04-19T01:05:32.297 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 600, 2024-04-19T01:05:32.297 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [ 2024-04-19T01:05:32.297 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.297 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.ds2 43/31 ECSubWrite(tid=38, reqid=client.4604.0:253, at_version=43'14, trim_to=0'0, roll_forward_to=43'12))", 2024-04-19T01:05:32.297 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.992478+0000", 2024-04-19T01:05:32.297 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 78.302500026000004, 2024-04-19T01:05:32.297 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.025708154, 2024-04-19T01:05:32.298 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:32.298 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:32.298 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:32.298 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:32.298 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.298 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:32.298 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.992478+0000", 2024-04-19T01:05:32.298 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.298 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.298 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.298 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:32.298 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.992478+0000", 2024-04-19T01:05:32.298 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.299 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.299 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.299 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:32.299 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.992480+0000", 2024-04-19T01:05:32.299 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9870000000000002e-06 2024-04-19T01:05:32.299 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.299 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.299 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:32.299 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.992860+0000", 2024-04-19T01:05:32.299 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00038019600000000002 2024-04-19T01:05:32.299 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.299 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.299 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:32.299 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.992891+0000", 2024-04-19T01:05:32.300 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.0982000000000003e-05 2024-04-19T01:05:32.300 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.300 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.300 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:32.300 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.992906+0000", 2024-04-19T01:05:32.300 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5503e-05 2024-04-19T01:05:32.300 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.300 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.300 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:32.301 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.993064+0000", 2024-04-19T01:05:32.301 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015782900000000001 2024-04-19T01:05:32.301 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.301 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.301 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:32.301 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.993111+0000", 2024-04-19T01:05:32.301 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.7138000000000002e-05 2024-04-19T01:05:32.301 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.301 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.302 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:32.302 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.018092+0000", 2024-04-19T01:05:32.302 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.024980822999999999 2024-04-19T01:05:32.302 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.302 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.302 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:32.302 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.018186+0000", 2024-04-19T01:05:32.302 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.3696000000000006e-05 2024-04-19T01:05:32.302 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.302 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:32.302 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.303 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.303 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.303 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.ds2 43/31 ECSubWrite(tid=39, reqid=client.4604.0:254, at_version=43'15, trim_to=0'0, roll_forward_to=43'12))", 2024-04-19T01:05:32.303 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.996882+0000", 2024-04-19T01:05:32.303 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 78.298095907000004, 2024-04-19T01:05:32.303 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.032356653999999999, 2024-04-19T01:05:32.303 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:32.303 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:32.303 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:32.303 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:32.303 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.304 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:32.304 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.996882+0000", 2024-04-19T01:05:32.304 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.304 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.304 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.304 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:32.304 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.996882+0000", 2024-04-19T01:05:32.304 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.304 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.304 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.304 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:32.305 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.996884+0000", 2024-04-19T01:05:32.305 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2029999999999999e-06 2024-04-19T01:05:32.305 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.305 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.305 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:32.305 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.997205+0000", 2024-04-19T01:05:32.305 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00032076899999999999 2024-04-19T01:05:32.305 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.305 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.305 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:32.305 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.997249+0000", 2024-04-19T01:05:32.305 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.4654000000000003e-05 2024-04-19T01:05:32.306 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.306 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.306 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:32.306 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.997273+0000", 2024-04-19T01:05:32.306 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3589000000000001e-05 2024-04-19T01:05:32.306 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.306 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.306 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:32.306 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.007691+0000", 2024-04-19T01:05:32.306 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.010418495999999999 2024-04-19T01:05:32.306 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.306 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.306 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:32.307 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.007717+0000", 2024-04-19T01:05:32.307 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5876000000000001e-05 2024-04-19T01:05:32.307 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.307 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.307 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:32.307 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.029151+0000", 2024-04-19T01:05:32.307 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.021433651000000001 2024-04-19T01:05:32.307 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.307 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.307 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:32.307 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.029238+0000", 2024-04-19T01:05:32.307 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.7416000000000005e-05 2024-04-19T01:05:32.307 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.308 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:32.308 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.308 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.308 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.308 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.ds2 43/31 ECSubWrite(tid=40, reqid=client.4604.0:255, at_version=43'16, trim_to=0'0, roll_forward_to=43'12))", 2024-04-19T01:05:32.308 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.002736+0000", 2024-04-19T01:05:32.308 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 78.292241668000003, 2024-04-19T01:05:32.308 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.026570379000000002, 2024-04-19T01:05:32.308 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:32.308 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:32.308 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:32.308 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:32.308 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.308 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:32.309 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.002736+0000", 2024-04-19T01:05:32.309 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.309 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.309 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.309 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:32.309 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.002736+0000", 2024-04-19T01:05:32.309 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.309 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.309 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.309 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:32.309 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.002738+0000", 2024-04-19T01:05:32.309 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0899999999999999e-06 2024-04-19T01:05:32.309 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.310 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.310 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:32.310 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.002782+0000", 2024-04-19T01:05:32.310 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.3611e-05 2024-04-19T01:05:32.310 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.310 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.310 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:32.310 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.002823+0000", 2024-04-19T01:05:32.310 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.0954999999999997e-05 2024-04-19T01:05:32.310 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.310 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.310 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:32.310 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.002845+0000", 2024-04-19T01:05:32.310 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2775000000000001e-05 2024-04-19T01:05:32.311 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.311 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.311 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:32.311 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.019866+0000", 2024-04-19T01:05:32.311 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.017021162999999999 2024-04-19T01:05:32.311 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.311 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.311 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:32.311 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.019893+0000", 2024-04-19T01:05:32.311 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6176000000000001e-05 2024-04-19T01:05:32.311 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.312 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.312 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:32.312 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.029258+0000", 2024-04-19T01:05:32.312 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0093652990000000005 2024-04-19T01:05:32.312 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.312 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.312 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:32.312 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.029306+0000", 2024-04-19T01:05:32.312 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.8310000000000003e-05 2024-04-19T01:05:32.312 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.312 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:32.312 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.312 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.312 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.312 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.1s4 44/31 ECSubWrite(tid=15, reqid=client.4604.0:289, at_version=44'8, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:05:32.313 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.875457+0000", 2024-04-19T01:05:32.313 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 77.419520343000002, 2024-04-19T01:05:32.313 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.036567641999999997, 2024-04-19T01:05:32.313 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:32.313 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:32.313 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:32.313 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:32.313 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.313 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:32.313 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875457+0000", 2024-04-19T01:05:32.314 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.314 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.314 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.314 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:32.314 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875457+0000", 2024-04-19T01:05:32.314 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.314 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.314 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.314 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:32.315 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875458+0000", 2024-04-19T01:05:32.315 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.8400000000000004e-07 2024-04-19T01:05:32.315 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.315 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.315 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:32.315 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875519+0000", 2024-04-19T01:05:32.315 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.1377000000000005e-05 2024-04-19T01:05:32.315 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.315 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.315 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:32.315 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875588+0000", 2024-04-19T01:05:32.315 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.9099999999999999e-05 2024-04-19T01:05:32.315 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.316 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.316 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:32.316 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875615+0000", 2024-04-19T01:05:32.316 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6902000000000001e-05 2024-04-19T01:05:32.316 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.316 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.316 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:32.316 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875846+0000", 2024-04-19T01:05:32.316 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00023076699999999999 2024-04-19T01:05:32.316 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.316 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.317 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:32.317 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875874+0000", 2024-04-19T01:05:32.317 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.7725e-05 2024-04-19T01:05:32.317 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.317 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.317 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:32.317 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.911826+0000", 2024-04-19T01:05:32.317 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.035952348000000002 2024-04-19T01:05:32.317 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.317 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.317 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:32.317 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.912025+0000", 2024-04-19T01:05:32.317 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000198739 2024-04-19T01:05:32.317 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.317 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:32.318 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.318 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.318 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.318 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 44/31 ECSubWrite(tid=82, reqid=client.4604.0:280, at_version=44'50, trim_to=0'0, roll_forward_to=43'48))", 2024-04-19T01:05:32.318 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.880504+0000", 2024-04-19T01:05:32.318 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 77.414473248999997, 2024-04-19T01:05:32.318 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.041548157000000002, 2024-04-19T01:05:32.318 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:32.318 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:32.318 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:32.318 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:32.318 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.318 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:32.319 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.880504+0000", 2024-04-19T01:05:32.319 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.319 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.319 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.319 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:32.319 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.880504+0000", 2024-04-19T01:05:32.319 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.319 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.319 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.319 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:32.319 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.880506+0000", 2024-04-19T01:05:32.319 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9439999999999999e-06 2024-04-19T01:05:32.319 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.320 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.320 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:32.320 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.880844+0000", 2024-04-19T01:05:32.320 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00033782799999999998 2024-04-19T01:05:32.320 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.320 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.320 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:32.323 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.880877+0000", 2024-04-19T01:05:32.323 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.2524999999999998e-05 2024-04-19T01:05:32.323 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.323 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.323 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:32.323 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.880899+0000", 2024-04-19T01:05:32.324 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2656000000000001e-05 2024-04-19T01:05:32.324 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.324 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.324 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:32.324 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.881129+0000", 2024-04-19T01:05:32.324 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00022929600000000001 2024-04-19T01:05:32.324 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.324 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.324 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:32.325 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.881155+0000", 2024-04-19T01:05:32.325 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6729000000000001e-05 2024-04-19T01:05:32.325 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.325 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.325 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:32.325 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.921810+0000", 2024-04-19T01:05:32.325 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.040654807000000001 2024-04-19T01:05:32.325 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.325 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.325 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:32.325 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.922052+0000", 2024-04-19T01:05:32.325 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000242372 2024-04-19T01:05:32.325 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.326 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:32.326 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.326 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.326 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.326 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.0s1 44/31 ECSubWrite(tid=25, reqid=client.4604.0:284, at_version=44'22, trim_to=0'0, roll_forward_to=44'20))", 2024-04-19T01:05:32.326 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.922536+0000", 2024-04-19T01:05:32.326 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 77.372441574000007, 2024-04-19T01:05:32.326 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.029007871000000001, 2024-04-19T01:05:32.326 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:32.326 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:32.326 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:32.326 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:32.326 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.327 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:32.327 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.922536+0000", 2024-04-19T01:05:32.327 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.327 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.327 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.327 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:32.327 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.922536+0000", 2024-04-19T01:05:32.327 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.327 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.327 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.327 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:32.328 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.922538+0000", 2024-04-19T01:05:32.328 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0829999999999998e-06 2024-04-19T01:05:32.328 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.328 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.328 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:32.328 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.923282+0000", 2024-04-19T01:05:32.328 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000743554 2024-04-19T01:05:32.328 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.328 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.329 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:32.329 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.923311+0000", 2024-04-19T01:05:32.329 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.9147999999999999e-05 2024-04-19T01:05:32.329 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.329 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.330 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:32.330 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.923337+0000", 2024-04-19T01:05:32.330 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6648999999999998e-05 2024-04-19T01:05:32.330 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.330 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.330 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:32.331 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.923466+0000", 2024-04-19T01:05:32.331 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00012821100000000001 2024-04-19T01:05:32.331 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.331 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.331 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:32.331 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.923478+0000", 2024-04-19T01:05:32.331 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.2291e-05 2024-04-19T01:05:32.331 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.331 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.331 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:32.331 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.951479+0000", 2024-04-19T01:05:32.331 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.02800066 2024-04-19T01:05:32.331 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.332 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.332 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:32.332 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.951544+0000", 2024-04-19T01:05:32.332 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.5275000000000001e-05 2024-04-19T01:05:32.332 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.332 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:32.332 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.332 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.332 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.332 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 74/31 ECSubWrite(tid=174, reqid=client.4604.0:548, at_version=74'110, trim_to=0'0, roll_forward_to=70'108))", 2024-04-19T01:05:32.332 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:44.595181+0000", 2024-04-19T01:05:32.332 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 47.699796632000002, 2024-04-19T01:05:32.332 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.023202622999999999, 2024-04-19T01:05:32.332 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:32.333 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:32.333 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:32.333 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:32.333 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.333 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:32.333 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.595181+0000", 2024-04-19T01:05:32.333 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.333 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.334 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.334 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:32.334 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.595181+0000", 2024-04-19T01:05:32.334 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.334 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.334 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.334 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:32.334 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.595182+0000", 2024-04-19T01:05:32.334 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.13e-06 2024-04-19T01:05:32.334 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.335 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.335 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:32.335 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.595558+0000", 2024-04-19T01:05:32.335 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00037566500000000002 2024-04-19T01:05:32.335 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.335 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.335 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:32.335 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.595596+0000", 2024-04-19T01:05:32.335 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.7815999999999998e-05 2024-04-19T01:05:32.335 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.335 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.335 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:32.336 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.595624+0000", 2024-04-19T01:05:32.336 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.8906000000000001e-05 2024-04-19T01:05:32.336 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.336 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.336 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:32.336 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.595857+0000", 2024-04-19T01:05:32.336 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00023279099999999999 2024-04-19T01:05:32.336 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.336 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.336 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:32.336 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.595886+0000", 2024-04-19T01:05:32.336 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.8770000000000001e-05 2024-04-19T01:05:32.336 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.337 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.337 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:32.337 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.618262+0000", 2024-04-19T01:05:32.337 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.022375757999999999 2024-04-19T01:05:32.337 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.337 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.337 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:32.337 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:44.618384+0000", 2024-04-19T01:05:32.337 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000121787 2024-04-19T01:05:32.337 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.337 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:32.337 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.337 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.337 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.337 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.es5 86/31 ECSubWrite(tid=195, reqid=client.4604.0:653, at_version=86'39, trim_to=0'0, roll_forward_to=84'38))", 2024-04-19T01:05:32.338 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:56.572682+0000", 2024-04-19T01:05:32.338 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 35.722295635000002, 2024-04-19T01:05:32.338 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.020998065999999999, 2024-04-19T01:05:32.338 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:32.338 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:32.338 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:32.338 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:32.338 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.338 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:32.338 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.572682+0000", 2024-04-19T01:05:32.338 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.338 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.338 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.338 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:32.339 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.572682+0000", 2024-04-19T01:05:32.340 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.340 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.340 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.340 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:32.340 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.572683+0000", 2024-04-19T01:05:32.340 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.1699999999999997e-07 2024-04-19T01:05:32.340 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.340 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.340 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:32.340 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.572975+0000", 2024-04-19T01:05:32.340 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000292596 2024-04-19T01:05:32.340 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.340 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.341 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:32.341 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.573007+0000", 2024-04-19T01:05:32.341 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.1785999999999997e-05 2024-04-19T01:05:32.341 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.341 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.341 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:32.341 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.573032+0000", 2024-04-19T01:05:32.342 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4896e-05 2024-04-19T01:05:32.342 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.342 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.342 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:32.342 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.573262+0000", 2024-04-19T01:05:32.342 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000230385 2024-04-19T01:05:32.342 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.342 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.342 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:32.342 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.573292+0000", 2024-04-19T01:05:32.342 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.9836e-05 2024-04-19T01:05:32.342 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.342 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.342 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:32.343 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.593615+0000", 2024-04-19T01:05:32.343 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.020323305999999999 2024-04-19T01:05:32.343 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.343 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.343 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:32.343 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.593680+0000", 2024-04-19T01:05:32.343 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.4543999999999997e-05 2024-04-19T01:05:32.343 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.343 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:32.343 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.343 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.343 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.343 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.0s1 91/31 ECSubWrite(tid=96, reqid=client.4604.0:683, at_version=91'89, trim_to=0'0, roll_forward_to=89'87))", 2024-04-19T01:05:32.343 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:01.594644+0000", 2024-04-19T01:05:32.343 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 30.700333573999998, 2024-04-19T01:05:32.344 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.02175761, 2024-04-19T01:05:32.344 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:32.344 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:32.344 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:32.344 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:32.344 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.344 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:32.344 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.594644+0000", 2024-04-19T01:05:32.344 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.344 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.344 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.344 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:32.344 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.594644+0000", 2024-04-19T01:05:32.344 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.345 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.345 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.345 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:32.345 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.594645+0000", 2024-04-19T01:05:32.345 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.9599999999999999e-07 2024-04-19T01:05:32.345 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.345 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.345 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:32.345 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.594912+0000", 2024-04-19T01:05:32.345 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00026693099999999998 2024-04-19T01:05:32.345 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.345 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.345 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:32.345 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.594948+0000", 2024-04-19T01:05:32.346 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.6582000000000003e-05 2024-04-19T01:05:32.346 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.346 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.346 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:32.346 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.594975+0000", 2024-04-19T01:05:32.346 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6458999999999999e-05 2024-04-19T01:05:32.346 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.346 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.346 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:32.346 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.595154+0000", 2024-04-19T01:05:32.347 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000179884 2024-04-19T01:05:32.347 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.347 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.347 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:32.347 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.595187+0000", 2024-04-19T01:05:32.347 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.2919000000000001e-05 2024-04-19T01:05:32.347 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.347 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.347 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:32.347 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.616333+0000", 2024-04-19T01:05:32.347 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.021145586000000001 2024-04-19T01:05:32.347 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.347 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.347 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:32.348 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:01.616402+0000", 2024-04-19T01:05:32.348 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.8652999999999995e-05 2024-04-19T01:05:32.348 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.348 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:32.348 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.348 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.348 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.348 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.es5 92/31 ECSubWrite(tid=209, reqid=client.4604.0:701, at_version=92'41, trim_to=0'0, roll_forward_to=86'39))", 2024-04-19T01:05:32.348 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:02.701192+0000", 2024-04-19T01:05:32.348 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 29.593786049999999, 2024-04-19T01:05:32.348 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.025340898000000001, 2024-04-19T01:05:32.348 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:32.348 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:32.348 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:32.348 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:32.349 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.349 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:32.349 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.701192+0000", 2024-04-19T01:05:32.349 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.349 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.349 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.349 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:32.349 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.701192+0000", 2024-04-19T01:05:32.350 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.350 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.350 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.350 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:32.350 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.701194+0000", 2024-04-19T01:05:32.350 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.114e-06 2024-04-19T01:05:32.350 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.350 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.350 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:32.350 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.701818+0000", 2024-04-19T01:05:32.351 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00062439099999999997 2024-04-19T01:05:32.351 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.351 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.351 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:32.351 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.701859+0000", 2024-04-19T01:05:32.351 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.1227999999999999e-05 2024-04-19T01:05:32.351 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.351 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.351 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:32.352 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.701887+0000", 2024-04-19T01:05:32.352 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.7276000000000001e-05 2024-04-19T01:05:32.352 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.352 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.352 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:32.352 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.702072+0000", 2024-04-19T01:05:32.352 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000185911 2024-04-19T01:05:32.352 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.352 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.352 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:32.352 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.702108+0000", 2024-04-19T01:05:32.352 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.6009e-05 2024-04-19T01:05:32.352 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.353 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.353 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:32.353 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.726458+0000", 2024-04-19T01:05:32.353 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.024349546 2024-04-19T01:05:32.353 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.353 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.353 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:32.353 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.726532+0000", 2024-04-19T01:05:32.353 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.4423000000000002e-05 2024-04-19T01:05:32.353 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.354 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:32.354 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.354 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.354 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.354 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.ds2 102/98 ECSubWrite(tid=208, reqid=client.4604.0:758, at_version=102'78, trim_to=0'0, roll_forward_to=0'0))", 2024-04-19T01:05:32.354 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:12.694276+0000", 2024-04-19T01:05:32.354 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 19.600701446999999, 2024-04-19T01:05:32.354 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.047864423000000003, 2024-04-19T01:05:32.354 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:32.355 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:32.355 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:32.355 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:32.355 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.355 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:32.355 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.694276+0000", 2024-04-19T01:05:32.355 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.355 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.355 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.355 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:32.355 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.694276+0000", 2024-04-19T01:05:32.355 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.355 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.356 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.356 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:32.356 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.694277+0000", 2024-04-19T01:05:32.356 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.4300000000000002e-07 2024-04-19T01:05:32.356 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.356 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.356 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:32.356 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.694596+0000", 2024-04-19T01:05:32.356 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000318986 2024-04-19T01:05:32.356 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.356 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.356 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:32.356 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.694626+0000", 2024-04-19T01:05:32.356 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.9791999999999999e-05 2024-04-19T01:05:32.357 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.357 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.357 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:32.357 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.694647+0000", 2024-04-19T01:05:32.357 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1140000000000001e-05 2024-04-19T01:05:32.357 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.357 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.357 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:32.357 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.694818+0000", 2024-04-19T01:05:32.357 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00017080400000000001 2024-04-19T01:05:32.357 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.357 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.357 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:32.357 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.694847+0000", 2024-04-19T01:05:32.357 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.9889000000000001e-05 2024-04-19T01:05:32.358 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.358 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.358 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:32.358 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.742097+0000", 2024-04-19T01:05:32.358 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.047249523000000002 2024-04-19T01:05:32.358 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.358 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.358 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:32.358 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.742141+0000", 2024-04-19T01:05:32.358 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.3545999999999999e-05 2024-04-19T01:05:32.358 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.358 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:32.358 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.358 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.359 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.359 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.ds2 102/98 ECSubWrite(tid=209, reqid=client.4604.0:759, at_version=102'79, trim_to=0'0, roll_forward_to=0'0))", 2024-04-19T01:05:32.359 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:12.701226+0000", 2024-04-19T01:05:32.359 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 19.593751798, 2024-04-19T01:05:32.359 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.040943449, 2024-04-19T01:05:32.359 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:32.359 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:32.359 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:32.359 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:32.359 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.359 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:32.360 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.701226+0000", 2024-04-19T01:05:32.360 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.360 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.360 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.360 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:32.360 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.701226+0000", 2024-04-19T01:05:32.360 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.360 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.360 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.360 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:32.360 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.701228+0000", 2024-04-19T01:05:32.360 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0329999999999998e-06 2024-04-19T01:05:32.360 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.361 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.361 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:32.361 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.701430+0000", 2024-04-19T01:05:32.361 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00020248600000000001 2024-04-19T01:05:32.361 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.361 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.361 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:32.361 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.701468+0000", 2024-04-19T01:05:32.361 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.7709000000000001e-05 2024-04-19T01:05:32.361 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.361 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.361 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:32.361 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.701492+0000", 2024-04-19T01:05:32.362 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3918999999999999e-05 2024-04-19T01:05:32.362 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.362 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.362 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:32.362 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.712216+0000", 2024-04-19T01:05:32.362 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.010723821 2024-04-19T01:05:32.362 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.363 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.363 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:32.363 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.712233+0000", 2024-04-19T01:05:32.363 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6833e-05 2024-04-19T01:05:32.363 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.363 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.363 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:32.363 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.742148+0000", 2024-04-19T01:05:32.363 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.029915483999999999 2024-04-19T01:05:32.363 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.363 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.364 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:32.364 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.742169+0000", 2024-04-19T01:05:32.364 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1163999999999999e-05 2024-04-19T01:05:32.364 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.364 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:32.364 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.364 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.364 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.364 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.ds2 102/98 ECSubWrite(tid=210, reqid=client.4604.0:760, at_version=102'80, trim_to=0'0, roll_forward_to=0'0))", 2024-04-19T01:05:32.364 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:12.712153+0000", 2024-04-19T01:05:32.364 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 19.582825026999998, 2024-04-19T01:05:32.364 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.030037707, 2024-04-19T01:05:32.364 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:32.365 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:32.365 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:32.365 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:32.365 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.365 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:32.365 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.712153+0000", 2024-04-19T01:05:32.365 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.365 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.365 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.365 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:32.365 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.712153+0000", 2024-04-19T01:05:32.365 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.365 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.365 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.366 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:32.366 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.712155+0000", 2024-04-19T01:05:32.366 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0250000000000001e-06 2024-04-19T01:05:32.366 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.366 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.366 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:32.366 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.712298+0000", 2024-04-19T01:05:32.366 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00014372699999999999 2024-04-19T01:05:32.366 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.366 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.366 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:32.366 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.712330+0000", 2024-04-19T01:05:32.366 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.1313000000000003e-05 2024-04-19T01:05:32.367 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.367 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.367 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:32.367 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.712351+0000", 2024-04-19T01:05:32.367 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1857000000000001e-05 2024-04-19T01:05:32.367 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.367 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.367 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:32.367 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.722562+0000", 2024-04-19T01:05:32.367 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.010210225999999999 2024-04-19T01:05:32.367 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.367 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.367 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:32.367 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.722577+0000", 2024-04-19T01:05:32.368 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5753e-05 2024-04-19T01:05:32.368 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.368 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.368 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:32.368 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.742173+0000", 2024-04-19T01:05:32.368 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.019595997 2024-04-19T01:05:32.368 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.368 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.368 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:32.368 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.742190+0000", 2024-04-19T01:05:32.368 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6809000000000001e-05 2024-04-19T01:05:32.368 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.368 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:32.368 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.368 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.369 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.369 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.ds2 102/98 ECSubWrite(tid=211, reqid=client.4604.0:761, at_version=102'81, trim_to=0'0, roll_forward_to=0'0))", 2024-04-19T01:05:32.369 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:12.716879+0000", 2024-04-19T01:05:32.369 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 19.578098529000002, 2024-04-19T01:05:32.369 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.026129459000000001, 2024-04-19T01:05:32.369 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:32.369 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:32.369 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:32.369 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:32.369 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.370 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:32.370 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.716879+0000", 2024-04-19T01:05:32.370 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.370 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.370 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.370 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:32.370 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.716879+0000", 2024-04-19T01:05:32.370 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.370 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.371 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.371 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:32.371 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.716881+0000", 2024-04-19T01:05:32.371 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.736e-06 2024-04-19T01:05:32.371 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.371 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.371 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:32.371 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.717035+0000", 2024-04-19T01:05:32.371 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015389099999999999 2024-04-19T01:05:32.371 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.371 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.371 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:32.371 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.717077+0000", 2024-04-19T01:05:32.371 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.2289000000000001e-05 2024-04-19T01:05:32.372 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.372 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.372 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:32.372 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.717154+0000", 2024-04-19T01:05:32.372 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.7164999999999997e-05 2024-04-19T01:05:32.372 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.373 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.373 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:32.373 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.735162+0000", 2024-04-19T01:05:32.373 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.018007842 2024-04-19T01:05:32.373 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.373 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.373 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:32.373 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.735171+0000", 2024-04-19T01:05:32.373 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.3710000000000006e-06 2024-04-19T01:05:32.373 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.373 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.373 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:32.373 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.742943+0000", 2024-04-19T01:05:32.373 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.007771554 2024-04-19T01:05:32.373 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.374 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.374 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:32.374 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.743008+0000", 2024-04-19T01:05:32.374 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.5611000000000006e-05 2024-04-19T01:05:32.374 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.374 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:32.374 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.374 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.374 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.374 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.ds2 102/98 ECSubWrite(tid=212, reqid=client.4604.0:762, at_version=102'82, trim_to=0'0, roll_forward_to=0'0))", 2024-04-19T01:05:32.374 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:12.720421+0000", 2024-04-19T01:05:32.374 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 19.574556115, 2024-04-19T01:05:32.374 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.022986143000000001, 2024-04-19T01:05:32.374 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:32.375 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:32.375 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:32.375 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:32.375 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.375 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:32.375 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.720421+0000", 2024-04-19T01:05:32.375 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.375 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.375 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.375 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:32.375 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.720421+0000", 2024-04-19T01:05:32.375 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.375 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.375 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.375 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:32.376 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.720424+0000", 2024-04-19T01:05:32.376 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1330000000000002e-06 2024-04-19T01:05:32.376 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.376 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.376 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:32.376 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.720470+0000", 2024-04-19T01:05:32.376 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.6539000000000003e-05 2024-04-19T01:05:32.376 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.376 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.376 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:32.376 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.720509+0000", 2024-04-19T01:05:32.376 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.8575000000000003e-05 2024-04-19T01:05:32.376 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.376 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.376 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:32.377 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.720531+0000", 2024-04-19T01:05:32.377 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.251e-05 2024-04-19T01:05:32.377 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.377 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.377 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:32.377 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.742264+0000", 2024-04-19T01:05:32.377 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.021732960999999999 2024-04-19T01:05:32.377 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.377 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.377 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:32.377 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.742281+0000", 2024-04-19T01:05:32.377 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6356000000000001e-05 2024-04-19T01:05:32.377 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.377 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.377 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:32.378 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.743355+0000", 2024-04-19T01:05:32.378 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001074775 2024-04-19T01:05:32.378 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.378 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.378 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:32.378 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:12.743408+0000", 2024-04-19T01:05:32.378 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.2293999999999998e-05 2024-04-19T01:05:32.378 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.378 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:32.378 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.379 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.379 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.379 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.cs4 107/98 ECSubWrite(tid=299, reqid=client.4604.0:827, at_version=107'75, trim_to=0'0, roll_forward_to=105'72))", 2024-04-19T01:05:32.379 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:17.655917+0000", 2024-04-19T01:05:32.379 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 14.639060819000001, 2024-04-19T01:05:32.379 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.021720140999999998, 2024-04-19T01:05:32.379 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:32.379 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:32.379 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:32.379 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:32.379 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.379 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:32.379 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.655917+0000", 2024-04-19T01:05:32.379 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.379 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.380 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.380 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:32.380 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.655917+0000", 2024-04-19T01:05:32.380 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.380 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.380 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.380 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:32.380 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.655919+0000", 2024-04-19T01:05:32.380 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0260000000000002e-06 2024-04-19T01:05:32.380 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.381 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.381 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:32.381 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.656582+0000", 2024-04-19T01:05:32.381 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00066314100000000001 2024-04-19T01:05:32.381 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.381 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.381 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:32.381 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.656619+0000", 2024-04-19T01:05:32.381 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.7548e-05 2024-04-19T01:05:32.381 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.381 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.381 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:32.381 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.656642+0000", 2024-04-19T01:05:32.381 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2645000000000001e-05 2024-04-19T01:05:32.381 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.382 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.382 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:32.382 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.656866+0000", 2024-04-19T01:05:32.382 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000223604 2024-04-19T01:05:32.382 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.382 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.382 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:32.382 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.656895+0000", 2024-04-19T01:05:32.382 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.9062000000000001e-05 2024-04-19T01:05:32.382 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.382 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.382 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:32.384 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.677547+0000", 2024-04-19T01:05:32.384 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.020652178 2024-04-19T01:05:32.384 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.384 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.384 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:32.384 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:17.677637+0000", 2024-04-19T01:05:32.384 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.9937000000000003e-05 2024-04-19T01:05:32.384 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.384 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:32.384 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.385 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.385 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.385 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 116/115 ECSubWrite(tid=271, reqid=osd.13.0:271, at_version=116'150, trim_to=0'0, roll_forward_to=0'0))", 2024-04-19T01:05:32.385 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:26.573938+0000", 2024-04-19T01:05:32.385 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 5.721039148, 2024-04-19T01:05:32.385 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.020414383000000001, 2024-04-19T01:05:32.385 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:32.385 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:32.385 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:32.385 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:32.385 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.385 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:32.385 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.573938+0000", 2024-04-19T01:05:32.385 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.386 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.386 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.386 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:32.386 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.573938+0000", 2024-04-19T01:05:32.386 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.386 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.386 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.386 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:32.386 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.573940+0000", 2024-04-19T01:05:32.386 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6330000000000001e-06 2024-04-19T01:05:32.386 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.386 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.386 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:32.386 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.573983+0000", 2024-04-19T01:05:32.387 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.2901000000000001e-05 2024-04-19T01:05:32.387 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.387 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.387 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:32.387 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.574002+0000", 2024-04-19T01:05:32.387 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.8898000000000001e-05 2024-04-19T01:05:32.387 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.387 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.387 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:32.387 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.574014+0000", 2024-04-19T01:05:32.387 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.2435999999999999e-05 2024-04-19T01:05:32.387 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.387 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.387 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:32.388 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.574141+0000", 2024-04-19T01:05:32.388 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00012696199999999999 2024-04-19T01:05:32.388 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.388 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.388 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:32.388 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.574162+0000", 2024-04-19T01:05:32.388 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0803999999999999e-05 2024-04-19T01:05:32.388 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.388 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.388 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:32.388 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.594178+0000", 2024-04-19T01:05:32.388 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.020015504 2024-04-19T01:05:32.388 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.389 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.389 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:32.389 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.594353+0000", 2024-04-19T01:05:32.389 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00017524500000000001 2024-04-19T01:05:32.389 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.389 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:32.389 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.389 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.389 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.389 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.1s4 116/115 ECSubWrite(tid=76, reqid=client.4604.0:899, at_version=116'72, trim_to=0'0, roll_forward_to=0'0))", 2024-04-19T01:05:32.389 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:26.575135+0000", 2024-04-19T01:05:32.389 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 5.7198423539999999, 2024-04-19T01:05:32.389 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.028412898999999998, 2024-04-19T01:05:32.389 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:32.389 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:32.390 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:32.390 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:32.390 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.390 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:32.390 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.575135+0000", 2024-04-19T01:05:32.390 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.390 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.390 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.390 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:32.390 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.575135+0000", 2024-04-19T01:05:32.390 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.390 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.390 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.391 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:32.391 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.575163+0000", 2024-04-19T01:05:32.391 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.7797000000000002e-05 2024-04-19T01:05:32.391 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.391 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.391 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:32.391 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.575569+0000", 2024-04-19T01:05:32.391 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00040614399999999999 2024-04-19T01:05:32.391 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.391 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.392 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:32.392 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.575597+0000", 2024-04-19T01:05:32.392 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.7671999999999999e-05 2024-04-19T01:05:32.392 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.392 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.392 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:32.392 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.575616+0000", 2024-04-19T01:05:32.392 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9225e-05 2024-04-19T01:05:32.392 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.392 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.392 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:32.393 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.575817+0000", 2024-04-19T01:05:32.393 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00020076 2024-04-19T01:05:32.393 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.393 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.393 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:32.393 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.575840+0000", 2024-04-19T01:05:32.393 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.3224e-05 2024-04-19T01:05:32.393 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.393 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.393 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:32.393 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.603488+0000", 2024-04-19T01:05:32.393 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.027647531999999999 2024-04-19T01:05:32.393 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.393 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.394 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:32.394 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.603548+0000", 2024-04-19T01:05:32.394 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.0544999999999997e-05 2024-04-19T01:05:32.394 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.394 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:32.394 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.394 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.394 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.395 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.9s3 116/115 ECSubWrite(tid=272, reqid=osd.13.0:272, at_version=116'152, trim_to=0'0, roll_forward_to=0'0))", 2024-04-19T01:05:32.395 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:26.576256+0000", 2024-04-19T01:05:32.395 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 5.7187220019999998, 2024-04-19T01:05:32.395 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.027455779999999999, 2024-04-19T01:05:32.395 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:32.395 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:32.395 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:32.395 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:32.395 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.395 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:32.395 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.576256+0000", 2024-04-19T01:05:32.395 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.396 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.396 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.396 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:32.396 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.576256+0000", 2024-04-19T01:05:32.396 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.396 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.396 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.396 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:32.396 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.576257+0000", 2024-04-19T01:05:32.396 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.454e-06 2024-04-19T01:05:32.396 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.396 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.397 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:32.397 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.576305+0000", 2024-04-19T01:05:32.397 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.7982000000000002e-05 2024-04-19T01:05:32.397 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.397 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.397 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:32.397 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.576333+0000", 2024-04-19T01:05:32.397 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.8036000000000001e-05 2024-04-19T01:05:32.397 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.397 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.397 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:32.398 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.576350+0000", 2024-04-19T01:05:32.398 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7377999999999999e-05 2024-04-19T01:05:32.398 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.398 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.398 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:32.398 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.590942+0000", 2024-04-19T01:05:32.398 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.014591994 2024-04-19T01:05:32.398 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.398 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.398 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:32.398 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.590964+0000", 2024-04-19T01:05:32.398 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1118999999999999e-05 2024-04-19T01:05:32.398 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.398 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.399 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:32.399 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.603676+0000", 2024-04-19T01:05:32.399 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.012712790999999999 2024-04-19T01:05:32.399 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.399 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.399 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:32.399 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.603711+0000", 2024-04-19T01:05:32.399 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.5026e-05 2024-04-19T01:05:32.399 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.399 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:32.399 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.399 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.399 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.399 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "MOSDECSubOpWrite(3.1s4 116/115 ECSubWrite(tid=77, reqid=client.4604.0:900, at_version=116'73, trim_to=0'0, roll_forward_to=0'0))", 2024-04-19T01:05:32.400 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:26.588424+0000", 2024-04-19T01:05:32.400 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 5.7065535579999995, 2024-04-19T01:05:32.400 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.027360082000000001, 2024-04-19T01:05:32.400 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:32.400 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:32.400 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "reached pg", 2024-04-19T01:05:32.400 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:32.400 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.400 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:32.400 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.588424+0000", 2024-04-19T01:05:32.400 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.400 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.400 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.400 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:32.401 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.588424+0000", 2024-04-19T01:05:32.401 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.401 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.401 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.401 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:32.401 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.588425+0000", 2024-04-19T01:05:32.401 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.4449999999999999e-06 2024-04-19T01:05:32.401 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.401 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.401 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:32.401 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.588696+0000", 2024-04-19T01:05:32.401 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00027022 2024-04-19T01:05:32.401 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.401 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.402 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:32.402 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.588720+0000", 2024-04-19T01:05:32.402 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4011000000000001e-05 2024-04-19T01:05:32.402 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.402 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.402 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:32.402 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.588735+0000", 2024-04-19T01:05:32.402 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5071e-05 2024-04-19T01:05:32.402 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.402 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.402 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:32.402 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.593189+0000", 2024-04-19T01:05:32.402 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0044545599999999998 2024-04-19T01:05:32.402 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.403 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.403 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:32.403 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.593207+0000", 2024-04-19T01:05:32.403 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7785e-05 2024-04-19T01:05:32.403 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.403 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.403 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:32.403 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.615736+0000", 2024-04-19T01:05:32.403 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.022528893000000001 2024-04-19T01:05:32.403 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.403 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.403 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:32.403 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:26.615784+0000", 2024-04-19T01:05:32.403 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.8096999999999997e-05 2024-04-19T01:05:32.404 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.404 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:32.404 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.404 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.404 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:32.404 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:05:32.406 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-04-19T01:05:32.407 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:05:32.402+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:05:32.507 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:05:32.508+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:05:32.574 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:05:32.574 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [], 2024-04-19T01:05:32.574 INFO:teuthology.orchestra.run.smithi110.stdout: "num_ops": 0 2024-04-19T01:05:32.574 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:05:32.588 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-04-19T01:05:32.632 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:05:32.632+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:05:32.639 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2024-04-19T01:05:32.651 INFO:tasks.rados.rados.0.smithi155.stdout:384: done (1 left) 2024-04-19T01:05:32.651 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 1 current snap is 46 2024-04-19T01:05:32.651 INFO:tasks.rados.rados.0.smithi155.stdout:385: seq_num 151 ranges {98304=131072} 2024-04-19T01:05:32.660 INFO:tasks.rados.rados.0.smithi155.stdout:385: writing smithi15528762-1 from 98304 to 229376 tid 1 2024-04-19T01:05:32.661 INFO:tasks.rados.rados.0.smithi155.stdout:386: rmattr oid 34 current snap is 46 2024-04-19T01:05:32.661 INFO:tasks.rados.rados.0.smithi155.stdout:386: done (2 left) 2024-04-19T01:05:32.661 INFO:tasks.rados.rados.0.smithi155.stdout:387: read oid 21 snap -1 2024-04-19T01:05:32.661 INFO:tasks.rados.rados.0.smithi155.stdout:387: expect deleted 2024-04-19T01:05:32.661 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 45 current snap is 46 2024-04-19T01:05:32.661 INFO:tasks.rados.rados.0.smithi155.stdout:388: seq_num 152 ranges {2785280=147456} 2024-04-19T01:05:32.710 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:05:32.708+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:05:32.721 INFO:tasks.rados.rados.0.smithi155.stdout:388: writing smithi15528762-45 from 2785280 to 2932736 tid 1 2024-04-19T01:05:32.721 INFO:tasks.rados.rados.0.smithi155.stdout:389: copy_from oid 20 from oid 18 current snap is 46 2024-04-19T01:05:32.721 INFO:tasks.rados.rados.0.smithi155.stdout:385: finishing write tid 1 to smithi15528762-1 2024-04-19T01:05:32.721 INFO:tasks.rados.rados.0.smithi155.stdout:385: finishing write tid 2 to smithi15528762-1 2024-04-19T01:05:32.721 INFO:tasks.rados.rados.0.smithi155.stdout:385: finishing write tid 3 to smithi15528762-1 2024-04-19T01:05:32.721 INFO:tasks.rados.rados.0.smithi155.stdout:385: oid 1 updating version 0 to 111 2024-04-19T01:05:32.721 INFO:tasks.rados.rados.0.smithi155.stdout:385: oid 1 version 111 is already newer than 110 2024-04-19T01:05:32.721 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 1 v 111 (ObjNum 151 snap 46 seq_num 151) dirty exists 2024-04-19T01:05:32.721 INFO:tasks.rados.rados.0.smithi155.stdout:385: left oid 1 (ObjNum 151 snap 46 seq_num 151) 2024-04-19T01:05:32.721 INFO:tasks.rados.rados.0.smithi155.stdout:385: done (4 left) 2024-04-19T01:05:32.722 INFO:tasks.rados.rados.0.smithi155.stdout:387: done (3 left) 2024-04-19T01:05:32.722 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 41 current snap is 46 2024-04-19T01:05:32.722 INFO:tasks.rados.rados.0.smithi155.stdout:390: seq_num 153 ranges {2392064=163840} 2024-04-19T01:05:32.746 INFO:tasks.rados.rados.0.smithi155.stdout:390: writing smithi15528762-41 from 2392064 to 2555904 tid 1 2024-04-19T01:05:32.746 INFO:tasks.rados.rados.0.smithi155.stdout:391: rollback oid 23 current snap is 46 2024-04-19T01:05:32.746 INFO:tasks.rados.rados.0.smithi155.stdout:rollback oid 23 to 35 2024-04-19T01:05:32.746 INFO:tasks.rados.rados.0.smithi155.stdout:383: expect (ObjNum 139 snap 39 seq_num 139) 2024-04-19T01:05:32.780 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:05:32.780 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [], 2024-04-19T01:05:32.780 INFO:teuthology.orchestra.run.smithi110.stdout: "complaint_time": 30, 2024-04-19T01:05:32.781 INFO:teuthology.orchestra.run.smithi110.stdout: "num_blocked_ops": 0 2024-04-19T01:05:32.781 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:05:32.794 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-04-19T01:05:32.832 INFO:tasks.rados.rados.0.smithi155.stdout:388: finishing write tid 1 to smithi15528762-45 2024-04-19T01:05:32.832 INFO:tasks.rados.rados.0.smithi155.stdout:388: finishing write tid 2 to smithi15528762-45 2024-04-19T01:05:32.832 INFO:tasks.rados.rados.0.smithi155.stdout:391: finishing rollback tid 0 to smithi15528762-23 2024-04-19T01:05:32.832 INFO:tasks.rados.rados.0.smithi155.stdout:389: finishing copy_from racing read to smithi15528762-20 2024-04-19T01:05:32.832 INFO:tasks.rados.rados.0.smithi155.stdout:388: finishing write tid 3 to smithi15528762-45 2024-04-19T01:05:32.832 INFO:tasks.rados.rados.0.smithi155.stdout:388: oid 45 updating version 0 to 78 2024-04-19T01:05:32.832 INFO:tasks.rados.rados.0.smithi155.stdout:388: oid 45 version 78 is already newer than 77 2024-04-19T01:05:32.832 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 45 v 78 (ObjNum 152 snap 46 seq_num 152) dirty exists 2024-04-19T01:05:32.833 INFO:tasks.rados.rados.0.smithi155.stdout:388: left oid 45 (ObjNum 152 snap 46 seq_num 152) 2024-04-19T01:05:32.833 INFO:tasks.rados.rados.0.smithi155.stdout:391: finishing rollback tid 1 to smithi15528762-23 2024-04-19T01:05:32.833 INFO:tasks.rados.rados.0.smithi155.stdout:391: finishing rollback tid 2 to smithi15528762-23 2024-04-19T01:05:32.833 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 23 v 109 (ObjNum 111 snap 26 seq_num 111) dirty exists 2024-04-19T01:05:32.833 INFO:tasks.rados.rados.0.smithi155.stdout:390: finishing write tid 1 to smithi15528762-41 2024-04-19T01:05:32.833 INFO:tasks.rados.rados.0.smithi155.stdout:390: finishing write tid 2 to smithi15528762-41 2024-04-19T01:05:32.833 INFO:tasks.rados.rados.0.smithi155.stdout:390: finishing write tid 3 to smithi15528762-41 2024-04-19T01:05:32.833 INFO:tasks.rados.rados.0.smithi155.stdout:390: oid 41 updating version 0 to 77 2024-04-19T01:05:32.833 INFO:tasks.rados.rados.0.smithi155.stdout:390: oid 41 updating version 77 to 78 2024-04-19T01:05:32.833 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 41 v 78 (ObjNum 153 snap 46 seq_num 153) dirty exists 2024-04-19T01:05:32.833 INFO:tasks.rados.rados.0.smithi155.stdout:390: left oid 41 (ObjNum 153 snap 46 seq_num 153) 2024-04-19T01:05:32.833 INFO:tasks.rados.rados.0.smithi155.stdout:389: finishing copy_from to smithi15528762-20 2024-04-19T01:05:32.833 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 20 v 157 (ObjNum 56 snap 5 seq_num 56) dirty exists 2024-04-19T01:05:32.833 INFO:tasks.rados.rados.0.smithi155.stdout:383: done (4 left) 2024-04-19T01:05:32.834 INFO:tasks.rados.rados.0.smithi155.stdout:388: done (3 left) 2024-04-19T01:05:32.834 INFO:tasks.rados.rados.0.smithi155.stdout:389: done (2 left) 2024-04-19T01:05:32.834 INFO:tasks.rados.rados.0.smithi155.stdout:390: done (1 left) 2024-04-19T01:05:32.834 INFO:tasks.rados.rados.0.smithi155.stdout:391: done (0 left) 2024-04-19T01:05:32.834 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 45 current snap is 46 2024-04-19T01:05:32.834 INFO:tasks.rados.rados.0.smithi155.stdout:392: seq_num 154 ranges {2932736=180224} 2024-04-19T01:05:32.839 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:05:32.840+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:05:32.867 INFO:tasks.rados.rados.0.smithi155.stdout:392: writing smithi15528762-45 from 2932736 to 3112960 tid 1 2024-04-19T01:05:32.867 INFO:tasks.rados.rados.0.smithi155.stdout:393: snap_remove snap 45 2024-04-19T01:05:32.912 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-04-19T01:05:32.909+0000 7f2d70738640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25027) UID: 0 2024-04-19T01:05:32.917 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-19T01:05:32.917 INFO:teuthology.orchestra.run.smithi110.stdout: "size": 20, 2024-04-19T01:05:32.918 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 600, 2024-04-19T01:05:32.918 INFO:teuthology.orchestra.run.smithi110.stdout: "ops": [ 2024-04-19T01:05:32.918 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.918 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:235 3.9s0 3:93d266e5:::smithi15528762-34:head [append 0~688128] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:05:32.918 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.857683+0000", 2024-04-19T01:05:32.918 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 79.057722415000001, 2024-04-19T01:05:32.918 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.044416514999999997, 2024-04-19T01:05:32.918 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:32.918 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:32.918 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:32.919 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:32.919 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:32.919 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:32.919 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 235 2024-04-19T01:05:32.919 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.919 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:32.919 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.919 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:32.919 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.857683+0000", 2024-04-19T01:05:32.919 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.919 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.919 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.919 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:32.920 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.857683+0000", 2024-04-19T01:05:32.920 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.920 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.920 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.920 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:32.920 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.857687+0000", 2024-04-19T01:05:32.920 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.1130000000000001e-06 2024-04-19T01:05:32.920 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.920 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.920 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:32.920 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.859118+0000", 2024-04-19T01:05:32.920 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001431114 2024-04-19T01:05:32.920 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.920 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.921 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:32.921 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.859150+0000", 2024-04-19T01:05:32.921 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.2589000000000003e-05 2024-04-19T01:05:32.921 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.921 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.921 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:32.921 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.860377+0000", 2024-04-19T01:05:32.921 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001226878 2024-04-19T01:05:32.921 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.921 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.921 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:32.921 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.866898+0000", 2024-04-19T01:05:32.921 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0065209250000000003 2024-04-19T01:05:32.922 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.922 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.922 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:32.922 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.867163+0000", 2024-04-19T01:05:32.922 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000264901 2024-04-19T01:05:32.922 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.922 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.922 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:32.922 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.869905+0000", 2024-04-19T01:05:32.922 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0027423550000000001 2024-04-19T01:05:32.922 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.923 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.923 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:32.923 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.897422+0000", 2024-04-19T01:05:32.923 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.027516445 2024-04-19T01:05:32.923 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.923 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.923 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:32.923 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.901877+0000", 2024-04-19T01:05:32.923 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.004454965 2024-04-19T01:05:32.924 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.924 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.924 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:32.924 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.902100+0000", 2024-04-19T01:05:32.924 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00022322999999999999 2024-04-19T01:05:32.924 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.924 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:32.924 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.924 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.924 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.924 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:238 3.9s0 3:93d266e5:::smithi15528762-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:05:32.924 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:13.861534+0000", 2024-04-19T01:05:32.924 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 79.053871650999994, 2024-04-19T01:05:32.924 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.046352598000000002, 2024-04-19T01:05:32.925 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:32.925 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:32.925 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:05:32.925 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:32.925 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:32.925 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:32.925 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 238 2024-04-19T01:05:32.925 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.925 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:32.925 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.925 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:32.925 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861534+0000", 2024-04-19T01:05:32.925 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.926 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.927 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.927 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:32.927 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861534+0000", 2024-04-19T01:05:32.927 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.927 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.927 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.927 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:32.927 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861536+0000", 2024-04-19T01:05:32.927 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.607e-06 2024-04-19T01:05:32.927 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.927 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.927 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:32.927 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861542+0000", 2024-04-19T01:05:32.927 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.1689999999999997e-06 2024-04-19T01:05:32.928 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.928 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.928 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:32.928 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861562+0000", 2024-04-19T01:05:32.928 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0043000000000001e-05 2024-04-19T01:05:32.928 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.928 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.928 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:32.928 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.861882+0000", 2024-04-19T01:05:32.928 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000320398 2024-04-19T01:05:32.928 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.928 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.928 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:32.928 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.901266+0000", 2024-04-19T01:05:32.929 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.039383373999999999 2024-04-19T01:05:32.929 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.929 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.929 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting for rw locks", 2024-04-19T01:05:32.929 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.901474+0000", 2024-04-19T01:05:32.929 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00020846999999999999 2024-04-19T01:05:32.929 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.930 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.930 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:32.931 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.904551+0000", 2024-04-19T01:05:32.931 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0030766690000000002 2024-04-19T01:05:32.931 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.931 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.931 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:32.931 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.904806+0000", 2024-04-19T01:05:32.931 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00025481899999999999 2024-04-19T01:05:32.931 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.932 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.932 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:32.932 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:13.907887+0000", 2024-04-19T01:05:32.932 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0030810490000000002 2024-04-19T01:05:32.932 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.932 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:32.932 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.932 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.932 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.933 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:267 3.9s0 3:93d266e5:::smithi15528762-34:head [append 0~737280] snapc 7=[7,4,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:05:32.933 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.013235+0000", 2024-04-19T01:05:32.933 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 78.902170530000006, 2024-04-19T01:05:32.933 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.043578804999999998, 2024-04-19T01:05:32.933 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:32.933 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:32.933 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:32.933 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:32.933 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:32.933 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:32.933 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 267 2024-04-19T01:05:32.933 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.933 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:32.933 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.934 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:32.934 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.013235+0000", 2024-04-19T01:05:32.934 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.934 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.934 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.934 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:32.934 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.013235+0000", 2024-04-19T01:05:32.934 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.934 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.934 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.934 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:32.934 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.013239+0000", 2024-04-19T01:05:32.934 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.6200000000000001e-06 2024-04-19T01:05:32.935 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.935 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.935 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:32.935 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.014549+0000", 2024-04-19T01:05:32.935 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001310075 2024-04-19T01:05:32.935 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.935 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.935 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:32.935 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.014585+0000", 2024-04-19T01:05:32.935 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.5494999999999998e-05 2024-04-19T01:05:32.935 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.935 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.935 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:32.936 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.014607+0000", 2024-04-19T01:05:32.936 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2371999999999999e-05 2024-04-19T01:05:32.936 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.936 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.936 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:32.936 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.014744+0000", 2024-04-19T01:05:32.936 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000136988 2024-04-19T01:05:32.936 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.936 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.936 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:32.936 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.015097+0000", 2024-04-19T01:05:32.937 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00035269400000000002 2024-04-19T01:05:32.937 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.937 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.937 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:32.937 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.020447+0000", 2024-04-19T01:05:32.937 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0053503739999999998 2024-04-19T01:05:32.937 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.937 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.937 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:32.937 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.047175+0000", 2024-04-19T01:05:32.937 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.026728372 2024-04-19T01:05:32.938 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.938 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.938 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:32.938 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.056595+0000", 2024-04-19T01:05:32.938 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0094197320000000001 2024-04-19T01:05:32.938 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.938 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.938 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:32.938 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.056814+0000", 2024-04-19T01:05:32.938 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00021908299999999999 2024-04-19T01:05:32.938 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.938 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:32.938 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.939 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.939 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.939 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:270 3.9s0 3:93d266e5:::smithi15528762-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-19T01:05:32.939 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.016503+0000", 2024-04-19T01:05:32.939 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 78.898902366000002, 2024-04-19T01:05:32.939 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.044494194000000001, 2024-04-19T01:05:32.939 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:32.939 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:32.939 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:05:32.939 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:32.939 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:32.939 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:32.939 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 270 2024-04-19T01:05:32.939 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.939 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:32.940 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.940 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:32.940 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016503+0000", 2024-04-19T01:05:32.940 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.940 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.940 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.940 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:32.940 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016503+0000", 2024-04-19T01:05:32.940 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.940 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.940 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.940 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:32.940 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016505+0000", 2024-04-19T01:05:32.940 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.2729999999999999e-06 2024-04-19T01:05:32.941 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.941 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.941 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:32.941 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016511+0000", 2024-04-19T01:05:32.941 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 5.7570000000000001e-06 2024-04-19T01:05:32.941 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.941 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.941 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:32.941 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016530+0000", 2024-04-19T01:05:32.941 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.9565999999999999e-05 2024-04-19T01:05:32.941 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.941 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.941 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:32.941 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.016545+0000", 2024-04-19T01:05:32.942 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.4963000000000001e-05 2024-04-19T01:05:32.942 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.942 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.942 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:32.942 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.049233+0000", 2024-04-19T01:05:32.942 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.032687815000000002 2024-04-19T01:05:32.942 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.943 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.943 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting for rw locks", 2024-04-19T01:05:32.943 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.049365+0000", 2024-04-19T01:05:32.943 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00013258399999999999 2024-04-19T01:05:32.943 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.943 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.943 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:32.943 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.058814+0000", 2024-04-19T01:05:32.943 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0094481080000000002 2024-04-19T01:05:32.943 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.943 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.943 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:32.943 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.059010+0000", 2024-04-19T01:05:32.944 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00019622699999999999 2024-04-19T01:05:32.944 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.944 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.944 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:32.944 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.060998+0000", 2024-04-19T01:05:32.944 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0019879009999999998 2024-04-19T01:05:32.944 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.944 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:32.944 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.944 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.944 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.944 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:280 3.9s0 3:98647342:::smithi15528762-43:head [append 0~753664] snapc 7=[7,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-19T01:05:32.944 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.869239+0000", 2024-04-19T01:05:32.944 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 78.046166497000002, 2024-04-19T01:05:32.945 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.062785076999999995, 2024-04-19T01:05:32.945 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:32.945 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:32.945 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:32.945 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:32.945 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:32.945 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:32.945 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 280 2024-04-19T01:05:32.945 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.945 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:32.945 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.945 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:32.945 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.869239+0000", 2024-04-19T01:05:32.946 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.946 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.946 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.946 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:32.946 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.869239+0000", 2024-04-19T01:05:32.946 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.946 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.946 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.946 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:32.946 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.869243+0000", 2024-04-19T01:05:32.946 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.444e-06 2024-04-19T01:05:32.946 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.946 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.946 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:32.947 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.870594+0000", 2024-04-19T01:05:32.947 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001351662 2024-04-19T01:05:32.947 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.947 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.947 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:32.947 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.870629+0000", 2024-04-19T01:05:32.947 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.4329000000000003e-05 2024-04-19T01:05:32.947 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.947 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.947 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:32.947 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.870647+0000", 2024-04-19T01:05:32.947 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7768999999999999e-05 2024-04-19T01:05:32.947 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.948 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.948 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:32.948 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.870806+0000", 2024-04-19T01:05:32.948 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015962300000000001 2024-04-19T01:05:32.948 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.948 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.948 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "op must wait for map", 2024-04-19T01:05:32.948 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.870819+0000", 2024-04-19T01:05:32.948 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.3266e-05 2024-04-19T01:05:32.948 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.948 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.948 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:32.948 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.875741+0000", 2024-04-19T01:05:32.948 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0049211599999999999 2024-04-19T01:05:32.949 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.949 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.949 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:32.949 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.876018+0000", 2024-04-19T01:05:32.949 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000276944 2024-04-19T01:05:32.949 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.949 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.949 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:32.949 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.880141+0000", 2024-04-19T01:05:32.949 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0041231030000000004 2024-04-19T01:05:32.949 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.949 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.949 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:32.949 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.915254+0000", 2024-04-19T01:05:32.950 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.035113610000000003 2024-04-19T01:05:32.950 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.950 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.950 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:32.950 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.931834+0000", 2024-04-19T01:05:32.950 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.016580212 2024-04-19T01:05:32.950 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.950 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.950 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:32.950 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.932024+0000", 2024-04-19T01:05:32.950 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00018995500000000001 2024-04-19T01:05:32.950 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.950 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:32.950 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.951 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.951 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.951 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:281 3.9s0 3:98647342:::smithi15528762-43:head [append 0~376832] snapc 7=[7,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-19T01:05:32.951 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.872157+0000", 2024-04-19T01:05:32.951 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 78.043248874, 2024-04-19T01:05:32.951 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.061221235999999998, 2024-04-19T01:05:32.951 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:32.951 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:32.951 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:32.951 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:32.951 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:32.951 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:32.951 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 281 2024-04-19T01:05:32.951 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.952 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:32.952 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.952 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:32.952 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872157+0000", 2024-04-19T01:05:32.952 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.952 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.952 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.952 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:32.953 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872157+0000", 2024-04-19T01:05:32.953 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.953 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.953 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.953 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:32.953 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872159+0000", 2024-04-19T01:05:32.953 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5129999999999999e-06 2024-04-19T01:05:32.953 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.953 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.953 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:32.953 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872435+0000", 2024-04-19T01:05:32.953 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00027559500000000001 2024-04-19T01:05:32.953 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.953 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.954 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:32.954 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872470+0000", 2024-04-19T01:05:32.954 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.5170999999999999e-05 2024-04-19T01:05:32.954 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.954 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.954 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:32.954 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872488+0000", 2024-04-19T01:05:32.954 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.8136000000000001e-05 2024-04-19T01:05:32.954 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.954 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.954 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:32.954 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872639+0000", 2024-04-19T01:05:32.954 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015092 2024-04-19T01:05:32.954 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.955 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.955 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:05:32.955 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872658+0000", 2024-04-19T01:05:32.955 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.8882999999999999e-05 2024-04-19T01:05:32.955 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.955 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.955 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:32.955 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.903660+0000", 2024-04-19T01:05:32.955 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.031001846999999999 2024-04-19T01:05:32.955 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.955 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.955 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:32.955 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.904268+0000", 2024-04-19T01:05:32.955 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00060793999999999996 2024-04-19T01:05:32.956 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.956 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.956 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:32.956 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.906536+0000", 2024-04-19T01:05:32.956 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0022680180000000001 2024-04-19T01:05:32.956 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.956 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.956 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:32.956 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.918698+0000", 2024-04-19T01:05:32.956 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.012161852000000001 2024-04-19T01:05:32.956 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.956 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.956 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:32.956 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.933228+0000", 2024-04-19T01:05:32.957 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.014529781 2024-04-19T01:05:32.957 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.957 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.957 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:32.957 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.933378+0000", 2024-04-19T01:05:32.957 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015058000000000001 2024-04-19T01:05:32.957 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.957 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:32.957 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.957 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.957 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.957 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:282 3.9s0 3:98647342:::smithi15528762-43:head [setxattr _header (56) in=7b] snapc 7=[7,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-19T01:05:32.957 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.872536+0000", 2024-04-19T01:05:32.957 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 78.042869370999995, 2024-04-19T01:05:32.957 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.062122945999999998, 2024-04-19T01:05:32.958 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:32.958 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:32.958 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:32.958 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:32.958 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:32.958 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:32.958 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 282 2024-04-19T01:05:32.958 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.958 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:32.958 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.958 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:32.958 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872536+0000", 2024-04-19T01:05:32.958 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.958 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.959 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.959 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:32.959 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872536+0000", 2024-04-19T01:05:32.959 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.959 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.959 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.959 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:32.959 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872538+0000", 2024-04-19T01:05:32.959 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.423e-06 2024-04-19T01:05:32.959 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.959 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.959 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:32.959 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872548+0000", 2024-04-19T01:05:32.959 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.8430000000000005e-06 2024-04-19T01:05:32.960 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.960 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.960 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:32.960 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872569+0000", 2024-04-19T01:05:32.960 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1336e-05 2024-04-19T01:05:32.960 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.960 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.960 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:32.960 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872585+0000", 2024-04-19T01:05:32.960 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6288999999999999e-05 2024-04-19T01:05:32.960 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.960 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.960 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:32.960 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872783+0000", 2024-04-19T01:05:32.961 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000197488 2024-04-19T01:05:32.961 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.961 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.961 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:05:32.961 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872804+0000", 2024-04-19T01:05:32.961 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0839000000000002e-05 2024-04-19T01:05:32.961 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.961 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.961 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:32.961 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.915520+0000", 2024-04-19T01:05:32.961 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.042715887000000001 2024-04-19T01:05:32.961 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.961 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.961 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:32.962 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.915852+0000", 2024-04-19T01:05:32.962 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00033199899999999998 2024-04-19T01:05:32.962 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.962 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.962 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:32.962 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.916938+0000", 2024-04-19T01:05:32.962 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0010863940000000001 2024-04-19T01:05:32.962 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.962 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.962 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:32.962 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.918701+0000", 2024-04-19T01:05:32.962 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001763351 2024-04-19T01:05:32.962 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.963 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.963 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:32.963 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.934283+0000", 2024-04-19T01:05:32.963 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.015581526 2024-04-19T01:05:32.963 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.963 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.963 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:32.963 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.934659+0000", 2024-04-19T01:05:32.963 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00037657100000000002 2024-04-19T01:05:32.963 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.963 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:32.963 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.963 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.963 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.964 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:283 3.9s0 3:98647342:::smithi15528762-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-04-19T01:05:32.964 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:14.872609+0000", 2024-04-19T01:05:32.964 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 78.042796521, 2024-04-19T01:05:32.964 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.067525222999999995, 2024-04-19T01:05:32.964 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:32.964 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:32.964 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:05:32.964 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:32.964 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:32.964 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:32.964 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 283 2024-04-19T01:05:32.964 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.964 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:32.964 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.965 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:32.965 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872609+0000", 2024-04-19T01:05:32.965 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.965 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.965 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.965 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:32.965 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872609+0000", 2024-04-19T01:05:32.965 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.965 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.965 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.965 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:32.965 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872611+0000", 2024-04-19T01:05:32.965 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.367e-06 2024-04-19T01:05:32.965 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.966 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.966 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:32.966 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872617+0000", 2024-04-19T01:05:32.966 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.2400000000000004e-06 2024-04-19T01:05:32.966 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.966 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.966 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:32.966 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872636+0000", 2024-04-19T01:05:32.966 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.8941999999999999e-05 2024-04-19T01:05:32.966 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.966 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.966 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:32.966 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872652+0000", 2024-04-19T01:05:32.966 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5651999999999999e-05 2024-04-19T01:05:32.967 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.967 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.967 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:32.967 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872858+0000", 2024-04-19T01:05:32.967 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00020631100000000001 2024-04-19T01:05:32.967 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.967 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.967 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting_for_map not empty", 2024-04-19T01:05:32.967 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.872867+0000", 2024-04-19T01:05:32.967 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.5400000000000001e-06 2024-04-19T01:05:32.967 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.967 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.967 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:32.968 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.917636+0000", 2024-04-19T01:05:32.968 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.044768372000000001 2024-04-19T01:05:32.968 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.968 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.968 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting for rw locks", 2024-04-19T01:05:32.968 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.917899+0000", 2024-04-19T01:05:32.968 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00026336799999999999 2024-04-19T01:05:32.968 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.968 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.968 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:32.969 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.935711+0000", 2024-04-19T01:05:32.969 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.017811862000000001 2024-04-19T01:05:32.969 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.969 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.969 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:32.969 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.935937+0000", 2024-04-19T01:05:32.969 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00022568800000000001 2024-04-19T01:05:32.969 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.969 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.969 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:32.969 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:14.940135+0000", 2024-04-19T01:05:32.969 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0041978809999999997 2024-04-19T01:05:32.969 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.969 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:32.970 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.970 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.970 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.970 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:653 3.es0 3:716a2d39:::smithi15528762-27:head [copy-from ver 38] snapc 1d=[1d,1c,1b,14] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-04-19T01:05:32.970 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:04:56.558085+0000", 2024-04-19T01:05:32.970 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 36.357321040999999, 2024-04-19T01:05:32.970 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.050389499999999997, 2024-04-19T01:05:32.970 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:32.970 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:32.970 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:32.970 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:32.970 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:32.970 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:32.971 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 653 2024-04-19T01:05:32.971 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.971 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:32.971 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.971 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:32.971 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.558085+0000", 2024-04-19T01:05:32.971 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.971 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.971 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.971 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:32.971 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.558085+0000", 2024-04-19T01:05:32.971 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.971 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.971 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.972 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:32.972 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.558089+0000", 2024-04-19T01:05:32.972 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.8190000000000002e-06 2024-04-19T01:05:32.972 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.972 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.972 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:32.972 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.558106+0000", 2024-04-19T01:05:32.972 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6883000000000001e-05 2024-04-19T01:05:32.972 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.972 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.972 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:32.972 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.558148+0000", 2024-04-19T01:05:32.972 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.2970999999999999e-05 2024-04-19T01:05:32.972 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.973 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.973 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:32.973 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.558171+0000", 2024-04-19T01:05:32.973 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2728999999999999e-05 2024-04-19T01:05:32.973 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.973 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.973 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:32.973 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.558355+0000", 2024-04-19T01:05:32.973 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00018391699999999999 2024-04-19T01:05:32.973 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.973 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.973 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:32.973 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.558621+0000", 2024-04-19T01:05:32.973 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00026582199999999998 2024-04-19T01:05:32.973 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.974 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.974 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:32.974 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.571924+0000", 2024-04-19T01:05:32.974 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.013303429 2024-04-19T01:05:32.974 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.974 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.974 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:32.974 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.598434+0000", 2024-04-19T01:05:32.974 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.026509805000000001 2024-04-19T01:05:32.974 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.974 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.974 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:32.974 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.608345+0000", 2024-04-19T01:05:32.974 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0099107599999999994 2024-04-19T01:05:32.975 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.975 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.975 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:32.975 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:04:56.608474+0000", 2024-04-19T01:05:32.975 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00012936499999999999 2024-04-19T01:05:32.975 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.975 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:32.975 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.975 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.975 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.975 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:691 3.9s0 3:9a4f6987:::smithi15528762-44:head [append 0~475136] snapc 20=[20,1f,1e,1c] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e92)", 2024-04-19T01:05:32.975 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:02.623135+0000", 2024-04-19T01:05:32.975 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 30.292270743, 2024-04-19T01:05:32.975 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.063498073000000002, 2024-04-19T01:05:32.976 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:32.976 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:32.976 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:32.976 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:32.976 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:32.976 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:32.976 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 691 2024-04-19T01:05:32.976 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.976 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:32.976 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.976 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:32.976 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.623135+0000", 2024-04-19T01:05:32.976 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.976 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.976 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.977 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:32.977 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.623135+0000", 2024-04-19T01:05:32.977 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.977 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.977 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.977 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:32.977 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.623137+0000", 2024-04-19T01:05:32.977 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.2060000000000001e-06 2024-04-19T01:05:32.977 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.977 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.977 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:32.977 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.624004+0000", 2024-04-19T01:05:32.977 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00086699399999999995 2024-04-19T01:05:32.977 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.978 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.978 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:32.978 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.624031+0000", 2024-04-19T01:05:32.978 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6951999999999999e-05 2024-04-19T01:05:32.978 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.978 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.978 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:32.978 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.624057+0000", 2024-04-19T01:05:32.978 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5471999999999999e-05 2024-04-19T01:05:32.978 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.978 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.978 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:32.978 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.624250+0000", 2024-04-19T01:05:32.978 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00019280499999999999 2024-04-19T01:05:32.979 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.979 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.979 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "op must wait for map", 2024-04-19T01:05:32.979 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.624267+0000", 2024-04-19T01:05:32.979 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7734999999999999e-05 2024-04-19T01:05:32.979 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.979 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.979 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:32.979 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.628695+0000", 2024-04-19T01:05:32.979 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0044277780000000003 2024-04-19T01:05:32.979 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.979 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.979 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:32.979 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.629013+0000", 2024-04-19T01:05:32.980 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00031820500000000001 2024-04-19T01:05:32.980 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.980 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.980 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:32.980 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.631338+0000", 2024-04-19T01:05:32.980 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.002324951 2024-04-19T01:05:32.980 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.980 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.980 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:32.980 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.662889+0000", 2024-04-19T01:05:32.980 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.031550750000000002 2024-04-19T01:05:32.980 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.980 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.981 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:32.981 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.686483+0000", 2024-04-19T01:05:32.981 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.023594542 2024-04-19T01:05:32.981 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.981 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.981 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:32.981 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.686633+0000", 2024-04-19T01:05:32.981 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00014968300000000001 2024-04-19T01:05:32.981 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.981 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:32.981 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.981 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.981 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.981 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:692 3.9s0 3:9a4f6987:::smithi15528762-44:head [append 0~655360] snapc 20=[20,1f,1e,1c] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-04-19T01:05:32.981 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:02.626896+0000", 2024-04-19T01:05:32.982 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 30.288509949000002, 2024-04-19T01:05:32.982 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.065157396000000006, 2024-04-19T01:05:32.982 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:32.982 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:32.982 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:32.982 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:32.982 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:32.982 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:32.982 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 692 2024-04-19T01:05:32.982 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.982 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:32.982 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.982 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:32.983 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.626896+0000", 2024-04-19T01:05:32.983 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.983 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.983 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.983 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:32.983 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.626896+0000", 2024-04-19T01:05:32.983 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.983 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.983 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.983 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:32.983 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.626899+0000", 2024-04-19T01:05:32.983 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6840000000000001e-06 2024-04-19T01:05:32.983 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.983 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.983 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:32.984 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.628053+0000", 2024-04-19T01:05:32.984 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001153933 2024-04-19T01:05:32.984 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.984 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.984 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:32.984 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.628087+0000", 2024-04-19T01:05:32.984 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.4159000000000003e-05 2024-04-19T01:05:32.984 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.984 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.984 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:32.984 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.628111+0000", 2024-04-19T01:05:32.984 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.4752e-05 2024-04-19T01:05:32.984 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.984 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.985 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:32.985 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.643061+0000", 2024-04-19T01:05:32.985 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.014949234 2024-04-19T01:05:32.985 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.985 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.985 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:32.985 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.643383+0000", 2024-04-19T01:05:32.985 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00032213800000000003 2024-04-19T01:05:32.985 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.985 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.985 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:32.985 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.645699+0000", 2024-04-19T01:05:32.985 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.002316041 2024-04-19T01:05:32.985 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.986 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.986 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:32.986 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.683741+0000", 2024-04-19T01:05:32.986 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.038042377000000002 2024-04-19T01:05:32.986 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.986 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.986 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:32.986 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.691942+0000", 2024-04-19T01:05:32.986 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0082007139999999996 2024-04-19T01:05:32.986 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.986 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.986 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:32.986 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.692053+0000", 2024-04-19T01:05:32.986 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000111364 2024-04-19T01:05:32.987 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.987 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:32.987 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.987 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.987 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.987 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:693 3.9s0 3:9a4f6987:::smithi15528762-44:head [append 0~622592] snapc 20=[20,1f,1e,1c] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-04-19T01:05:32.987 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:02.632540+0000", 2024-04-19T01:05:32.987 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 30.282865817000001, 2024-04-19T01:05:32.987 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.061723067999999999, 2024-04-19T01:05:32.987 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:32.987 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:32.987 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:32.987 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:32.987 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:32.988 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:32.988 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 693 2024-04-19T01:05:32.988 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.988 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:32.988 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.988 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:32.988 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.632540+0000", 2024-04-19T01:05:32.988 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.988 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.988 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.988 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:32.988 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.632540+0000", 2024-04-19T01:05:32.988 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.988 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.988 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.989 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:32.989 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.632542+0000", 2024-04-19T01:05:32.989 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.04e-06 2024-04-19T01:05:32.989 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.989 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.989 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:32.989 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.633600+0000", 2024-04-19T01:05:32.989 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0010575750000000001 2024-04-19T01:05:32.989 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.989 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.989 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:32.989 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.633632+0000", 2024-04-19T01:05:32.989 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.2199000000000001e-05 2024-04-19T01:05:32.989 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.990 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.990 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:32.990 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.633648+0000", 2024-04-19T01:05:32.990 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6019e-05 2024-04-19T01:05:32.990 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.990 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.990 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:32.990 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.663173+0000", 2024-04-19T01:05:32.990 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.029525526 2024-04-19T01:05:32.990 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.990 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.990 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:32.990 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.663337+0000", 2024-04-19T01:05:32.990 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00016387300000000001 2024-04-19T01:05:32.992 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.992 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.992 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:32.992 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.664809+0000", 2024-04-19T01:05:32.992 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001471633 2024-04-19T01:05:32.992 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.992 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.992 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:32.992 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.683991+0000", 2024-04-19T01:05:32.992 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.01918226 2024-04-19T01:05:32.992 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.992 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.993 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:32.993 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.694158+0000", 2024-04-19T01:05:32.993 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.010166735999999999 2024-04-19T01:05:32.993 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.993 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.993 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:32.993 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.694263+0000", 2024-04-19T01:05:32.993 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000105207 2024-04-19T01:05:32.993 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.993 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:32.993 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:32.993 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.993 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.993 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:694 3.9s0 3:9a4f6987:::smithi15528762-44:head [append 0~278528] snapc 20=[20,1f,1e,1c] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-04-19T01:05:32.994 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:02.635263+0000", 2024-04-19T01:05:32.994 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 30.280143173999999, 2024-04-19T01:05:32.994 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.059764666000000001, 2024-04-19T01:05:32.994 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:32.994 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:32.994 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:32.994 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:32.994 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:32.994 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:32.994 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 694 2024-04-19T01:05:32.994 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.994 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:32.994 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.995 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:32.995 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.635263+0000", 2024-04-19T01:05:32.995 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.995 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.995 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.995 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:32.995 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.635263+0000", 2024-04-19T01:05:32.995 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:32.995 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.995 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.995 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:32.995 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.635265+0000", 2024-04-19T01:05:32.995 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.6129999999999998e-06 2024-04-19T01:05:32.995 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.995 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.996 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:32.996 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.635755+0000", 2024-04-19T01:05:32.996 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00048946700000000005 2024-04-19T01:05:32.996 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.996 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.996 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:32.996 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.635792+0000", 2024-04-19T01:05:32.996 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.6789000000000003e-05 2024-04-19T01:05:32.996 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.996 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.996 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:32.996 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.635810+0000", 2024-04-19T01:05:32.996 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.7999000000000002e-05 2024-04-19T01:05:32.996 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.997 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.997 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:32.997 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.676857+0000", 2024-04-19T01:05:32.997 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.041047229999999997 2024-04-19T01:05:32.997 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.997 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.997 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:32.997 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.677021+0000", 2024-04-19T01:05:32.997 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000164339 2024-04-19T01:05:32.997 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.997 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.997 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:32.997 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.678298+0000", 2024-04-19T01:05:32.997 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001277048 2024-04-19T01:05:32.998 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.998 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.999 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:32.999 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.686286+0000", 2024-04-19T01:05:32.999 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0079883290000000006 2024-04-19T01:05:32.999 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.999 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.999 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:32.999 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.694934+0000", 2024-04-19T01:05:32.999 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0086471780000000002 2024-04-19T01:05:32.999 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:32.999 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:32.999 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:32.999 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.695027+0000", 2024-04-19T01:05:32.999 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.3673999999999998e-05 2024-04-19T01:05:32.999 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:33.000 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:33.000 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:33.000 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.000 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.000 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:695 3.9s0 3:9a4f6987:::smithi15528762-44:head [setxattr _header (57) in=7b] snapc 20=[20,1f,1e,1c] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-04-19T01:05:33.000 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:02.635844+0000", 2024-04-19T01:05:33.000 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 30.279561594, 2024-04-19T01:05:33.000 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.060048563999999999, 2024-04-19T01:05:33.000 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:33.000 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:33.000 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:33.000 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:33.000 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:33.001 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:33.001 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 695 2024-04-19T01:05:33.001 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.001 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:33.001 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.001 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:33.001 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.635844+0000", 2024-04-19T01:05:33.001 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:33.001 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.001 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.001 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:33.001 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.635844+0000", 2024-04-19T01:05:33.001 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:33.001 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.002 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.002 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:33.002 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.635846+0000", 2024-04-19T01:05:33.002 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0260000000000002e-06 2024-04-19T01:05:33.002 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.002 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.002 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:33.002 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.635855+0000", 2024-04-19T01:05:33.002 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.8200000000000003e-06 2024-04-19T01:05:33.002 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.002 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.002 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:33.002 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.635880+0000", 2024-04-19T01:05:33.003 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.5386000000000001e-05 2024-04-19T01:05:33.003 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.003 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.003 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:33.003 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.635896+0000", 2024-04-19T01:05:33.003 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.5702e-05 2024-04-19T01:05:33.003 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.003 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.003 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:33.003 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.684339+0000", 2024-04-19T01:05:33.003 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.048443084999999997 2024-04-19T01:05:33.003 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.003 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.003 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:33.004 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.684491+0000", 2024-04-19T01:05:33.004 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00015182999999999999 2024-04-19T01:05:33.004 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.004 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.004 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:33.004 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.685330+0000", 2024-04-19T01:05:33.004 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00083926499999999995 2024-04-19T01:05:33.004 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.004 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.004 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:33.004 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.686992+0000", 2024-04-19T01:05:33.004 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0016613839999999999 2024-04-19T01:05:33.004 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.004 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.004 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:33.005 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.695762+0000", 2024-04-19T01:05:33.005 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0087698370000000008 2024-04-19T01:05:33.005 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.005 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.005 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:33.005 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.695893+0000", 2024-04-19T01:05:33.005 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00013122899999999999 2024-04-19T01:05:33.005 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:33.005 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:33.005 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:33.005 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.005 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.005 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:696 3.9s0 3:9a4f6987:::smithi15528762-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e92)", 2024-04-19T01:05:33.005 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:02.635912+0000", 2024-04-19T01:05:33.006 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 30.279493556999999, 2024-04-19T01:05:33.006 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.063685439999999996, 2024-04-19T01:05:33.006 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:33.006 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:33.006 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:05:33.006 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:33.006 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:33.006 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:33.006 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 696 2024-04-19T01:05:33.006 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.006 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:33.006 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.006 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:33.006 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.635912+0000", 2024-04-19T01:05:33.007 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:33.007 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.007 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.007 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:33.007 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.635912+0000", 2024-04-19T01:05:33.007 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:33.008 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.008 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.008 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:33.008 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.635913+0000", 2024-04-19T01:05:33.008 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.7000000000000003e-07 2024-04-19T01:05:33.008 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.008 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.008 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:33.008 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.635918+0000", 2024-04-19T01:05:33.008 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.6929999999999998e-06 2024-04-19T01:05:33.008 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.008 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.008 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:33.008 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.635929+0000", 2024-04-19T01:05:33.009 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.1436e-05 2024-04-19T01:05:33.009 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.009 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.009 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:33.009 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.635938+0000", 2024-04-19T01:05:33.009 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.5799999999999992e-06 2024-04-19T01:05:33.009 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.009 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.009 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:33.009 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.685913+0000", 2024-04-19T01:05:33.009 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.049975064999999999 2024-04-19T01:05:33.009 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.009 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.009 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting for rw locks", 2024-04-19T01:05:33.010 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.686062+0000", 2024-04-19T01:05:33.010 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000149233 2024-04-19T01:05:33.010 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.010 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.010 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:33.010 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.696480+0000", 2024-04-19T01:05:33.010 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.010417493 2024-04-19T01:05:33.010 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.011 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.011 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:33.011 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.696592+0000", 2024-04-19T01:05:33.011 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000112725 2024-04-19T01:05:33.011 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.011 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.011 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:33.011 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.699598+0000", 2024-04-19T01:05:33.011 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0030052450000000001 2024-04-19T01:05:33.011 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:33.012 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:33.012 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:33.012 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.012 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.012 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:701 3.es0 3:7dbf7e9d:::smithi15528762-19:head [copy-from ver 30] snapc 20=[20,1f,1e,1c] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-04-19T01:05:33.012 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:02.648461+0000", 2024-04-19T01:05:33.012 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 30.266945204999999, 2024-04-19T01:05:33.012 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.078755795000000003, 2024-04-19T01:05:33.012 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:33.012 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:33.012 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:33.012 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:33.012 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:33.012 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:33.013 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 701 2024-04-19T01:05:33.013 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.013 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:33.013 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.013 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:33.013 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.648461+0000", 2024-04-19T01:05:33.013 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:33.013 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.013 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.013 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:33.013 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.648461+0000", 2024-04-19T01:05:33.013 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:33.013 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.013 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.014 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:33.014 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.648464+0000", 2024-04-19T01:05:33.014 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.2969999999999999e-06 2024-04-19T01:05:33.014 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.014 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.014 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:33.014 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.648477+0000", 2024-04-19T01:05:33.014 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.345e-05 2024-04-19T01:05:33.014 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.014 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.014 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:33.014 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.648515+0000", 2024-04-19T01:05:33.014 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 3.7314999999999998e-05 2024-04-19T01:05:33.014 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.015 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.015 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:33.015 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.648536+0000", 2024-04-19T01:05:33.015 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.0995000000000001e-05 2024-04-19T01:05:33.015 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.015 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.015 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:33.015 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.648698+0000", 2024-04-19T01:05:33.015 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000162509 2024-04-19T01:05:33.015 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.015 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.015 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:33.015 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.648869+0000", 2024-04-19T01:05:33.016 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00017036300000000001 2024-04-19T01:05:33.016 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.016 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.016 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_started", 2024-04-19T01:05:33.016 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.701127+0000", 2024-04-19T01:05:33.016 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.052258617 2024-04-19T01:05:33.016 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.016 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.016 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "sub_op_committed", 2024-04-19T01:05:33.016 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.726095+0000", 2024-04-19T01:05:33.016 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.024967335 2024-04-19T01:05:33.016 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.016 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.016 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "commit_sent", 2024-04-19T01:05:33.017 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.727001+0000", 2024-04-19T01:05:33.017 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00090626200000000002 2024-04-19T01:05:33.017 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.017 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.017 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:33.017 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.727216+0000", 2024-04-19T01:05:33.017 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00021565199999999999 2024-04-19T01:05:33.017 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:33.017 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:33.017 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:33.017 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.017 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.017 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:702 3.es0 3:7dbf7e9d:::smithi15528762-19:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e92)", 2024-04-19T01:05:33.017 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:02.648585+0000", 2024-04-19T01:05:33.018 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 30.266821293, 2024-04-19T01:05:33.018 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.076795295999999999, 2024-04-19T01:05:33.018 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:33.018 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:33.018 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:05:33.018 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:33.018 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:33.018 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:33.018 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 702 2024-04-19T01:05:33.018 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.018 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:33.018 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.018 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:33.019 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.648585+0000", 2024-04-19T01:05:33.019 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:33.019 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.019 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.019 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:33.019 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.648585+0000", 2024-04-19T01:05:33.019 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:33.019 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.019 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.019 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:33.019 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.648586+0000", 2024-04-19T01:05:33.019 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6360000000000001e-06 2024-04-19T01:05:33.019 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.019 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.019 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:33.020 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.648594+0000", 2024-04-19T01:05:33.020 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.5900000000000002e-06 2024-04-19T01:05:33.020 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.020 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.020 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:33.020 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.648615+0000", 2024-04-19T01:05:33.020 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.1274999999999999e-05 2024-04-19T01:05:33.020 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.020 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.020 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:33.020 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.648632+0000", 2024-04-19T01:05:33.020 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6555999999999999e-05 2024-04-19T01:05:33.020 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.020 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.021 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:33.021 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.649195+0000", 2024-04-19T01:05:33.021 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00056332500000000005 2024-04-19T01:05:33.021 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.021 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.021 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "waiting for blocked object", 2024-04-19T01:05:33.021 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.649276+0000", 2024-04-19T01:05:33.021 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.1063000000000006e-05 2024-04-19T01:05:33.021 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.021 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.021 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:33.021 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.724709+0000", 2024-04-19T01:05:33.021 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.075432568000000005 2024-04-19T01:05:33.022 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.022 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.022 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:33.022 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.725009+0000", 2024-04-19T01:05:33.022 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00030049999999999999 2024-04-19T01:05:33.022 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.022 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.022 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:33.022 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:02.725380+0000", 2024-04-19T01:05:33.022 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00037078300000000002 2024-04-19T01:05:33.022 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:33.022 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:33.022 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:33.022 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.023 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.023 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:718 3.9s0 3:93d266e5:::smithi15528762-34:1f [read 0~3637248,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e96)", 2024-04-19T01:05:33.023 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:06.525331+0000", 2024-04-19T01:05:33.023 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 26.390075144000001, 2024-04-19T01:05:33.023 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.043542420999999998, 2024-04-19T01:05:33.023 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:33.023 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:33.023 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:05:33.023 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:33.023 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:33.023 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:33.023 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 718 2024-04-19T01:05:33.023 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.023 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:33.024 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.024 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:33.024 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.525331+0000", 2024-04-19T01:05:33.024 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:33.024 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.024 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.024 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:33.024 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.525331+0000", 2024-04-19T01:05:33.024 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:33.024 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.024 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.024 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:33.024 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.525333+0000", 2024-04-19T01:05:33.024 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 2.08e-06 2024-04-19T01:05:33.025 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.025 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.025 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:33.025 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.525349+0000", 2024-04-19T01:05:33.025 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.6330000000000001e-05 2024-04-19T01:05:33.025 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.025 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.025 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:33.025 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.525362+0000", 2024-04-19T01:05:33.025 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.2697999999999999e-05 2024-04-19T01:05:33.025 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.025 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.025 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:33.025 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.525373+0000", 2024-04-19T01:05:33.026 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.1348999999999999e-05 2024-04-19T01:05:33.026 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.026 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.026 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:33.026 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.525454+0000", 2024-04-19T01:05:33.026 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.0939999999999994e-05 2024-04-19T01:05:33.026 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.026 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.026 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "op must wait for map", 2024-04-19T01:05:33.026 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.525463+0000", 2024-04-19T01:05:33.026 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.7250000000000003e-06 2024-04-19T01:05:33.026 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.026 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.026 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:33.027 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.528004+0000", 2024-04-19T01:05:33.027 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.0025409289999999999 2024-04-19T01:05:33.027 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.027 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.027 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:33.027 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.528118+0000", 2024-04-19T01:05:33.027 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000114113 2024-04-19T01:05:33.027 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.027 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.027 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:33.027 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.568873+0000", 2024-04-19T01:05:33.027 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.040755257000000003 2024-04-19T01:05:33.027 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:33.028 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:33.028 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:33.028 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.028 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.028 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:719 3.9s0 3:93d266e5:::smithi15528762-34:1f [read 0~3637248,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e96)", 2024-04-19T01:05:33.028 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:06.527295+0000", 2024-04-19T01:05:33.028 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 26.388110862000001, 2024-04-19T01:05:33.028 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.044122639999999998, 2024-04-19T01:05:33.028 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:33.028 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:33.028 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:05:33.028 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:33.028 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:33.028 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:33.029 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 719 2024-04-19T01:05:33.029 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.029 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:33.029 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.029 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:33.029 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.527295+0000", 2024-04-19T01:05:33.029 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:33.029 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.029 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.029 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:33.029 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.527295+0000", 2024-04-19T01:05:33.029 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:33.029 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.029 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.030 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:33.030 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.527296+0000", 2024-04-19T01:05:33.030 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.9999999999999995e-07 2024-04-19T01:05:33.030 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.030 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.030 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:33.030 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.528753+0000", 2024-04-19T01:05:33.030 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.001457042 2024-04-19T01:05:33.030 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.030 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.030 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:33.030 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.528767+0000", 2024-04-19T01:05:33.030 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 1.3534e-05 2024-04-19T01:05:33.030 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.031 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.031 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:33.031 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.528775+0000", 2024-04-19T01:05:33.031 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.8939999999999999e-06 2024-04-19T01:05:33.031 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.031 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.031 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:33.031 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.528944+0000", 2024-04-19T01:05:33.031 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00016899699999999999 2024-04-19T01:05:33.031 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.031 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.031 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:33.031 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.529052+0000", 2024-04-19T01:05:33.031 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.000107044 2024-04-19T01:05:33.032 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.032 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.032 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:33.032 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.571418+0000", 2024-04-19T01:05:33.032 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.042366129000000002 2024-04-19T01:05:33.032 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:33.032 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:33.032 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:33.032 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.032 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.032 INFO:teuthology.orchestra.run.smithi110.stdout: "description": "osd_op(client.4604.0:720 3.9s0 3:93d266e5:::smithi15528762-34:1f [read 0~3637248,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e96)", 2024-04-19T01:05:33.032 INFO:teuthology.orchestra.run.smithi110.stdout: "initiated_at": "2024-04-19T01:05:06.528800+0000", 2024-04-19T01:05:33.032 INFO:teuthology.orchestra.run.smithi110.stdout: "age": 26.386605431, 2024-04-19T01:05:33.033 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.044470728000000001, 2024-04-19T01:05:33.033 INFO:teuthology.orchestra.run.smithi110.stdout: "continuous": false, 2024-04-19T01:05:33.033 INFO:teuthology.orchestra.run.smithi110.stdout: "type_data": { 2024-04-19T01:05:33.033 INFO:teuthology.orchestra.run.smithi110.stdout: "flag_point": "started", 2024-04-19T01:05:33.033 INFO:teuthology.orchestra.run.smithi110.stdout: "client_info": { 2024-04-19T01:05:33.033 INFO:teuthology.orchestra.run.smithi110.stdout: "client": "client.4604", 2024-04-19T01:05:33.033 INFO:teuthology.orchestra.run.smithi110.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:33.033 INFO:teuthology.orchestra.run.smithi110.stdout: "tid": 720 2024-04-19T01:05:33.033 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.033 INFO:teuthology.orchestra.run.smithi110.stdout: "events": [ 2024-04-19T01:05:33.033 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.034 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "initiated", 2024-04-19T01:05:33.034 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.528800+0000", 2024-04-19T01:05:33.034 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:33.034 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.034 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.034 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "header_read", 2024-04-19T01:05:33.034 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.528800+0000", 2024-04-19T01:05:33.034 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0 2024-04-19T01:05:33.034 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.034 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.034 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "throttled", 2024-04-19T01:05:33.034 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.528801+0000", 2024-04-19T01:05:33.034 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 7.7499999999999999e-07 2024-04-19T01:05:33.034 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.035 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.035 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "all_read", 2024-04-19T01:05:33.035 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.528806+0000", 2024-04-19T01:05:33.035 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 4.9529999999999998e-06 2024-04-19T01:05:33.035 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.035 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.035 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "dispatched", 2024-04-19T01:05:33.035 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.528815+0000", 2024-04-19T01:05:33.035 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 8.5350000000000004e-06 2024-04-19T01:05:33.035 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.035 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.035 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "queued_for_pg", 2024-04-19T01:05:33.035 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.528821+0000", 2024-04-19T01:05:33.036 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 6.6950000000000003e-06 2024-04-19T01:05:33.036 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.036 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.036 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "reached_pg", 2024-04-19T01:05:33.036 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.529427+0000", 2024-04-19T01:05:33.036 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.00060544199999999998 2024-04-19T01:05:33.036 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.036 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.036 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "started", 2024-04-19T01:05:33.036 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.529519+0000", 2024-04-19T01:05:33.036 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 9.1864e-05 2024-04-19T01:05:33.036 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-19T01:05:33.036 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-04-19T01:05:33.036 INFO:teuthology.orchestra.run.smithi110.stdout: "event": "done", 2024-04-19T01:05:33.036 INFO:teuthology.orchestra.run.smithi110.stdout: "time": "2024-04-19T01:05:06.573271+0000", 2024-04-19T01:05:33.037 INFO:teuthology.orchestra.run.smithi110.stdout: "duration": 0.043752463999999998 2024-04-19T01:05:33.037 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:33.037 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:33.037 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:33.037 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-19T01:05:33.037 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-19T01:05:33.037 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-19T01:05:33.039 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-19T01:05:33.039 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:05:33.037+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:05:33.139 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:05:33.140+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:05:33.174 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:05:33.174 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-04-19T01:05:33.175 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-04-19T01:05:33.175 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:05:33.188 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-19T01:05:33.241 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:05:33.240+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:05:33.314 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:05:33.314 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-04-19T01:05:33.314 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-04-19T01:05:33.314 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-04-19T01:05:33.314 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:05:33.328 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-19T01:05:33.375 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:05:33.374+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:05:33.443 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:05:33.442+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:05:33.449 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:05:33.450 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-04-19T01:05:33.450 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-04-19T01:05:33.450 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-04-19T01:05:33.450 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.450 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 43/31 ECSubWrite(tid=43, reqid=client.4604.0:251, at_version=43'29, trim_to=0'0, roll_forward_to=43'27))", 2024-04-19T01:05:33.450 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.986035+0000", 2024-04-19T01:05:33.450 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 79.463111802, 2024-04-19T01:05:33.450 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024180535999999999, 2024-04-19T01:05:33.450 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:33.450 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:33.450 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:33.450 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:33.450 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.450 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:33.451 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986035+0000", 2024-04-19T01:05:33.451 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.451 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.451 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.451 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:33.451 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986035+0000", 2024-04-19T01:05:33.451 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.451 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.451 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.451 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:33.451 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986058+0000", 2024-04-19T01:05:33.451 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2903000000000001e-05 2024-04-19T01:05:33.451 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.452 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.452 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:33.452 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986841+0000", 2024-04-19T01:05:33.452 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00078285599999999996 2024-04-19T01:05:33.452 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.452 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.452 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:33.452 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986887+0000", 2024-04-19T01:05:33.452 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6202000000000003e-05 2024-04-19T01:05:33.452 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.452 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.452 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:33.452 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.986913+0000", 2024-04-19T01:05:33.452 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6162000000000001e-05 2024-04-19T01:05:33.452 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.453 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.453 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:33.453 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987069+0000", 2024-04-19T01:05:33.453 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015625099999999999 2024-04-19T01:05:33.453 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.453 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.453 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:33.453 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987099+0000", 2024-04-19T01:05:33.453 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9578999999999999e-05 2024-04-19T01:05:33.453 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.453 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.453 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:33.453 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.010110+0000", 2024-04-19T01:05:33.453 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023011357999999999 2024-04-19T01:05:33.454 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.454 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.454 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:33.454 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.010215+0000", 2024-04-19T01:05:33.454 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000105225 2024-04-19T01:05:33.454 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.454 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:33.454 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.454 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.454 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.454 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds5 43/31 ECSubWrite(tid=38, reqid=client.4604.0:253, at_version=43'14, trim_to=0'0, roll_forward_to=43'12))", 2024-04-19T01:05:33.454 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.987245+0000", 2024-04-19T01:05:33.454 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 79.461901780000005, 2024-04-19T01:05:33.454 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022838027, 2024-04-19T01:05:33.454 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:33.455 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:33.455 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:33.455 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:33.455 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.455 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:33.455 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987245+0000", 2024-04-19T01:05:33.455 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.455 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.455 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.455 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:33.455 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987245+0000", 2024-04-19T01:05:33.455 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.455 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.455 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.456 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:33.456 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.987247+0000", 2024-04-19T01:05:33.456 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2800000000000002e-06 2024-04-19T01:05:33.456 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.456 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.456 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:33.456 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988142+0000", 2024-04-19T01:05:33.456 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00089449199999999999 2024-04-19T01:05:33.456 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.456 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.456 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:33.456 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988180+0000", 2024-04-19T01:05:33.456 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8615000000000002e-05 2024-04-19T01:05:33.456 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.456 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.457 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:33.457 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988206+0000", 2024-04-19T01:05:33.457 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6023000000000001e-05 2024-04-19T01:05:33.457 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.457 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.457 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:33.457 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988349+0000", 2024-04-19T01:05:33.457 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014252199999999999 2024-04-19T01:05:33.457 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.457 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.457 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:33.457 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.988365+0000", 2024-04-19T01:05:33.457 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5855e-05 2024-04-19T01:05:33.457 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.458 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.458 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:33.458 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.010029+0000", 2024-04-19T01:05:33.458 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.021664454999999999 2024-04-19T01:05:33.458 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.458 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.458 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:33.458 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.010083+0000", 2024-04-19T01:05:33.458 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3785000000000001e-05 2024-04-19T01:05:33.459 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.459 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:33.459 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.459 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.459 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.459 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.fs6 43/31 ECSubWrite(tid=47, reqid=client.4604.0:263, at_version=43'8, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T01:05:33.459 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:13.994092+0000", 2024-04-19T01:05:33.459 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 79.455054794999995, 2024-04-19T01:05:33.459 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.027264745999999999, 2024-04-19T01:05:33.459 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:33.459 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:33.459 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:33.459 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:33.459 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.459 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:33.460 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.994092+0000", 2024-04-19T01:05:33.460 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.460 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.460 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.460 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:33.460 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.994092+0000", 2024-04-19T01:05:33.460 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.460 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.460 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.460 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:33.460 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.994094+0000", 2024-04-19T01:05:33.460 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1399999999999998e-06 2024-04-19T01:05:33.460 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.460 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.461 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:33.461 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.994786+0000", 2024-04-19T01:05:33.461 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00069247999999999998 2024-04-19T01:05:33.461 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.461 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.461 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:33.461 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.994827+0000", 2024-04-19T01:05:33.461 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0105000000000003e-05 2024-04-19T01:05:33.461 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.461 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.461 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:33.461 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.994853+0000", 2024-04-19T01:05:33.461 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6259e-05 2024-04-19T01:05:33.461 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.461 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.462 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:33.462 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.995043+0000", 2024-04-19T01:05:33.462 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0001906 2024-04-19T01:05:33.462 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.462 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.462 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:33.462 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:13.995069+0000", 2024-04-19T01:05:33.462 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5633e-05 2024-04-19T01:05:33.462 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.462 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.462 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:33.462 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.021288+0000", 2024-04-19T01:05:33.462 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.026218865000000001 2024-04-19T01:05:33.462 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.463 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.463 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:33.463 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:14.021357+0000", 2024-04-19T01:05:33.463 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8664000000000005e-05 2024-04-19T01:05:33.463 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.463 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:33.463 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.463 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.463 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.463 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.es4 47/31 ECSubWrite(tid=104, reqid=client.4604.0:312, at_version=47'18, trim_to=0'0, roll_forward_to=45'16))", 2024-04-19T01:05:33.463 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.582390+0000", 2024-04-19T01:05:33.463 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 75.866757014000001, 2024-04-19T01:05:33.463 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022795626999999999, 2024-04-19T01:05:33.463 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:33.464 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:33.464 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:33.464 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:33.464 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.464 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:33.464 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.582390+0000", 2024-04-19T01:05:33.464 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.464 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.464 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.464 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:33.464 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.582390+0000", 2024-04-19T01:05:33.464 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.464 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.464 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.464 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:33.465 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.582390+0000", 2024-04-19T01:05:33.465 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8299999999999996e-07 2024-04-19T01:05:33.465 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.465 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.465 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:33.465 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.582798+0000", 2024-04-19T01:05:33.465 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00040746499999999998 2024-04-19T01:05:33.465 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.465 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.465 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:33.465 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.582829+0000", 2024-04-19T01:05:33.465 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0988999999999998e-05 2024-04-19T01:05:33.465 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.465 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.465 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:33.466 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.582846+0000", 2024-04-19T01:05:33.466 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7252e-05 2024-04-19T01:05:33.466 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.466 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.466 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:33.466 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.582989+0000", 2024-04-19T01:05:33.466 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014342000000000001 2024-04-19T01:05:33.466 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.466 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.466 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:33.466 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.583006+0000", 2024-04-19T01:05:33.466 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6671999999999999e-05 2024-04-19T01:05:33.466 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.466 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.467 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:33.467 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.605104+0000", 2024-04-19T01:05:33.467 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022098222000000001 2024-04-19T01:05:33.467 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.467 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.467 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:33.467 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.605185+0000", 2024-04-19T01:05:33.467 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0823999999999997e-05 2024-04-19T01:05:33.467 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.467 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:33.467 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.467 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.467 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.467 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds5 47/31 ECSubWrite(tid=61, reqid=client.4604.0:330, at_version=47'32, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:05:33.467 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.677579+0000", 2024-04-19T01:05:33.468 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 75.771567954999995, 2024-04-19T01:05:33.468 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.033702862, 2024-04-19T01:05:33.468 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:33.468 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:33.468 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:33.468 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:33.468 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.468 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:33.468 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.677579+0000", 2024-04-19T01:05:33.468 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.468 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.468 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.468 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:33.469 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.677579+0000", 2024-04-19T01:05:33.469 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.469 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.469 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.469 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:33.469 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.677581+0000", 2024-04-19T01:05:33.469 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2800000000000002e-06 2024-04-19T01:05:33.469 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.469 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.469 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:33.470 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678062+0000", 2024-04-19T01:05:33.470 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00048067900000000002 2024-04-19T01:05:33.470 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.470 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.470 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:33.470 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678096+0000", 2024-04-19T01:05:33.470 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4381999999999997e-05 2024-04-19T01:05:33.470 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.470 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.470 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:33.470 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678117+0000", 2024-04-19T01:05:33.470 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0679999999999999e-05 2024-04-19T01:05:33.470 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.470 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.471 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:33.471 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678249+0000", 2024-04-19T01:05:33.471 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013211899999999999 2024-04-19T01:05:33.471 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.471 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.471 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:33.471 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.678270+0000", 2024-04-19T01:05:33.471 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1409000000000001e-05 2024-04-19T01:05:33.471 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.471 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.471 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:33.471 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711228+0000", 2024-04-19T01:05:33.471 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.032957575000000003 2024-04-19T01:05:33.471 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.471 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.472 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:33.472 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711281+0000", 2024-04-19T01:05:33.472 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3737999999999999e-05 2024-04-19T01:05:33.472 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.472 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:33.472 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.472 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.473 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.473 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds5 47/31 ECSubWrite(tid=62, reqid=client.4604.0:331, at_version=47'33, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:05:33.473 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.682545+0000", 2024-04-19T01:05:33.473 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 75.766601914000006, 2024-04-19T01:05:33.473 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.028771769999999999, 2024-04-19T01:05:33.473 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:33.473 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:33.473 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:33.473 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:33.473 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.473 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:33.473 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682545+0000", 2024-04-19T01:05:33.474 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.474 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.474 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.474 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:33.474 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682545+0000", 2024-04-19T01:05:33.474 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.474 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.474 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.474 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:33.474 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682546+0000", 2024-04-19T01:05:33.474 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.567e-06 2024-04-19T01:05:33.475 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.475 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.475 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:33.475 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682965+0000", 2024-04-19T01:05:33.475 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00041881399999999999 2024-04-19T01:05:33.475 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.475 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.476 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:33.476 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.682999+0000", 2024-04-19T01:05:33.476 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3738999999999997e-05 2024-04-19T01:05:33.476 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.476 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.476 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:33.476 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.683021+0000", 2024-04-19T01:05:33.477 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2592e-05 2024-04-19T01:05:33.477 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.477 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.477 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:33.477 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.691707+0000", 2024-04-19T01:05:33.477 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0086853629999999998 2024-04-19T01:05:33.477 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.477 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.477 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:33.477 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.691732+0000", 2024-04-19T01:05:33.477 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5366000000000001e-05 2024-04-19T01:05:33.477 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.477 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.478 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:33.478 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711288+0000", 2024-04-19T01:05:33.478 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019555922999999999 2024-04-19T01:05:33.478 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.478 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.478 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:33.478 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711316+0000", 2024-04-19T01:05:33.478 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8405999999999999e-05 2024-04-19T01:05:33.478 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.478 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:33.478 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.478 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.478 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.479 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds5 47/31 ECSubWrite(tid=63, reqid=client.4604.0:332, at_version=47'34, trim_to=0'0, roll_forward_to=47'30))", 2024-04-19T01:05:33.479 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:17.688708+0000", 2024-04-19T01:05:33.479 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 75.760439016999996, 2024-04-19T01:05:33.479 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.02327926, 2024-04-19T01:05:33.479 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:33.479 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:33.479 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:33.479 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:33.479 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.480 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:33.480 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.688708+0000", 2024-04-19T01:05:33.480 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.480 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.480 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.480 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:33.480 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.688708+0000", 2024-04-19T01:05:33.480 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.480 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.480 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.480 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:33.480 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.688709+0000", 2024-04-19T01:05:33.480 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5e-06 2024-04-19T01:05:33.481 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.481 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.481 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:33.481 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.688989+0000", 2024-04-19T01:05:33.481 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00028006800000000002 2024-04-19T01:05:33.481 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.481 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.481 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:33.481 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.689021+0000", 2024-04-19T01:05:33.481 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2026000000000001e-05 2024-04-19T01:05:33.481 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.481 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.481 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:33.481 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.689047+0000", 2024-04-19T01:05:33.481 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5809999999999999e-05 2024-04-19T01:05:33.482 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.482 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.482 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:33.482 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.704765+0000", 2024-04-19T01:05:33.482 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015717913 2024-04-19T01:05:33.483 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.483 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.483 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:33.483 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.704793+0000", 2024-04-19T01:05:33.483 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8143000000000001e-05 2024-04-19T01:05:33.483 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.483 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.483 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:33.483 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711907+0000", 2024-04-19T01:05:33.483 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071141879999999996 2024-04-19T01:05:33.483 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.483 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.483 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:33.484 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:17.711987+0000", 2024-04-19T01:05:33.484 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9611999999999996e-05 2024-04-19T01:05:33.484 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.484 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:33.484 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.484 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.485 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.485 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 59/31 ECSubWrite(tid=66, reqid=client.4604.0:423, at_version=59'51, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:05:33.485 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.541317+0000", 2024-04-19T01:05:33.485 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.907829411000002, 2024-04-19T01:05:33.485 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.047133948000000002, 2024-04-19T01:05:33.485 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:33.485 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:33.485 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:33.485 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:33.485 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.485 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:33.486 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.541317+0000", 2024-04-19T01:05:33.486 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.486 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.486 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.486 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:33.486 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.541317+0000", 2024-04-19T01:05:33.486 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.486 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.486 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.487 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:33.487 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.541318+0000", 2024-04-19T01:05:33.487 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3399999999999999e-07 2024-04-19T01:05:33.487 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.487 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.487 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:33.487 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.543831+0000", 2024-04-19T01:05:33.487 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025130999999999999 2024-04-19T01:05:33.487 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.487 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.487 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:33.487 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.543879+0000", 2024-04-19T01:05:33.488 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8208999999999999e-05 2024-04-19T01:05:33.488 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.488 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.488 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:33.488 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.543903+0000", 2024-04-19T01:05:33.488 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.404e-05 2024-04-19T01:05:33.488 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.488 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.488 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:33.488 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.544070+0000", 2024-04-19T01:05:33.488 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000166372 2024-04-19T01:05:33.488 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.488 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.488 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:33.489 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.544097+0000", 2024-04-19T01:05:33.489 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7256000000000001e-05 2024-04-19T01:05:33.489 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.489 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.489 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:33.489 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.588382+0000", 2024-04-19T01:05:33.489 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.044284780000000003 2024-04-19T01:05:33.489 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.489 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.489 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:33.489 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.588451+0000", 2024-04-19T01:05:33.489 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9556999999999998e-05 2024-04-19T01:05:33.489 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.489 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:33.490 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.490 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.490 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.490 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 59/31 ECSubWrite(tid=67, reqid=client.4604.0:424, at_version=59'52, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:05:33.490 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.548963+0000", 2024-04-19T01:05:33.490 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.900183644999998, 2024-04-19T01:05:33.490 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.039534443000000002, 2024-04-19T01:05:33.490 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:33.490 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:33.490 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:33.490 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:33.490 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.490 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:33.490 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.548963+0000", 2024-04-19T01:05:33.491 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.491 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.491 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.491 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:33.491 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.548963+0000", 2024-04-19T01:05:33.491 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.491 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.491 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.491 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:33.491 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.548965+0000", 2024-04-19T01:05:33.491 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1500000000000002e-06 2024-04-19T01:05:33.491 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.492 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.492 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:33.492 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549551+0000", 2024-04-19T01:05:33.492 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00058640299999999999 2024-04-19T01:05:33.492 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.492 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.492 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:33.492 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549596+0000", 2024-04-19T01:05:33.492 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4604999999999997e-05 2024-04-19T01:05:33.492 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.492 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.492 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:33.492 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.549631+0000", 2024-04-19T01:05:33.493 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4725999999999999e-05 2024-04-19T01:05:33.493 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.493 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.493 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:33.493 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.567909+0000", 2024-04-19T01:05:33.493 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018277755999999999 2024-04-19T01:05:33.493 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.493 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.494 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:33.494 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.567944+0000", 2024-04-19T01:05:33.495 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5862999999999999e-05 2024-04-19T01:05:33.495 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.495 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.495 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:33.495 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.588471+0000", 2024-04-19T01:05:33.495 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020526902999999999 2024-04-19T01:05:33.495 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.495 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.495 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:33.496 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.588497+0000", 2024-04-19T01:05:33.496 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6037000000000002e-05 2024-04-19T01:05:33.496 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.496 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:33.496 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.496 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.496 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.496 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 59/31 ECSubWrite(tid=68, reqid=client.4604.0:425, at_version=59'53, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:05:33.496 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.556523+0000", 2024-04-19T01:05:33.497 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.892623233999998, 2024-04-19T01:05:33.497 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.032109275999999999, 2024-04-19T01:05:33.497 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:33.497 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:33.497 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:33.497 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:33.497 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.497 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:33.497 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556523+0000", 2024-04-19T01:05:33.497 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.497 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.497 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.497 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:33.497 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556523+0000", 2024-04-19T01:05:33.498 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.498 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.498 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.498 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:33.498 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556526+0000", 2024-04-19T01:05:33.498 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4930000000000001e-06 2024-04-19T01:05:33.498 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.498 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.498 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:33.498 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556650+0000", 2024-04-19T01:05:33.498 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000124487 2024-04-19T01:05:33.499 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.499 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.499 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:33.499 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556701+0000", 2024-04-19T01:05:33.499 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1158e-05 2024-04-19T01:05:33.499 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.499 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.499 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:33.499 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.556727+0000", 2024-04-19T01:05:33.499 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5843999999999999e-05 2024-04-19T01:05:33.499 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.499 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.500 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:33.500 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.585945+0000", 2024-04-19T01:05:33.500 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0292181 2024-04-19T01:05:33.500 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.500 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.500 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:33.500 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.585972+0000", 2024-04-19T01:05:33.500 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6356e-05 2024-04-19T01:05:33.500 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.500 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.500 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:33.500 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.588601+0000", 2024-04-19T01:05:33.500 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0026295839999999999 2024-04-19T01:05:33.500 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.501 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.501 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:33.501 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.588633+0000", 2024-04-19T01:05:33.501 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1254000000000003e-05 2024-04-19T01:05:33.501 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.501 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:33.501 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.501 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.501 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.501 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 59/31 ECSubWrite(tid=69, reqid=client.4604.0:426, at_version=59'54, trim_to=0'0, roll_forward_to=58'50))", 2024-04-19T01:05:33.501 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:29.558033+0000", 2024-04-19T01:05:33.501 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.891113283999999, 2024-04-19T01:05:33.501 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.030934886000000002, 2024-04-19T01:05:33.501 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:33.501 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:33.502 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:33.502 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:33.502 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.502 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:33.502 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558033+0000", 2024-04-19T01:05:33.502 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.502 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.502 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.502 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:33.502 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558033+0000", 2024-04-19T01:05:33.502 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.502 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.502 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.502 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:33.503 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558035+0000", 2024-04-19T01:05:33.503 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7370000000000001e-06 2024-04-19T01:05:33.503 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.503 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.503 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:33.503 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558080+0000", 2024-04-19T01:05:33.503 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5089000000000001e-05 2024-04-19T01:05:33.503 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.503 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.504 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:33.504 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558126+0000", 2024-04-19T01:05:33.504 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5902000000000002e-05 2024-04-19T01:05:33.504 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.504 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.504 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:33.504 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.558149+0000", 2024-04-19T01:05:33.504 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3136e-05 2024-04-19T01:05:33.504 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.504 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.504 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:33.504 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.588071+0000", 2024-04-19T01:05:33.504 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.029921515999999999 2024-04-19T01:05:33.506 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.506 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.506 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:33.506 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.588089+0000", 2024-04-19T01:05:33.506 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8788e-05 2024-04-19T01:05:33.510 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.510 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.510 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:33.510 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.588944+0000", 2024-04-19T01:05:33.510 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00085441300000000005 2024-04-19T01:05:33.510 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.510 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.510 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:33.510 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:29.588968+0000", 2024-04-19T01:05:33.510 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4304999999999999e-05 2024-04-19T01:05:33.510 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.510 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:33.511 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.511 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.511 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.511 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 79/31 ECSubWrite(tid=115, reqid=client.4604.0:591, at_version=79'90, trim_to=0'0, roll_forward_to=79'86))", 2024-04-19T01:05:33.511 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:49.657667+0000", 2024-04-19T01:05:33.511 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 43.791479574, 2024-04-19T01:05:33.512 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.042570781000000002, 2024-04-19T01:05:33.512 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:33.512 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:33.512 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:33.513 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:33.513 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.513 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:33.513 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.657667+0000", 2024-04-19T01:05:33.513 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.513 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.513 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.513 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:33.513 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.657667+0000", 2024-04-19T01:05:33.513 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.513 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.513 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.513 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:33.513 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.657669+0000", 2024-04-19T01:05:33.513 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.564e-06 2024-04-19T01:05:33.514 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.514 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.514 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:33.514 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658422+0000", 2024-04-19T01:05:33.514 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00075376200000000005 2024-04-19T01:05:33.514 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.514 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.514 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:33.514 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658456+0000", 2024-04-19T01:05:33.514 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4059e-05 2024-04-19T01:05:33.514 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.514 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.514 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:33.514 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658479+0000", 2024-04-19T01:05:33.515 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2235e-05 2024-04-19T01:05:33.515 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.515 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.515 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:33.515 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658610+0000", 2024-04-19T01:05:33.515 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013109600000000001 2024-04-19T01:05:33.515 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.515 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.515 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:33.515 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.658624+0000", 2024-04-19T01:05:33.515 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4756000000000001e-05 2024-04-19T01:05:33.515 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.515 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.515 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:33.515 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.699665+0000", 2024-04-19T01:05:33.516 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.041040764 2024-04-19T01:05:33.516 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.516 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.516 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:33.516 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.700238+0000", 2024-04-19T01:05:33.516 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00057254500000000004 2024-04-19T01:05:33.516 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.516 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:33.516 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.516 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.516 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.516 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 79/31 ECSubWrite(tid=116, reqid=client.4604.0:592, at_version=79'91, trim_to=0'0, roll_forward_to=79'88))", 2024-04-19T01:05:33.516 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:49.665950+0000", 2024-04-19T01:05:33.516 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 43.783196988999997, 2024-04-19T01:05:33.516 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.037031065000000002, 2024-04-19T01:05:33.517 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:33.517 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:33.517 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:33.517 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:33.517 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.517 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:33.517 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.665950+0000", 2024-04-19T01:05:33.517 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.517 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.517 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.518 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:33.518 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.665950+0000", 2024-04-19T01:05:33.518 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.518 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.518 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.518 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:33.518 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.665952+0000", 2024-04-19T01:05:33.518 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0360000000000001e-06 2024-04-19T01:05:33.518 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.518 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.518 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:33.518 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.666593+0000", 2024-04-19T01:05:33.518 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00064100300000000002 2024-04-19T01:05:33.518 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.518 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.519 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:33.519 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.666640+0000", 2024-04-19T01:05:33.519 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7817e-05 2024-04-19T01:05:33.519 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.519 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.519 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:33.519 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.666676+0000", 2024-04-19T01:05:33.519 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5222000000000003e-05 2024-04-19T01:05:33.519 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.519 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.519 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:33.519 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.676934+0000", 2024-04-19T01:05:33.519 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010258801999999999 2024-04-19T01:05:33.520 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.520 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.520 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:33.520 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.676964+0000", 2024-04-19T01:05:33.520 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9108999999999999e-05 2024-04-19T01:05:33.520 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.520 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.520 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:33.520 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.702830+0000", 2024-04-19T01:05:33.520 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025866865999999999 2024-04-19T01:05:33.520 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.520 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.520 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:33.520 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.702981+0000", 2024-04-19T01:05:33.520 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015020999999999999 2024-04-19T01:05:33.521 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.521 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:33.521 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.521 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.521 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.521 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 79/31 ECSubWrite(tid=117, reqid=client.4604.0:593, at_version=79'92, trim_to=0'0, roll_forward_to=79'88))", 2024-04-19T01:05:33.521 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:49.674833+0000", 2024-04-19T01:05:33.521 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 43.774313470000003, 2024-04-19T01:05:33.521 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.028247395000000002, 2024-04-19T01:05:33.521 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:33.521 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:33.521 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:33.521 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:33.521 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.521 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:33.522 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.674833+0000", 2024-04-19T01:05:33.522 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.522 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.522 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.522 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:33.522 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.674833+0000", 2024-04-19T01:05:33.522 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.522 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.522 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.522 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:33.522 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.674834+0000", 2024-04-19T01:05:33.522 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3680000000000001e-06 2024-04-19T01:05:33.522 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.522 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.523 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:33.523 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.675074+0000", 2024-04-19T01:05:33.523 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00023945699999999999 2024-04-19T01:05:33.523 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.523 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.523 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:33.523 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.675108+0000", 2024-04-19T01:05:33.523 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4424000000000001e-05 2024-04-19T01:05:33.523 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.523 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.523 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:33.523 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.675127+0000", 2024-04-19T01:05:33.523 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9038999999999998e-05 2024-04-19T01:05:33.523 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.524 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.524 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:33.524 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.690509+0000", 2024-04-19T01:05:33.524 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015381234000000001 2024-04-19T01:05:33.524 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.524 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.524 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:33.524 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.690527+0000", 2024-04-19T01:05:33.524 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8155000000000002e-05 2024-04-19T01:05:33.524 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.524 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.524 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:33.524 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.702999+0000", 2024-04-19T01:05:33.524 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.01247173 2024-04-19T01:05:33.524 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.525 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.525 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:33.525 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.703081+0000", 2024-04-19T01:05:33.525 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1988000000000002e-05 2024-04-19T01:05:33.525 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.525 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:33.525 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.525 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.525 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.525 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 79/31 ECSubWrite(tid=118, reqid=client.4604.0:594, at_version=79'93, trim_to=0'0, roll_forward_to=79'88))", 2024-04-19T01:05:33.525 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:49.679572+0000", 2024-04-19T01:05:33.525 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 43.769574431000002, 2024-04-19T01:05:33.525 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024026347999999999, 2024-04-19T01:05:33.525 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:33.525 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:33.526 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:33.526 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:33.526 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.526 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:33.526 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.679572+0000", 2024-04-19T01:05:33.526 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.526 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.526 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.526 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:33.526 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.679572+0000", 2024-04-19T01:05:33.526 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.526 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.526 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.526 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:33.527 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.679574+0000", 2024-04-19T01:05:33.527 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9700000000000002e-06 2024-04-19T01:05:33.527 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.527 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.527 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:33.527 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.679637+0000", 2024-04-19T01:05:33.527 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2646999999999998e-05 2024-04-19T01:05:33.527 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.527 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.527 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:33.527 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.679677+0000", 2024-04-19T01:05:33.528 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0540000000000001e-05 2024-04-19T01:05:33.528 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.528 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.528 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:33.528 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.679697+0000", 2024-04-19T01:05:33.528 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0108000000000001e-05 2024-04-19T01:05:33.528 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.528 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.528 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:33.528 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.700314+0000", 2024-04-19T01:05:33.528 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020616520999999999 2024-04-19T01:05:33.528 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.528 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.528 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:33.528 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.700332+0000", 2024-04-19T01:05:33.529 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7665000000000001e-05 2024-04-19T01:05:33.529 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.529 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.529 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:33.529 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.703547+0000", 2024-04-19T01:05:33.529 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0032152729999999998 2024-04-19T01:05:33.529 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.529 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.529 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:33.529 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:49.703598+0000", 2024-04-19T01:05:33.529 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1623999999999999e-05 2024-04-19T01:05:33.529 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.529 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:33.529 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.529 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.530 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.530 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.es4 86/31 ECSubWrite(tid=195, reqid=client.4604.0:653, at_version=86'39, trim_to=0'0, roll_forward_to=84'38))", 2024-04-19T01:05:33.530 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:04:56.572490+0000", 2024-04-19T01:05:33.530 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 36.876656304999997, 2024-04-19T01:05:33.530 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022945307000000002, 2024-04-19T01:05:33.530 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:33.530 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:33.530 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:33.530 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:33.530 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.530 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:33.530 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.572490+0000", 2024-04-19T01:05:33.530 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.530 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.531 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.531 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:33.531 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.572490+0000", 2024-04-19T01:05:33.531 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.531 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.531 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.531 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:33.531 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.572491+0000", 2024-04-19T01:05:33.531 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1800000000000005e-07 2024-04-19T01:05:33.531 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.531 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.531 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:33.531 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.574244+0000", 2024-04-19T01:05:33.531 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017527179999999999 2024-04-19T01:05:33.531 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.532 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.532 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:33.532 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.574268+0000", 2024-04-19T01:05:33.532 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3689999999999998e-05 2024-04-19T01:05:33.532 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.532 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.532 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:33.532 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.574282+0000", 2024-04-19T01:05:33.532 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4011e-05 2024-04-19T01:05:33.532 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.532 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.532 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:33.532 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.574403+0000", 2024-04-19T01:05:33.533 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012181 2024-04-19T01:05:33.533 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.534 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.534 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:33.534 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.574420+0000", 2024-04-19T01:05:33.534 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6390000000000001e-05 2024-04-19T01:05:33.534 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.534 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.534 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:33.534 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.595364+0000", 2024-04-19T01:05:33.534 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020944312999999999 2024-04-19T01:05:33.534 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.534 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.534 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:33.534 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:04:56.595436+0000", 2024-04-19T01:05:33.535 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1557000000000006e-05 2024-04-19T01:05:33.535 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.535 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:33.535 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.535 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.535 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.535 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds5 102/98 ECSubWrite(tid=209, reqid=client.4604.0:759, at_version=102'79, trim_to=0'0, roll_forward_to=0'0))", 2024-04-19T01:05:33.535 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:05:12.701960+0000", 2024-04-19T01:05:33.535 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.747186872, 2024-04-19T01:05:33.535 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.039216858, 2024-04-19T01:05:33.535 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:33.535 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:33.535 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:33.535 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:33.535 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.536 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:33.536 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.701960+0000", 2024-04-19T01:05:33.536 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.536 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.536 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.536 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:33.536 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.701960+0000", 2024-04-19T01:05:33.536 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.536 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.536 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.536 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:33.536 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.701962+0000", 2024-04-19T01:05:33.536 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.103e-06 2024-04-19T01:05:33.536 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.537 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.537 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:33.537 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.702295+0000", 2024-04-19T01:05:33.537 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00033357999999999999 2024-04-19T01:05:33.537 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.537 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.537 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:33.537 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.702341+0000", 2024-04-19T01:05:33.537 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5207999999999998e-05 2024-04-19T01:05:33.537 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.537 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.537 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:33.537 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.702367+0000", 2024-04-19T01:05:33.537 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6876000000000002e-05 2024-04-19T01:05:33.537 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.538 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.538 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:33.538 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.707690+0000", 2024-04-19T01:05:33.538 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0053220890000000003 2024-04-19T01:05:33.538 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.538 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.538 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:33.538 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.707719+0000", 2024-04-19T01:05:33.538 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9899e-05 2024-04-19T01:05:33.538 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.538 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.538 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:33.538 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.741009+0000", 2024-04-19T01:05:33.538 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.033289524000000001 2024-04-19T01:05:33.538 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.539 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.539 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:33.539 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.741177+0000", 2024-04-19T01:05:33.539 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000167579 2024-04-19T01:05:33.539 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.539 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:33.539 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.539 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.539 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.539 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds5 102/98 ECSubWrite(tid=210, reqid=client.4604.0:760, at_version=102'80, trim_to=0'0, roll_forward_to=0'0))", 2024-04-19T01:05:33.539 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:05:12.712751+0000", 2024-04-19T01:05:33.539 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.736395480999999, 2024-04-19T01:05:33.539 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.028519213000000002, 2024-04-19T01:05:33.539 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:33.539 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:33.540 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:33.540 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:33.540 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.540 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:33.540 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.712751+0000", 2024-04-19T01:05:33.540 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.540 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.540 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.540 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:33.540 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.712751+0000", 2024-04-19T01:05:33.540 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.540 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.540 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.540 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:33.541 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.712753+0000", 2024-04-19T01:05:33.541 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0099999999999998e-06 2024-04-19T01:05:33.541 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.541 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.541 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:33.541 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.713133+0000", 2024-04-19T01:05:33.541 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000380274 2024-04-19T01:05:33.541 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.541 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.541 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:33.541 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.713172+0000", 2024-04-19T01:05:33.541 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9072000000000001e-05 2024-04-19T01:05:33.541 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.541 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.541 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:33.541 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.713196+0000", 2024-04-19T01:05:33.542 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3929000000000001e-05 2024-04-19T01:05:33.542 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.542 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.542 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:33.542 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.717092+0000", 2024-04-19T01:05:33.542 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003896039 2024-04-19T01:05:33.542 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.542 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.542 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:33.542 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.717108+0000", 2024-04-19T01:05:33.542 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5990000000000001e-05 2024-04-19T01:05:33.542 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.542 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.542 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:33.543 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.741186+0000", 2024-04-19T01:05:33.543 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024077352 2024-04-19T01:05:33.543 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.543 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.543 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:33.543 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.741270+0000", 2024-04-19T01:05:33.543 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4547000000000002e-05 2024-04-19T01:05:33.543 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.543 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:33.543 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.543 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.543 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.543 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.ds5 102/98 ECSubWrite(tid=211, reqid=client.4604.0:761, at_version=102'81, trim_to=0'0, roll_forward_to=0'0))", 2024-04-19T01:05:33.543 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:05:12.717642+0000", 2024-04-19T01:05:33.543 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.731504498, 2024-04-19T01:05:33.544 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023817917000000001, 2024-04-19T01:05:33.544 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:33.544 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:33.544 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:33.544 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:33.544 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.544 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:33.544 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.717642+0000", 2024-04-19T01:05:33.544 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.544 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.544 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.544 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:33.544 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.717642+0000", 2024-04-19T01:05:33.544 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.544 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.545 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.545 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:33.545 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.717644+0000", 2024-04-19T01:05:33.545 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5230000000000001e-06 2024-04-19T01:05:33.545 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.545 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.545 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:33.545 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.717868+0000", 2024-04-19T01:05:33.545 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00022392899999999999 2024-04-19T01:05:33.545 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.545 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.545 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:33.545 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.717911+0000", 2024-04-19T01:05:33.545 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3256000000000001e-05 2024-04-19T01:05:33.545 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.546 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.546 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:33.546 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.717926+0000", 2024-04-19T01:05:33.546 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5642e-05 2024-04-19T01:05:33.546 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.546 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.546 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:33.546 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.731101+0000", 2024-04-19T01:05:33.546 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013174079999999999 2024-04-19T01:05:33.546 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.546 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.546 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:33.546 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.731126+0000", 2024-04-19T01:05:33.546 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5986000000000002e-05 2024-04-19T01:05:33.546 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.547 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.547 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:33.547 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.741364+0000", 2024-04-19T01:05:33.547 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010237227999999999 2024-04-19T01:05:33.547 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.547 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.547 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:33.547 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:12.741460+0000", 2024-04-19T01:05:33.547 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6273000000000005e-05 2024-04-19T01:05:33.547 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.547 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:33.547 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.547 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.548 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.548 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.5s5 107/31 ECSubWrite(tid=214, reqid=client.4604.0:812, at_version=107'201, trim_to=0'0, roll_forward_to=106'199))", 2024-04-19T01:05:33.548 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:05:17.601383+0000", 2024-04-19T01:05:33.548 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.847763827, 2024-04-19T01:05:33.548 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024521777000000002, 2024-04-19T01:05:33.548 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:33.548 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:33.548 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:33.548 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:33.548 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.548 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:33.548 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:17.601383+0000", 2024-04-19T01:05:33.548 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.548 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.548 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.549 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:33.549 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:17.601383+0000", 2024-04-19T01:05:33.549 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.549 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.549 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.549 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:33.549 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:17.601384+0000", 2024-04-19T01:05:33.549 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4699999999999999e-06 2024-04-19T01:05:33.549 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.549 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.549 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:33.549 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:17.601600+0000", 2024-04-19T01:05:33.549 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021619599999999999 2024-04-19T01:05:33.549 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.549 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.550 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:33.550 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:17.601620+0000", 2024-04-19T01:05:33.550 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9701999999999999e-05 2024-04-19T01:05:33.550 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.550 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.550 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:33.550 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:17.601638+0000", 2024-04-19T01:05:33.550 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8142e-05 2024-04-19T01:05:33.550 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.550 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.550 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:33.550 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:17.602043+0000", 2024-04-19T01:05:33.550 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00040467200000000002 2024-04-19T01:05:33.550 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.550 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.551 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-04-19T01:05:33.551 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:17.602056+0000", 2024-04-19T01:05:33.551 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2856e-05 2024-04-19T01:05:33.551 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.551 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.551 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-04-19T01:05:33.551 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:17.625732+0000", 2024-04-19T01:05:33.552 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023676363999999998 2024-04-19T01:05:33.552 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.552 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.552 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-04-19T01:05:33.552 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:17.625905+0000", 2024-04-19T01:05:33.552 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000172375 2024-04-19T01:05:33.552 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.552 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:33.552 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.552 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.552 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:33.552 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:05:33.553 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-19T01:05:33.555 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-19T01:05:33.555 INFO:tasks.ceph.osd.14.smithi118.stderr:2024-04-19T01:05:33.550+0000 7f283c47c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25026) UID: 0 2024-04-19T01:05:33.655 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:05:33.654+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:05:33.667 INFO:tasks.rados.rados.0.smithi155.stdout:393: done (1 left) 2024-04-19T01:05:33.667 INFO:tasks.rados.rados.0.smithi155.stdout:394: read oid 6 snap -1 2024-04-19T01:05:33.667 INFO:tasks.rados.rados.0.smithi155.stdout:394: expect (ObjNum 143 snap 42 seq_num 143) 2024-04-19T01:05:33.668 INFO:tasks.rados.rados.0.smithi155.stdout:395: read oid 17 snap -1 2024-04-19T01:05:33.668 INFO:tasks.rados.rados.0.smithi155.stdout:395: expect (ObjNum 130 snap 37 seq_num 130) 2024-04-19T01:05:33.668 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 29 current snap is 46 2024-04-19T01:05:33.668 INFO:tasks.rados.rados.0.smithi155.stdout:392: finishing write tid 1 to smithi15528762-45 2024-04-19T01:05:33.668 INFO:tasks.rados.rados.0.smithi155.stdout:392: finishing write tid 2 to smithi15528762-45 2024-04-19T01:05:33.668 INFO:tasks.rados.rados.0.smithi155.stdout:392: finishing write tid 3 to smithi15528762-45 2024-04-19T01:05:33.668 INFO:tasks.rados.rados.0.smithi155.stdout:392: oid 45 updating version 0 to 159 2024-04-19T01:05:33.668 INFO:tasks.rados.rados.0.smithi155.stdout:392: oid 45 version 159 is already newer than 158 2024-04-19T01:05:33.668 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 45 v 159 (ObjNum 154 snap 46 seq_num 154) dirty exists 2024-04-19T01:05:33.668 INFO:tasks.rados.rados.0.smithi155.stdout:392: left oid 45 (ObjNum 154 snap 46 seq_num 154) 2024-04-19T01:05:33.668 INFO:tasks.rados.rados.0.smithi155.stdout:396: seq_num 155 ranges {2654208=212992} 2024-04-19T01:05:33.715 INFO:tasks.rados.rados.0.smithi155.stdout:396: writing smithi15528762-29 from 2654208 to 2867200 tid 1 2024-04-19T01:05:33.715 INFO:tasks.rados.rados.0.smithi155.stdout:392: done (3 left) 2024-04-19T01:05:33.715 INFO:tasks.rados.rados.0.smithi155.stdout:397: read oid 47 snap -1 2024-04-19T01:05:33.715 INFO:tasks.rados.rados.0.smithi155.stdout:397: expect (ObjNum 132 snap 38 seq_num 132) 2024-04-19T01:05:33.715 INFO:tasks.rados.rados.0.smithi155.stdout:398: snap_create 2024-04-19T01:05:33.715 INFO:tasks.rados.rados.0.smithi155.stdout:395: expect (ObjNum 130 snap 37 seq_num 130) 2024-04-19T01:05:33.730 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-04-19T01:05:33.730 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T01:05:33.749 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-04-19T01:05:33.750 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-04-19T01:05:33.750 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.750 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDScrubReserve(3.20s6,RELEASE,e:123,from:1(0),reservation_nonce:0)", 2024-04-19T01:05:33.750 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-04-19T01:05:33.751525+0000", 2024-04-19T01:05:33.750 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 7.3102999999999994e-05, 2024-04-19T01:05:33.750 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000112098, 2024-04-19T01:05:33.750 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-04-19T01:05:33.750 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-04-19T01:05:33.750 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-04-19T01:05:33.750 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-04-19T01:05:33.750 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.751 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-04-19T01:05:33.751 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:33.751525+0000", 2024-04-19T01:05:33.751 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.751 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.751 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.751 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-04-19T01:05:33.751 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:33.751525+0000", 2024-04-19T01:05:33.751 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-04-19T01:05:33.751 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.751 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.751 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-04-19T01:05:33.751 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:33.751527+0000", 2024-04-19T01:05:33.751 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7999999999999999e-06 2024-04-19T01:05:33.751 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.752 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.752 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-04-19T01:05:33.752 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:33.751535+0000", 2024-04-19T01:05:33.752 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4200000000000007e-06 2024-04-19T01:05:33.752 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.752 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.752 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-04-19T01:05:33.752 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:33.751560+0000", 2024-04-19T01:05:33.752 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4640000000000001e-05 2024-04-19T01:05:33.752 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.752 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.752 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-04-19T01:05:33.752 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:33.751575+0000", 2024-04-19T01:05:33.753 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4956e-05 2024-04-19T01:05:33.753 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-04-19T01:05:33.753 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-04-19T01:05:33.753 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-04-19T01:05:33.753 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-04-19T01:05:33.751646+0000", 2024-04-19T01:05:33.753 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1167000000000005e-05 2024-04-19T01:05:33.753 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.753 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-04-19T01:05:33.753 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.753 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-04-19T01:05:33.753 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-04-19T01:05:33.753 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 1 2024-04-19T01:05:33.753 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-04-19T01:05:33.754 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-04-19T01:05:33.754 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T01:05:33.755 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:05:33.754+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:05:33.764 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-19T01:05:33.771 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-04-19T01:05:33.771 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T01:05:33.790 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-04-19T01:05:33.790 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T01:05:33.808 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-04-19T01:05:33.808 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T01:05:33.830 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-04-19T01:05:33.830 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T01:05:33.848 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-04-19T01:05:33.848 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T01:05:33.852 INFO:tasks.rados.rados.0.smithi155.stdout:394: expect (ObjNum 143 snap 42 seq_num 143) 2024-04-19T01:05:33.856 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:05:33.856+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:05:33.866 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-04-19T01:05:33.866 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T01:05:33.884 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2024-04-19T01:05:33.885 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T01:05:33.904 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2024-04-19T01:05:33.906 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T01:05:33.923 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2024-04-19T01:05:33.924 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T01:05:33.936 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:33.936 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:33.945 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2024-04-19T01:05:33.945 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T01:05:33.947 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-19T01:05:33.959 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:05:33.956+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:05:33.963 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2024-04-19T01:05:33.964 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-19T01:05:33.986 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2024-04-19T01:05:33.986 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-19T01:05:34.005 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2024-04-19T01:05:34.006 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-19T01:05:34.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:05:34.024+0000 7f70577fe640 0 -- 172.21.15.33:0/201985671 >> [v2:172.21.15.155:6807/741132602,v1:172.21.15.155:6810/741132602] conn(0x7f7058226110 msgr2=0x7f7058228500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-19T01:05:34.033 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2024-04-19T01:05:34.060 INFO:tasks.ceph.osd.11.smithi155.stderr:2024-04-19T01:05:34.060+0000 7f4f7c689640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25351) UID: 0 2024-04-19T01:05:34.068 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:34.068 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:34.082 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-19T01:05:34.161 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:05:34.160+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:05:34.175 INFO:tasks.rados.rados.0.smithi155.stdout:396: finishing write tid 1 to smithi15528762-29 2024-04-19T01:05:34.175 INFO:tasks.rados.rados.0.smithi155.stdout:396: finishing write tid 2 to smithi15528762-29 2024-04-19T01:05:34.175 INFO:tasks.rados.rados.0.smithi155.stdout:396: finishing write tid 3 to smithi15528762-29 2024-04-19T01:05:34.175 INFO:tasks.rados.rados.0.smithi155.stdout:396: oid 29 updating version 0 to 106 2024-04-19T01:05:34.175 INFO:tasks.rados.rados.0.smithi155.stdout:396: oid 29 updating version 106 to 107 2024-04-19T01:05:34.175 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 29 v 107 (ObjNum 155 snap 46 seq_num 155) dirty exists 2024-04-19T01:05:34.176 INFO:tasks.rados.rados.0.smithi155.stdout:396: left oid 29 (ObjNum 155 snap 46 seq_num 155) 2024-04-19T01:05:34.176 INFO:tasks.rados.rados.0.smithi155.stdout:397: expect (ObjNum 132 snap 38 seq_num 132) 2024-04-19T01:05:34.206 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:34.207 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:34.221 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-19T01:05:34.262 INFO:tasks.ceph.osd.12.smithi033.stderr:2024-04-19T01:05:34.260+0000 7fbc98e2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25031) UID: 0 2024-04-19T01:05:34.344 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:34.345 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:34.359 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-19T01:05:34.362 INFO:tasks.ceph.osd.13.smithi110.stderr:2024-04-19T01:05:34.360+0000 7f4acc18d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 25045) UID: 0 2024-04-19T01:05:34.464 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:05:34.462+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:05:34.481 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:34.482 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:34.495 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-04-19T01:05:34.565 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:05:34.564+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:05:34.644 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [0] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-04-19T01:05:34.644 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-19T01:05:34.644 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-19T01:05:34.644 INFO:tasks.thrashosds.thrasher:Setting osd 14 primary_affinity to 0.593582 2024-04-19T01:05:34.644 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 14 0.5935818713886365 2024-04-19T01:05:34.666 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:05:34.664+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:05:34.667 INFO:tasks.rados.rados.0.smithi155.stdout:394: done (4 left) 2024-04-19T01:05:34.668 INFO:tasks.rados.rados.0.smithi155.stdout:395: done (3 left) 2024-04-19T01:05:34.668 INFO:tasks.rados.rados.0.smithi155.stdout:396: done (2 left) 2024-04-19T01:05:34.668 INFO:tasks.rados.rados.0.smithi155.stdout:397: done (1 left) 2024-04-19T01:05:34.668 INFO:tasks.rados.rados.0.smithi155.stdout:398: done (0 left) 2024-04-19T01:05:34.668 INFO:tasks.rados.rados.0.smithi155.stdout:append oid 29 current snap is 47 2024-04-19T01:05:34.668 INFO:tasks.rados.rados.0.smithi155.stdout:399: seq_num 156 ranges {2867200=229376} 2024-04-19T01:05:34.669 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:34.670 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:34.678 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-19T01:05:34.688 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-04-19T01:05:34.709 INFO:tasks.rados.rados.0.smithi155.stdout:399: writing smithi15528762-29 from 2867200 to 3096576 tid 1 2024-04-19T01:05:34.709 INFO:tasks.rados.rados.0.smithi155.stdout: waiting on 1 2024-04-19T01:05:34.718 INFO:tasks.rados.rados.0.smithi155.stdout:399: finishing write tid 1 to smithi15528762-29 2024-04-19T01:05:34.719 INFO:tasks.rados.rados.0.smithi155.stdout:399: finishing write tid 2 to smithi15528762-29 2024-04-19T01:05:34.721 INFO:tasks.rados.rados.0.smithi155.stdout:399: finishing write tid 3 to smithi15528762-29 2024-04-19T01:05:34.721 INFO:tasks.rados.rados.0.smithi155.stdout:399: oid 29 updating version 0 to 110 2024-04-19T01:05:34.721 INFO:tasks.rados.rados.0.smithi155.stdout:399: oid 29 version 110 is already newer than 109 2024-04-19T01:05:34.721 INFO:tasks.rados.rados.0.smithi155.stdout:update_object_version oid 29 v 110 (ObjNum 156 snap 47 seq_num 156) dirty exists 2024-04-19T01:05:34.721 INFO:tasks.rados.rados.0.smithi155.stdout:399: left oid 29 (ObjNum 156 snap 47 seq_num 156) 2024-04-19T01:05:34.722 INFO:tasks.rados.rados.0.smithi155.stdout:399: done (0 left) 2024-04-19T01:05:34.735 INFO:tasks.rados.rados.0.smithi155.stderr:0 errors. 2024-04-19T01:05:34.735 INFO:tasks.rados.rados.0.smithi155.stderr: 2024-04-19T01:05:34.742 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-19T01:05:34.767 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:05:34.766+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:05:34.814 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:34.814 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:34.827 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-04-19T01:05:34.871 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:05:34.870+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:05:34.970 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:05:34.968+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:05:34.995 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:34.996 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:35.009 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-19T01:05:35.053 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:05:35.054 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":124,"fsid":"f11bd2d5-4456-4ffa-83c0-722dddd22ad3","created":"2024-04-19T01:03:36.766089+0000","modified":"2024-04-19T01:05:34.658457+0000","last_up_change":"2024-04-19T01:03:50.591452+0000","last_in_change":"2024-04-19T01:05:05.609495+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-19T01:03:49.571609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-19T01:03:51.913782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.75,"raw_score_acting":4.5,"raw_score_stable":4.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2024-04-19T01:04:00.723622+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":7,"min_size":5,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":36,"pg_placement_num":17,"pg_placement_num_target":36,"pg_num_target":36,"pg_num_pending":36,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"121","auid":0,"snap_mode":"selfmanaged","snap_seq":48,"snap_epoch":124,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"shecprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":16384,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"296899fd-2b2b-4071-8b1f-0f92c3d0bbc0","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":1633616806},{"type":"v1","addr":"172.21.15.33:6825","nonce":1633616806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":1633616806},{"type":"v1","addr":"172.21.15.33:6827","nonce":1633616806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":1633616806},{"type":"v1","addr":"172.21.15.33:6831","nonce":1633616806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":1633616806},{"type":"v1","addr":"172.21.15.33:6829","nonce":1633616806}]},"public_addr":"172.21.15.33:6825/1633616806","cluster_addr":"172.21.15.33:6827/1633616806","heartbeat_back_addr":"172.21.15.33:6831/1633616806","heartbeat_front_addr":"172.21.15.33:6829/1633616806","state":["exists","up"]},{"osd":1,"uuid":"021f7190-0ec3-4553-a44f-15095765cccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":4270148213},{"type":"v1","addr":"172.21.15.110:6808","nonce":4270148213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6811","nonce":4270148213},{"type":"v1","addr":"172.21.15.110:6814","nonce":4270148213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":4270148213},{"type":"v1","addr":"172.21.15.110:6823","nonce":4270148213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6817","nonce":4270148213},{"type":"v1","addr":"172.21.15.110:6820","nonce":4270148213}]},"public_addr":"172.21.15.110:6808/4270148213","cluster_addr":"172.21.15.110:6814/4270148213","heartbeat_back_addr":"172.21.15.110:6823/4270148213","heartbeat_front_addr":"172.21.15.110:6820/4270148213","state":["exists","up"]},{"osd":2,"uuid":"c283d0bc-5e34-4763-9019-328ee9b6c07a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":13248188},{"type":"v1","addr":"172.21.15.118:6809","nonce":13248188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":13248188},{"type":"v1","addr":"172.21.15.118:6811","nonce":13248188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":13248188},{"type":"v1","addr":"172.21.15.118:6815","nonce":13248188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":13248188},{"type":"v1","addr":"172.21.15.118:6813","nonce":13248188}]},"public_addr":"172.21.15.118:6809/13248188","cluster_addr":"172.21.15.118:6811/13248188","heartbeat_back_addr":"172.21.15.118:6815/13248188","heartbeat_front_addr":"172.21.15.118:6813/13248188","state":["exists","up"]},{"osd":3,"uuid":"d45888f2-eecd-480b-80f5-c0731a06ff3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":2658385904},{"type":"v1","addr":"172.21.15.155:6803","nonce":2658385904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":2658385904},{"type":"v1","addr":"172.21.15.155:6805","nonce":2658385904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6813","nonce":2658385904},{"type":"v1","addr":"172.21.15.155:6816","nonce":2658385904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":2658385904},{"type":"v1","addr":"172.21.15.155:6809","nonce":2658385904}]},"public_addr":"172.21.15.155:6803/2658385904","cluster_addr":"172.21.15.155:6805/2658385904","heartbeat_back_addr":"172.21.15.155:6816/2658385904","heartbeat_front_addr":"172.21.15.155:6809/2658385904","state":["exists","up"]},{"osd":4,"uuid":"d060d939-8c94-4a07-9fd5-42af0044a16f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":2389566155},{"type":"v1","addr":"172.21.15.33:6801","nonce":2389566155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":2389566155},{"type":"v1","addr":"172.21.15.33:6803","nonce":2389566155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":2389566155},{"type":"v1","addr":"172.21.15.33:6807","nonce":2389566155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":2389566155},{"type":"v1","addr":"172.21.15.33:6805","nonce":2389566155}]},"public_addr":"172.21.15.33:6801/2389566155","cluster_addr":"172.21.15.33:6803/2389566155","heartbeat_back_addr":"172.21.15.33:6807/2389566155","heartbeat_front_addr":"172.21.15.33:6805/2389566155","state":["exists","up"]},{"osd":5,"uuid":"9700d2f7-ed56-4143-ba9d-6cebe524e877","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":2821892844},{"type":"v1","addr":"172.21.15.110:6825","nonce":2821892844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":2821892844},{"type":"v1","addr":"172.21.15.110:6827","nonce":2821892844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":2821892844},{"type":"v1","addr":"172.21.15.110:6831","nonce":2821892844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":2821892844},{"type":"v1","addr":"172.21.15.110:6829","nonce":2821892844}]},"public_addr":"172.21.15.110:6825/2821892844","cluster_addr":"172.21.15.110:6827/2821892844","heartbeat_back_addr":"172.21.15.110:6831/2821892844","heartbeat_front_addr":"172.21.15.110:6829/2821892844","state":["exists","up"]},{"osd":6,"uuid":"49324695-d83b-41ad-92e1-5348e8e9149f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":273776428},{"type":"v1","addr":"172.21.15.118:6818","nonce":273776428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":273776428},{"type":"v1","addr":"172.21.15.118:6822","nonce":273776428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":273776428},{"type":"v1","addr":"172.21.15.118:6830","nonce":273776428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":273776428},{"type":"v1","addr":"172.21.15.118:6826","nonce":273776428}]},"public_addr":"172.21.15.118:6818/273776428","cluster_addr":"172.21.15.118:6822/273776428","heartbeat_back_addr":"172.21.15.118:6830/273776428","heartbeat_front_addr":"172.21.15.118:6826/273776428","state":["exists","up"]},{"osd":7,"uuid":"afef160d-0911-4406-ad49-cd6d61ee8a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1006872173},{"type":"v1","addr":"172.21.15.155:6827","nonce":1006872173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1006872173},{"type":"v1","addr":"172.21.15.155:6829","nonce":1006872173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1006872173},{"type":"v1","addr":"172.21.15.155:6833","nonce":1006872173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1006872173},{"type":"v1","addr":"172.21.15.155:6831","nonce":1006872173}]},"public_addr":"172.21.15.155:6827/1006872173","cluster_addr":"172.21.15.155:6829/1006872173","heartbeat_back_addr":"172.21.15.155:6833/1006872173","heartbeat_front_addr":"172.21.15.155:6831/1006872173","state":["exists","up"]},{"osd":8,"uuid":"298396bf-31a2-4832-91a4-55fcb050130f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":744651323},{"type":"v1","addr":"172.21.15.33:6817","nonce":744651323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":744651323},{"type":"v1","addr":"172.21.15.33:6819","nonce":744651323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":744651323},{"type":"v1","addr":"172.21.15.33:6823","nonce":744651323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":744651323},{"type":"v1","addr":"172.21.15.33:6821","nonce":744651323}]},"public_addr":"172.21.15.33:6817/744651323","cluster_addr":"172.21.15.33:6819/744651323","heartbeat_back_addr":"172.21.15.33:6823/744651323","heartbeat_front_addr":"172.21.15.33:6821/744651323","state":["exists","up"]},{"osd":9,"uuid":"c151ae6b-279d-4174-aceb-4671fbdac3b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":872519654},{"type":"v1","addr":"172.21.15.110:6802","nonce":872519654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":872519654},{"type":"v1","addr":"172.21.15.110:6807","nonce":872519654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":872519654},{"type":"v1","addr":"172.21.15.110:6819","nonce":872519654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":872519654},{"type":"v1","addr":"172.21.15.110:6813","nonce":872519654}]},"public_addr":"172.21.15.110:6802/872519654","cluster_addr":"172.21.15.110:6807/872519654","heartbeat_back_addr":"172.21.15.110:6819/872519654","heartbeat_front_addr":"172.21.15.110:6813/872519654","state":["exists","up"]},{"osd":10,"uuid":"0b292dfb-2029-4e1e-b206-ea7e0f6969ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6817","nonce":4099141917},{"type":"v1","addr":"172.21.15.118:6819","nonce":4099141917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6821","nonce":4099141917},{"type":"v1","addr":"172.21.15.118:6823","nonce":4099141917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6829","nonce":4099141917},{"type":"v1","addr":"172.21.15.118:6831","nonce":4099141917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":4099141917},{"type":"v1","addr":"172.21.15.118:6827","nonce":4099141917}]},"public_addr":"172.21.15.118:6819/4099141917","cluster_addr":"172.21.15.118:6823/4099141917","heartbeat_back_addr":"172.21.15.118:6831/4099141917","heartbeat_front_addr":"172.21.15.118:6827/4099141917","state":["exists","up"]},{"osd":11,"uuid":"10c61619-ad11-40f0-b013-9ef52bcabf59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":3981951559},{"type":"v1","addr":"172.21.15.155:6811","nonce":3981951559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":3981951559},{"type":"v1","addr":"172.21.15.155:6817","nonce":3981951559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":3981951559},{"type":"v1","addr":"172.21.15.155:6825","nonce":3981951559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3981951559},{"type":"v1","addr":"172.21.15.155:6823","nonce":3981951559}]},"public_addr":"172.21.15.155:6811/3981951559","cluster_addr":"172.21.15.155:6817/3981951559","heartbeat_back_addr":"172.21.15.155:6825/3981951559","heartbeat_front_addr":"172.21.15.155:6823/3981951559","state":["exists","up"]},{"osd":12,"uuid":"3f0d26e5-f3e2-47b6-91a8-0daa30d0544f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":4133738119},{"type":"v1","addr":"172.21.15.33:6809","nonce":4133738119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":4133738119},{"type":"v1","addr":"172.21.15.33:6811","nonce":4133738119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":4133738119},{"type":"v1","addr":"172.21.15.33:6815","nonce":4133738119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":4133738119},{"type":"v1","addr":"172.21.15.33:6813","nonce":4133738119}]},"public_addr":"172.21.15.33:6809/4133738119","cluster_addr":"172.21.15.33:6811/4133738119","heartbeat_back_addr":"172.21.15.33:6815/4133738119","heartbeat_front_addr":"172.21.15.33:6813/4133738119","state":["exists","up"]},{"osd":13,"uuid":"c8d6213f-af5e-4646-9241-6f91589b0e1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":1583232449},{"type":"v1","addr":"172.21.15.110:6803","nonce":1583232449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":1583232449},{"type":"v1","addr":"172.21.15.110:6809","nonce":1583232449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":1583232449},{"type":"v1","addr":"172.21.15.110:6821","nonce":1583232449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":1583232449},{"type":"v1","addr":"172.21.15.110:6815","nonce":1583232449}]},"public_addr":"172.21.15.110:6803/1583232449","cluster_addr":"172.21.15.110:6809/1583232449","heartbeat_back_addr":"172.21.15.110:6821/1583232449","heartbeat_front_addr":"172.21.15.110:6815/1583232449","state":["exists","up"]},{"osd":14,"uuid":"7b3d56df-62bf-4877-b984-6f17fe1fab33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":506416463},{"type":"v1","addr":"172.21.15.118:6801","nonce":506416463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":506416463},{"type":"v1","addr":"172.21.15.118:6803","nonce":506416463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":506416463},{"type":"v1","addr":"172.21.15.118:6807","nonce":506416463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":506416463},{"type":"v1","addr":"172.21.15.118:6805","nonce":506416463}]},"public_addr":"172.21.15.118:6801/506416463","cluster_addr":"172.21.15.118:6803/506416463","heartbeat_back_addr":"172.21.15.118:6807/506416463","heartbeat_front_addr":"172.21.15.118:6805/506416463","state":["exists","up"]},{"osd":15,"uuid":"6d526edb-f1d8-415a-ad55-81e810ff4944","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6807","nonce":741132602},{"type":"v1","addr":"172.21.15.155:6810","nonce":741132602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":741132602},{"type":"v1","addr":"172.21.15.155:6815","nonce":741132602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6821","nonce":741132602},{"type":"v1","addr":"172.21.15.155:6822","nonce":741132602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":741132602},{"type":"v1","addr":"172.21.15.155:6819","nonce":741132602}]},"public_addr":"172.21.15.155:6810/741132602","cluster_addr":"172.21.15.155:6815/741132602","heartbeat_back_addr":"172.21.15.155:6822/741132602","heartbeat_front_addr":"172.21.15.155:6819/741132602","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-04-19T01:05:00.394086+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:04:50.943238+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:04:49.660131+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:05:14.387087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:05:07.754528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:05:03.768853+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:04:49.986620+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:05:03.346710+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:04:51.543181+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:05:11.056736+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:05:06.097115+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:05:05.201499+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:04:57.929819+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:04:50.297489+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:05:02.554338+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:05:06.403111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":36,"length":1},{"begin":39,"length":1},{"begin":41,"length":1},{"begin":43,"length":1},{"begin":46,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T01:05:35.071 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:05:35.072+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:05:35.073 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T01:05:35.142 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:35.143 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:35.155 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-19T01:05:35.172 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-04-19T01:05:35.168+0000 7f66a6bb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25024) UID: 0 2024-04-19T01:05:35.274 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:05:35.272+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:05:35.276 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:35.276 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:35.290 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-19T01:05:35.375 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:05:35.372+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:05:35.410 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:05:35.411 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-04-19T01:05:35.420 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:35.420 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:35.429 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":162,"stamp":"2024-04-19T01:05:34.665434+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":112161312,"num_objects":78,"num_object_clones":32,"num_object_copies":536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":4,"num_read":265,"num_read_kb":85919,"num_write":437,"num_write_kb":48908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":6938624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3421,"ondisk_log_size":3421,"up":263,"acting":263,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":235,"num_osds":15,"num_per_pool_osds":15,"num_per_pool_omap_osds":15,"kb":1415577600,"kb_used":703960,"kb_used_data":201464,"kb_used_omap":23,"kb_used_meta":502184,"kb_avail":1414873640,"statfs":{"total":1449551462400,"available":1448830607360,"internally_reserved":0,"allocated":206299136,"data_stored":285090572,"data_compressed":2985727,"data_compressed_allocated":6135808,"data_compressed_original":13164544,"omap_allocated":23925,"internal_metadata":514237067},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8974336,"num_objects":6,"num_object_clones":6,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":-1,"num_read":-282,"num_read_kb":-102419,"num_write":-585,"num_write_kb":-98248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-25,"num_bytes_recovered":-9527296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.036257"},"pg_stats":[{"pgid":"3.23","version":"116'107","reported_seq":628,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-04-19T01:05:33.712654+0000","last_change":"2024-04-19T01:05:33.712654+0000","last_active":"2024-04-19T01:05:33.712654+0000","last_peered":"2024-04-19T01:05:33.712654+0000","last_clean":"2024-04-19T01:05:33.712654+0000","last_became_active":"2024-04-19T01:05:32.689521+0000","last_became_peered":"2024-04-19T01:05:32.689521+0000","last_unstale":"2024-04-19T01:05:33.712654+0000","last_undegraded":"2024-04-19T01:05:33.712654+0000","last_fullsized":"2024-04-19T01:05:33.712654+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":6,"last_scrub":"116'107","last_scrub_stamp":"2024-04-19T01:05:33.413539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:33.413539+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:47.177754+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.010427751000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2,3,1,4,12],"acting":[15,8,2,3,1,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"2e","length":"2"}]},{"pgid":"3.22","version":"116'63","reported_seq":568,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-04-19T01:05:33.783260+0000","last_change":"2024-04-19T01:05:33.783260+0000","last_active":"2024-04-19T01:05:33.783260+0000","last_peered":"2024-04-19T01:05:33.783260+0000","last_clean":"2024-04-19T01:05:33.783260+0000","last_became_active":"2024-04-19T01:05:32.718235+0000","last_became_peered":"2024-04-19T01:05:32.718235+0000","last_unstale":"2024-04-19T01:05:33.783260+0000","last_undegraded":"2024-04-19T01:05:33.783260+0000","last_fullsized":"2024-04-19T01:05:33.783260+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":6,"last_scrub":"116'63","last_scrub_stamp":"2024-04-19T01:05:33.783016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:33.783016+0000","objects_scrubbed":2,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:55.913601+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0060865700000000003,"stat_sum":{"num_bytes":4620288,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,13,4,8,3,7],"acting":[11,5,13,4,8,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"2e","length":"2"}]},{"pgid":"3.21","version":"116'75","reported_seq":546,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-04-19T01:05:33.683963+0000","last_change":"2024-04-19T01:05:33.683963+0000","last_active":"2024-04-19T01:05:33.683963+0000","last_peered":"2024-04-19T01:05:33.683963+0000","last_clean":"2024-04-19T01:05:33.683963+0000","last_became_active":"2024-04-19T01:05:32.709381+0000","last_became_peered":"2024-04-19T01:05:32.709381+0000","last_unstale":"2024-04-19T01:05:33.683963+0000","last_undegraded":"2024-04-19T01:05:33.683963+0000","last_fullsized":"2024-04-19T01:05:33.683963+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":6,"last_scrub":"116'75","last_scrub_stamp":"2024-04-19T01:05:31.634839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:31.634839+0000","objects_scrubbed":2,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0019870930000000001,"stat_sum":{"num_bytes":1597440,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,11,9,1,7],"acting":[6,4,3,11,9,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"2e","length":"2"}]},{"pgid":"3.20","version":"117'128","reported_seq":620,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-04-19T01:05:33.749985+0000","last_change":"2024-04-19T01:05:33.749985+0000","last_active":"2024-04-19T01:05:33.749985+0000","last_peered":"2024-04-19T01:05:33.749985+0000","last_clean":"2024-04-19T01:05:33.749985+0000","last_became_active":"2024-04-19T01:05:32.709950+0000","last_became_peered":"2024-04-19T01:05:32.709950+0000","last_unstale":"2024-04-19T01:05:33.749985+0000","last_undegraded":"2024-04-19T01:05:33.749985+0000","last_fullsized":"2024-04-19T01:05:33.749985+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":6,"last_scrub":"117'128","last_scrub_stamp":"2024-04-19T01:05:33.749610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:33.749610+0000","objects_scrubbed":3,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:42.166711+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.002391392,"stat_sum":{"num_bytes":7864320,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4,3,10,11,6],"acting":[1,9,4,3,10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"2e","length":"2"}]},{"pgid":"3.1f","version":"117'104","reported_seq":548,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-19T01:05:32.719913+0000","last_change":"2024-04-19T01:05:32.719803+0000","last_active":"2024-04-19T01:05:32.719913+0000","last_peered":"2024-04-19T01:05:32.719913+0000","last_clean":"2024-04-19T01:05:32.719913+0000","last_became_active":"2024-04-19T01:05:32.714922+0000","last_became_peered":"2024-04-19T01:05:32.714922+0000","last_unstale":"2024-04-19T01:05:32.719913+0000","last_undegraded":"2024-04-19T01:05:32.719913+0000","last_fullsized":"2024-04-19T01:05:32.719913+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":5,"last_scrub":"106'81","last_scrub_stamp":"2024-04-19T01:05:18.498676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:18.498676+0000","objects_scrubbed":4,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0018122710000000001,"stat_sum":{"num_bytes":6668288,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":2646,"num_write":31,"num_write_kb":6512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,13,5,11,7,1,2],"acting":[10,13,5,11,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"2e","length":"1"}]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-19T01:05:31.627924+0000","last_change":"2024-04-19T01:05:31.627924+0000","last_active":"2024-04-19T01:05:31.627924+0000","last_peered":"2024-04-19T01:05:31.627924+0000","last_clean":"2024-04-19T01:05:31.627924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-19T01:05:31.627924+0000","last_undegraded":"2024-04-19T01:05:31.627924+0000","last_fullsized":"2024-04-19T01:05:31.627924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:31.627924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:31.627924+0000","last_clean_scrub_stamp":"2024-04-19T01:05:31.627924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"123'113","reported_seq":597,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-04-19T01:05:33.727537+0000","last_change":"2024-04-19T01:05:33.727537+0000","last_active":"2024-04-19T01:05:33.727537+0000","last_peered":"2024-04-19T01:05:33.727537+0000","last_clean":"2024-04-19T01:05:33.727537+0000","last_became_active":"2024-04-19T01:05:32.678185+0000","last_became_peered":"2024-04-19T01:05:32.678185+0000","last_unstale":"2024-04-19T01:05:33.727537+0000","last_undegraded":"2024-04-19T01:05:33.727537+0000","last_fullsized":"2024-04-19T01:05:33.727537+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.010794335,"stat_sum":{"num_bytes":4743168,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":2166,"num_write":21,"num_write_kb":5176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":2375680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,9,1,8,2,6],"acting":[12,4,9,1,8,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"2e","length":"2"}]},{"pgid":"3.1c","version":"120'89","reported_seq":546,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-04-19T01:05:33.683398+0000","last_change":"2024-04-19T01:05:33.683398+0000","last_active":"2024-04-19T01:05:33.683398+0000","last_peered":"2024-04-19T01:05:33.683398+0000","last_clean":"2024-04-19T01:05:33.683398+0000","last_became_active":"2024-04-19T01:05:32.715180+0000","last_became_peered":"2024-04-19T01:05:32.715180+0000","last_unstale":"2024-04-19T01:05:33.683398+0000","last_undegraded":"2024-04-19T01:05:33.683398+0000","last_fullsized":"2024-04-19T01:05:33.683398+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060957479999999998,"stat_sum":{"num_bytes":1523712,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":1228,"num_write":24,"num_write_kb":2152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,12,1,9,2,6],"acting":[11,10,12,1,9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"2e","length":"2"}]},{"pgid":"3.1b","version":"105'56","reported_seq":519,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-19T01:05:32.719633+0000","last_change":"2024-04-19T01:05:32.719633+0000","last_active":"2024-04-19T01:05:32.719633+0000","last_peered":"2024-04-19T01:05:32.719633+0000","last_clean":"2024-04-19T01:05:32.719633+0000","last_became_active":"2024-04-19T01:05:32.715027+0000","last_became_peered":"2024-04-19T01:05:32.715027+0000","last_unstale":"2024-04-19T01:05:32.719633+0000","last_undegraded":"2024-04-19T01:05:32.719633+0000","last_fullsized":"2024-04-19T01:05:32.719633+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026713470000000001,"stat_sum":{"num_bytes":2547712,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":4091,"num_write":20,"num_write_kb":2488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11,15,1,4,14],"acting":[10,6,11,15,1,4,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"2e","length":"1"}]},{"pgid":"3.1a","version":"115'114","reported_seq":594,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-04-19T01:05:34.542461+0000","last_change":"2024-04-19T01:05:34.542461+0000","last_active":"2024-04-19T01:05:34.542461+0000","last_peered":"2024-04-19T01:05:34.542461+0000","last_clean":"2024-04-19T01:05:34.542461+0000","last_became_active":"2024-04-19T01:05:32.677602+0000","last_became_peered":"2024-04-19T01:05:32.677602+0000","last_unstale":"2024-04-19T01:05:34.542461+0000","last_undegraded":"2024-04-19T01:05:34.542461+0000","last_fullsized":"2024-04-19T01:05:34.542461+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":5,"last_scrub":"115'114","last_scrub_stamp":"2024-04-19T01:05:34.542157+0000","last_deep_scrub":"115'114","last_deep_scrub_stamp":"2024-04-19T01:05:34.542157+0000","last_clean_scrub_stamp":"2024-04-19T01:05:34.542157+0000","objects_scrubbed":1,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:56.562562+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0038441289999999999,"stat_sum":{"num_bytes":4423680,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,7,3,13,1,8],"acting":[12,4,7,3,13,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"2e","length":"2"}]},{"pgid":"3.19","version":"113'148","reported_seq":624,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-19T01:05:30.759614+0000","last_change":"2024-04-19T01:05:29.742520+0000","last_active":"2024-04-19T01:05:30.759614+0000","last_peered":"2024-04-19T01:05:30.759614+0000","last_clean":"2024-04-19T01:05:30.759614+0000","last_became_active":"2024-04-19T01:05:26.595940+0000","last_became_peered":"2024-04-19T01:05:26.595940+0000","last_unstale":"2024-04-19T01:05:30.759614+0000","last_undegraded":"2024-04-19T01:05:30.759614+0000","last_fullsized":"2024-04-19T01:05:30.759614+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":5,"last_scrub":"113'148","last_scrub_stamp":"2024-04-19T01:05:29.741878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:29.741878+0000","objects_scrubbed":4,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:37.706878+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0016723689999999999,"stat_sum":{"num_bytes":5226496,"num_objects":4,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11,14,9,4,10,6],"acting":[13,11,14,9,4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"}]},{"pgid":"3.18","version":"112'67","reported_seq":579,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-04-19T01:05:33.683635+0000","last_change":"2024-04-19T01:05:33.685665+0000","last_active":"2024-04-19T01:05:33.683635+0000","last_peered":"2024-04-19T01:05:33.683635+0000","last_clean":"2024-04-19T01:05:33.683635+0000","last_became_active":"2024-04-19T01:05:26.599002+0000","last_became_peered":"2024-04-19T01:05:26.599002+0000","last_unstale":"2024-04-19T01:05:33.683635+0000","last_undegraded":"2024-04-19T01:05:33.683635+0000","last_fullsized":"2024-04-19T01:05:33.683635+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":5,"last_scrub":"112'67","last_scrub_stamp":"2024-04-19T01:05:30.743412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:30.743412+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:33.199553+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.0079882210000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,1,15,14,6,5],"acting":[11,10,1,15,14,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"2e","length":"2"}]},{"pgid":"3.17","version":"112'27","reported_seq":509,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-19T01:05:31.276883+0000","last_change":"2024-04-19T01:05:31.276969+0000","last_active":"2024-04-19T01:05:31.276883+0000","last_peered":"2024-04-19T01:05:31.276883+0000","last_clean":"2024-04-19T01:05:31.276883+0000","last_became_active":"2024-04-19T01:05:26.562785+0000","last_became_peered":"2024-04-19T01:05:26.562785+0000","last_unstale":"2024-04-19T01:05:31.276883+0000","last_undegraded":"2024-04-19T01:05:31.276883+0000","last_fullsized":"2024-04-19T01:05:31.276883+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":5,"last_scrub":"112'27","last_scrub_stamp":"2024-04-19T01:05:31.276692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:31.276692+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:34.272525+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.001361117,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,7,5,11,6,9],"acting":[14,13,7,5,11,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"}]},{"pgid":"3.16","version":"0'0","reported_seq":463,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-19T01:05:30.734437+0000","last_change":"2024-04-19T01:05:30.255216+0000","last_active":"2024-04-19T01:05:30.734437+0000","last_peered":"2024-04-19T01:05:30.734437+0000","last_clean":"2024-04-19T01:05:30.734437+0000","last_became_active":"2024-04-19T01:05:26.562883+0000","last_became_peered":"2024-04-19T01:05:26.562883+0000","last_unstale":"2024-04-19T01:05:30.734437+0000","last_undegraded":"2024-04-19T01:05:30.734437+0000","last_fullsized":"2024-04-19T01:05:30.734437+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:30.254960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:30.254960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:33.760753+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.001596641,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7,15,8,11,1,10],"acting":[14,7,15,8,11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"}]},{"pgid":"3.15","version":"113'215","reported_seq":700,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-04-19T01:05:33.673821+0000","last_change":"2024-04-19T01:05:33.677867+0000","last_active":"2024-04-19T01:05:33.673821+0000","last_peered":"2024-04-19T01:05:33.673821+0000","last_clean":"2024-04-19T01:05:33.673821+0000","last_became_active":"2024-04-19T01:05:26.599866+0000","last_became_peered":"2024-04-19T01:05:26.599866+0000","last_unstale":"2024-04-19T01:05:33.673821+0000","last_undegraded":"2024-04-19T01:05:33.673821+0000","last_fullsized":"2024-04-19T01:05:33.673821+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":5,"last_scrub":"113'215","last_scrub_stamp":"2024-04-19T01:05:30.198787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:30.198787+0000","objects_scrubbed":3,"log_size":215,"log_dups_size":0,"ondisk_log_size":215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:31.635117+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0042919209999999998,"stat_sum":{"num_bytes":5488640,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,14,11,10,2,5],"acting":[8,6,14,11,10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"2e","length":"2"}]},{"pgid":"3.14","version":"112'91","reported_seq":568,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-19T01:05:32.653220+0000","last_change":"2024-04-19T01:05:32.655124+0000","last_active":"2024-04-19T01:05:32.653220+0000","last_peered":"2024-04-19T01:05:32.653220+0000","last_clean":"2024-04-19T01:05:32.653220+0000","last_became_active":"2024-04-19T01:05:26.582876+0000","last_became_peered":"2024-04-19T01:05:26.582876+0000","last_unstale":"2024-04-19T01:05:32.653220+0000","last_undegraded":"2024-04-19T01:05:32.653220+0000","last_fullsized":"2024-04-19T01:05:32.653220+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":5,"last_scrub":"112'91","last_scrub_stamp":"2024-04-19T01:05:30.467237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:30.467237+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:47.882466+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0076891720000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,13,14,3,15,1,5],"acting":[10,13,14,3,15,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"2e","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":202,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-04-19T01:05:33.673281+0000","last_change":"2024-04-19T01:03:53.939441+0000","last_active":"2024-04-19T01:05:33.673281+0000","last_peered":"2024-04-19T01:05:33.673281+0000","last_clean":"2024-04-19T01:05:33.673281+0000","last_became_active":"2024-04-19T01:03:53.938774+0000","last_became_peered":"2024-04-19T01:03:53.938774+0000","last_unstale":"2024-04-19T01:05:33.673281+0000","last_undegraded":"2024-04-19T01:05:33.673281+0000","last_fullsized":"2024-04-19T01:05:33.673281+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":455,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-19T01:05:30.734440+0000","last_change":"2024-04-19T01:05:27.636589+0000","last_active":"2024-04-19T01:05:30.734440+0000","last_peered":"2024-04-19T01:05:30.734440+0000","last_clean":"2024-04-19T01:05:30.734440+0000","last_became_active":"2024-04-19T01:05:26.554043+0000","last_became_peered":"2024-04-19T01:05:26.554043+0000","last_unstale":"2024-04-19T01:05:30.734440+0000","last_undegraded":"2024-04-19T01:05:30.734440+0000","last_fullsized":"2024-04-19T01:05:30.734440+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:19.329813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:19.329813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.002806401,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7,15,8,11,1,10],"acting":[14,7,15,8,11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":203,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-19T01:05:30.734752+0000","last_change":"2024-04-19T01:05:11.353464+0000","last_active":"2024-04-19T01:05:30.734752+0000","last_peered":"2024-04-19T01:05:30.734752+0000","last_clean":"2024-04-19T01:05:30.734752+0000","last_became_active":"2024-04-19T01:03:53.735549+0000","last_became_peered":"2024-04-19T01:03:53.735549+0000","last_unstale":"2024-04-19T01:05:30.734752+0000","last_undegraded":"2024-04-19T01:05:30.734752+0000","last_fullsized":"2024-04-19T01:05:30.734752+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:11.353213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:11.353213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:29.090848+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13],"acting":[14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.7","version":"116'29","reported_seq":504,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-19T01:05:30.734491+0000","last_change":"2024-04-19T01:05:27.638083+0000","last_active":"2024-04-19T01:05:30.734491+0000","last_peered":"2024-04-19T01:05:30.734491+0000","last_clean":"2024-04-19T01:05:30.734491+0000","last_became_active":"2024-04-19T01:05:26.561696+0000","last_became_peered":"2024-04-19T01:05:26.561696+0000","last_unstale":"2024-04-19T01:05:30.734491+0000","last_undegraded":"2024-04-19T01:05:30.734491+0000","last_fullsized":"2024-04-19T01:05:30.734491+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'23","last_scrub_stamp":"2024-04-19T01:05:18.361237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:18.361237+0000","objects_scrubbed":2,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0019724629999999998,"stat_sum":{"num_bytes":1540096,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":9026,"num_write":14,"num_write_kb":1504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1540096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,7,5,11,6,9],"acting":[14,13,7,5,11,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":203,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-19T01:05:30.644442+0000","last_change":"2024-04-19T01:05:21.106153+0000","last_active":"2024-04-19T01:05:30.644442+0000","last_peered":"2024-04-19T01:05:30.644442+0000","last_clean":"2024-04-19T01:05:30.644442+0000","last_became_active":"2024-04-19T01:03:53.771887+0000","last_became_peered":"2024-04-19T01:03:53.771887+0000","last_unstale":"2024-04-19T01:05:30.644442+0000","last_undegraded":"2024-04-19T01:05:30.644442+0000","last_fullsized":"2024-04-19T01:05:30.644442+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:21.105806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:21.105806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:40.011866+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.4","version":"116'93","reported_seq":571,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-19T01:05:32.653446+0000","last_change":"2024-04-19T01:05:32.656908+0000","last_active":"2024-04-19T01:05:32.653446+0000","last_peered":"2024-04-19T01:05:32.653446+0000","last_clean":"2024-04-19T01:05:32.653446+0000","last_became_active":"2024-04-19T01:05:26.560752+0000","last_became_peered":"2024-04-19T01:05:26.560752+0000","last_unstale":"2024-04-19T01:05:32.653446+0000","last_undegraded":"2024-04-19T01:05:32.653446+0000","last_fullsized":"2024-04-19T01:05:32.653446+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'93","last_scrub_stamp":"2024-04-19T01:05:31.485556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:31.485556+0000","objects_scrubbed":3,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:31.627833+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0082277610000000001,"stat_sum":{"num_bytes":5357568,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,13,14,3,15,1,5],"acting":[10,13,14,3,15,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"2e","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":209,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-04-19T01:05:33.680070+0000","last_change":"2024-04-19T01:05:10.593158+0000","last_active":"2024-04-19T01:05:33.680070+0000","last_peered":"2024-04-19T01:05:33.680070+0000","last_clean":"2024-04-19T01:05:33.680070+0000","last_became_active":"2024-04-19T01:03:53.716040+0000","last_became_peered":"2024-04-19T01:03:53.716040+0000","last_unstale":"2024-04-19T01:05:33.680070+0000","last_undegraded":"2024-04-19T01:05:33.680070+0000","last_fullsized":"2024-04-19T01:05:33.680070+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:10.592906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:10.592906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:39.747266+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"117'222","reported_seq":708,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-04-19T01:05:33.696645+0000","last_change":"2024-04-19T01:05:33.679662+0000","last_active":"2024-04-19T01:05:33.696645+0000","last_peered":"2024-04-19T01:05:33.696645+0000","last_clean":"2024-04-19T01:05:33.696645+0000","last_became_active":"2024-04-19T01:05:26.537782+0000","last_became_peered":"2024-04-19T01:05:26.537782+0000","last_unstale":"2024-04-19T01:05:33.696645+0000","last_undegraded":"2024-04-19T01:05:33.696645+0000","last_fullsized":"2024-04-19T01:05:33.696645+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'222","last_scrub_stamp":"2024-04-19T01:05:31.150482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:31.150482+0000","objects_scrubbed":7,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:38.793367+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0058953540000000002,"stat_sum":{"num_bytes":6127616,"num_objects":7,"num_object_clones":3,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":8928,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,14,11,10,2,5],"acting":[8,6,14,11,10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"2e","length":"2"}]},{"pgid":"3.3","version":"116'107","reported_seq":629,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-04-19T01:05:34.442370+0000","last_change":"2024-04-19T01:05:34.442529+0000","last_active":"2024-04-19T01:05:34.442370+0000","last_peered":"2024-04-19T01:05:34.442370+0000","last_clean":"2024-04-19T01:05:34.442370+0000","last_became_active":"2024-04-19T01:05:32.690355+0000","last_became_peered":"2024-04-19T01:05:32.690355+0000","last_unstale":"2024-04-19T01:05:34.442370+0000","last_undegraded":"2024-04-19T01:05:34.442370+0000","last_fullsized":"2024-04-19T01:05:34.442370+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'107","last_scrub_stamp":"2024-04-19T01:05:34.442172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:34.442172+0000","objects_scrubbed":3,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:48.822452+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0025400969999999998,"stat_sum":{"num_bytes":2916352,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2,3,1,4,12],"acting":[15,8,2,3,1,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"2e","length":"2"}]},{"pgid":"2.2","version":"0'0","reported_seq":210,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-04-19T01:05:33.673579+0000","last_change":"2024-04-19T01:05:08.432764+0000","last_active":"2024-04-19T01:05:33.673579+0000","last_peered":"2024-04-19T01:05:33.673579+0000","last_clean":"2024-04-19T01:05:33.673579+0000","last_became_active":"2024-04-19T01:03:54.011460+0000","last_became_peered":"2024-04-19T01:03:54.011460+0000","last_unstale":"2024-04-19T01:05:33.673579+0000","last_undegraded":"2024-04-19T01:05:33.673579+0000","last_fullsized":"2024-04-19T01:05:33.673579+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:08.432519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:08.432519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:10.143460+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.0","version":"117'128","reported_seq":611,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-04-19T01:05:33.693432+0000","last_change":"2024-04-19T01:05:33.698401+0000","last_active":"2024-04-19T01:05:33.693432+0000","last_peered":"2024-04-19T01:05:33.693432+0000","last_clean":"2024-04-19T01:05:33.693432+0000","last_became_active":"2024-04-19T01:05:32.706123+0000","last_became_peered":"2024-04-19T01:05:32.706123+0000","last_unstale":"2024-04-19T01:05:33.693432+0000","last_undegraded":"2024-04-19T01:05:33.693432+0000","last_fullsized":"2024-04-19T01:05:33.693432+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'128","last_scrub_stamp":"2024-04-19T01:05:30.689665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:30.689665+0000","objects_scrubbed":3,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0056890070000000003,"stat_sum":{"num_bytes":3932160,"num_objects":1,"num_object_clones":1,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4,3,10,11,6],"acting":[1,9,4,3,10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"2e","length":"2"}]},{"pgid":"2.1","version":"0'0","reported_seq":200,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-04-19T01:05:33.700642+0000","last_change":"2024-04-19T01:03:53.650434+0000","last_active":"2024-04-19T01:05:33.700642+0000","last_peered":"2024-04-19T01:05:33.700642+0000","last_clean":"2024-04-19T01:05:33.700642+0000","last_became_active":"2024-04-19T01:03:53.649606+0000","last_became_peered":"2024-04-19T01:03:53.649606+0000","last_unstale":"2024-04-19T01:05:33.700642+0000","last_undegraded":"2024-04-19T01:05:33.700642+0000","last_fullsized":"2024-04-19T01:05:33.700642+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:03:52.624648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:05:34.381895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.1","version":"123'80","reported_seq":559,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-04-19T01:05:33.706692+0000","last_change":"2024-04-19T01:05:33.707497+0000","last_active":"2024-04-19T01:05:33.706692+0000","last_peered":"2024-04-19T01:05:33.706692+0000","last_clean":"2024-04-19T01:05:33.706692+0000","last_became_active":"2024-04-19T01:05:32.696604+0000","last_became_peered":"2024-04-19T01:05:32.696604+0000","last_unstale":"2024-04-19T01:05:33.706692+0000","last_undegraded":"2024-04-19T01:05:33.706692+0000","last_fullsized":"2024-04-19T01:05:33.706692+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'78","last_scrub_stamp":"2024-04-19T01:05:33.652652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:33.652652+0000","objects_scrubbed":2,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:53.790617+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.024792403000000001,"stat_sum":{"num_bytes":2555904,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,11,9,1,7],"acting":[6,4,3,11,9,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"2e","length":"2"}]},{"pgid":"2.0","version":"0'0","reported_seq":210,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-04-19T01:05:33.677602+0000","last_change":"2024-04-19T01:05:32.160602+0000","last_active":"2024-04-19T01:05:33.677602+0000","last_peered":"2024-04-19T01:05:33.677602+0000","last_clean":"2024-04-19T01:05:33.677602+0000","last_became_active":"2024-04-19T01:03:53.759326+0000","last_became_peered":"2024-04-19T01:03:53.759326+0000","last_unstale":"2024-04-19T01:05:33.677602+0000","last_undegraded":"2024-04-19T01:05:33.677602+0000","last_fullsized":"2024-04-19T01:05:33.677602+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:32.160242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:32.160242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:40.352873+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.2","version":"116'63","reported_seq":563,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-04-19T01:05:33.680826+0000","last_change":"2024-04-19T01:05:33.684202+0000","last_active":"2024-04-19T01:05:33.680826+0000","last_peered":"2024-04-19T01:05:33.680826+0000","last_clean":"2024-04-19T01:05:33.680826+0000","last_became_active":"2024-04-19T01:05:32.703917+0000","last_became_peered":"2024-04-19T01:05:32.703917+0000","last_unstale":"2024-04-19T01:05:33.680826+0000","last_undegraded":"2024-04-19T01:05:33.680826+0000","last_fullsized":"2024-04-19T01:05:33.680826+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074466209999999996,"stat_sum":{"num_bytes":1740800,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6998,"num_write":5,"num_write_kb":1784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":647168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,13,4,8,3,7],"acting":[11,5,13,4,8,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"2e","length":"2"}]},{"pgid":"2.3","version":"0'0","reported_seq":215,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-04-19T01:05:33.675221+0000","last_change":"2024-04-19T01:05:07.549099+0000","last_active":"2024-04-19T01:05:33.675221+0000","last_peered":"2024-04-19T01:05:33.675221+0000","last_clean":"2024-04-19T01:05:33.675221+0000","last_became_active":"2024-04-19T01:05:06.529945+0000","last_became_peered":"2024-04-19T01:05:06.529945+0000","last_unstale":"2024-04-19T01:05:33.675221+0000","last_undegraded":"2024-04-19T01:05:33.675221+0000","last_fullsized":"2024-04-19T01:05:33.675221+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:07.548635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:07.548635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:16.001957+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"25'157","reported_seq":412,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-04-19T01:05:33.679315+0000","last_change":"2024-04-19T01:05:11.601754+0000","last_active":"2024-04-19T01:05:33.679315+0000","last_peered":"2024-04-19T01:05:33.679315+0000","last_clean":"2024-04-19T01:05:33.679315+0000","last_became_active":"2024-04-19T01:03:50.774078+0000","last_became_peered":"2024-04-19T01:03:50.774078+0000","last_unstale":"2024-04-19T01:05:33.679315+0000","last_undegraded":"2024-04-19T01:05:33.679315+0000","last_fullsized":"2024-04-19T01:05:33.679315+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'157","last_scrub_stamp":"2024-04-19T01:05:11.601630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:49.580851+0000","last_clean_scrub_stamp":"2024-04-19T01:05:11.601630+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:12.040402+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.8","version":"123'86","reported_seq":598,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-04-19T01:05:33.713529+0000","last_change":"2024-04-19T01:05:33.715906+0000","last_active":"2024-04-19T01:05:33.713529+0000","last_peered":"2024-04-19T01:05:33.713529+0000","last_clean":"2024-04-19T01:05:33.713529+0000","last_became_active":"2024-04-19T01:05:26.588578+0000","last_became_peered":"2024-04-19T01:05:26.588578+0000","last_unstale":"2024-04-19T01:05:33.713529+0000","last_undegraded":"2024-04-19T01:05:33.713529+0000","last_fullsized":"2024-04-19T01:05:33.713529+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'75","last_scrub_stamp":"2024-04-19T01:05:31.759068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:31.759068+0000","objects_scrubbed":5,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:45.143590+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.036491469999999998,"stat_sum":{"num_bytes":4702208,"num_objects":7,"num_object_clones":4,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":5,"num_write_kb":1552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,1,15,14,6,5],"acting":[11,10,1,15,14,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"2e","length":"2"}]},{"pgid":"3.9","version":"119'155","reported_seq":627,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-19T01:05:30.769880+0000","last_change":"2024-04-19T01:05:30.769988+0000","last_active":"2024-04-19T01:05:30.769880+0000","last_peered":"2024-04-19T01:05:30.769880+0000","last_clean":"2024-04-19T01:05:30.769880+0000","last_became_active":"2024-04-19T01:05:26.556542+0000","last_became_peered":"2024-04-19T01:05:26.556542+0000","last_unstale":"2024-04-19T01:05:30.769880+0000","last_undegraded":"2024-04-19T01:05:30.769880+0000","last_fullsized":"2024-04-19T01:05:30.769880+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'155","last_scrub_stamp":"2024-04-19T01:05:30.769682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:30.769682+0000","objects_scrubbed":5,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:55.520466+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0025435929999999998,"stat_sum":{"num_bytes":6864896,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11,14,9,4,10,6],"acting":[13,11,14,9,4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"}]},{"pgid":"3.a","version":"115'114","reported_seq":597,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-04-19T01:05:33.727657+0000","last_change":"2024-04-19T01:05:33.720636+0000","last_active":"2024-04-19T01:05:33.727657+0000","last_peered":"2024-04-19T01:05:33.727657+0000","last_clean":"2024-04-19T01:05:33.727657+0000","last_became_active":"2024-04-19T01:05:32.663812+0000","last_became_peered":"2024-04-19T01:05:32.663812+0000","last_unstale":"2024-04-19T01:05:33.727657+0000","last_undegraded":"2024-04-19T01:05:33.727657+0000","last_fullsized":"2024-04-19T01:05:33.727657+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'114","last_scrub_stamp":"2024-04-19T01:05:33.556107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:33.556107+0000","objects_scrubbed":2,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:07:01.429242+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0039639560000000003,"stat_sum":{"num_bytes":3702784,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":10849,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,7,3,13,1,8],"acting":[12,4,7,3,13,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"2e","length":"2"}]},{"pgid":"3.b","version":"105'56","reported_seq":527,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-19T01:05:33.522366+0000","last_change":"2024-04-19T01:05:33.522494+0000","last_active":"2024-04-19T01:05:33.522366+0000","last_peered":"2024-04-19T01:05:33.522366+0000","last_clean":"2024-04-19T01:05:33.522366+0000","last_became_active":"2024-04-19T01:05:32.690324+0000","last_became_peered":"2024-04-19T01:05:32.690324+0000","last_unstale":"2024-04-19T01:05:33.522366+0000","last_undegraded":"2024-04-19T01:05:33.522366+0000","last_fullsized":"2024-04-19T01:05:33.522366+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'56","last_scrub_stamp":"2024-04-19T01:05:33.522276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:33.522276+0000","objects_scrubbed":2,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:07:01.969534+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0049222689999999999,"stat_sum":{"num_bytes":3489792,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11,15,1,4,14],"acting":[10,6,11,15,1,4,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"2e","length":"1"}]},{"pgid":"3.c","version":"122'91","reported_seq":548,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-04-19T01:05:33.684563+0000","last_change":"2024-04-19T01:05:33.687961+0000","last_active":"2024-04-19T01:05:33.684563+0000","last_peered":"2024-04-19T01:05:33.684563+0000","last_clean":"2024-04-19T01:05:33.684563+0000","last_became_active":"2024-04-19T01:05:32.708435+0000","last_became_peered":"2024-04-19T01:05:32.708435+0000","last_unstale":"2024-04-19T01:05:33.684563+0000","last_undegraded":"2024-04-19T01:05:33.684563+0000","last_fullsized":"2024-04-19T01:05:33.684563+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010112682,"stat_sum":{"num_bytes":1523712,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":1227,"num_write":23,"num_write_kb":2152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,12,1,9,2,6],"acting":[11,10,12,1,9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"2e","length":"2"}]},{"pgid":"3.d","version":"120'108","reported_seq":591,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-04-19T01:05:33.722140+0000","last_change":"2024-04-19T01:05:33.722866+0000","last_active":"2024-04-19T01:05:33.722140+0000","last_peered":"2024-04-19T01:05:33.722140+0000","last_clean":"2024-04-19T01:05:33.722140+0000","last_became_active":"2024-04-19T01:05:32.667378+0000","last_became_peered":"2024-04-19T01:05:32.667378+0000","last_unstale":"2024-04-19T01:05:33.722140+0000","last_undegraded":"2024-04-19T01:05:33.722140+0000","last_fullsized":"2024-04-19T01:05:33.722140+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:04:01.712349+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058020160000000001,"stat_sum":{"num_bytes":4612096,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":2164,"num_write":19,"num_write_kb":5048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":2375680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,9,1,8,2,6],"acting":[12,4,9,1,8,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"2e","length":"2"}]},{"pgid":"3.e","version":"115'57","reported_seq":507,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-19T01:05:30.760029+0000","last_change":"2024-04-19T01:05:27.630977+0000","last_active":"2024-04-19T01:05:30.760029+0000","last_peered":"2024-04-19T01:05:30.760029+0000","last_clean":"2024-04-19T01:05:30.760029+0000","last_became_active":"2024-04-19T01:04:03.238386+0000","last_became_peered":"2024-04-19T01:04:03.238386+0000","last_unstale":"2024-04-19T01:05:30.760029+0000","last_undegraded":"2024-04-19T01:05:30.760029+0000","last_fullsized":"2024-04-19T01:05:30.760029+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'45","last_scrub_stamp":"2024-04-19T01:05:05.862139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:05.862139+0000","objects_scrubbed":3,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:11.931468+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0020063640000000001,"stat_sum":{"num_bytes":2277376,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":22,"num_read_kb":8695,"num_write":17,"num_write_kb":5088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11,5,4,2,9,15],"acting":[13,11,5,4,2,9,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"}]},{"pgid":"3.f","version":"117'104","reported_seq":548,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-04-19T01:05:32.699419+0000","last_change":"2024-04-19T01:05:32.699419+0000","last_active":"2024-04-19T01:05:32.699419+0000","last_peered":"2024-04-19T01:05:32.699419+0000","last_clean":"2024-04-19T01:05:32.699419+0000","last_became_active":"2024-04-19T01:05:32.689829+0000","last_became_peered":"2024-04-19T01:05:32.689829+0000","last_unstale":"2024-04-19T01:05:32.699419+0000","last_undegraded":"2024-04-19T01:05:32.699419+0000","last_fullsized":"2024-04-19T01:05:32.699419+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'81","last_scrub_stamp":"2024-04-19T01:05:18.498676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:18.498676+0000","objects_scrubbed":4,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0054793940000000003,"stat_sum":{"num_bytes":6668288,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":2646,"num_write":31,"num_write_kb":6512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,13,5,11,7,1,2],"acting":[10,13,5,11,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"2e","length":"1"}]},{"pgid":"3.10","version":"117'130","reported_seq":574,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-19T01:05:29.262962+0000","last_change":"2024-04-19T01:05:27.620672+0000","last_active":"2024-04-19T01:05:29.262962+0000","last_peered":"2024-04-19T01:05:29.262962+0000","last_clean":"2024-04-19T01:05:29.262962+0000","last_became_active":"2024-04-19T01:05:26.609028+0000","last_became_peered":"2024-04-19T01:05:26.609028+0000","last_unstale":"2024-04-19T01:05:29.262962+0000","last_undegraded":"2024-04-19T01:05:29.262962+0000","last_fullsized":"2024-04-19T01:05:29.262962+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'96","last_scrub_stamp":"2024-04-19T01:05:06.447000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:06.447000+0000","objects_scrubbed":5,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.011808848,"stat_sum":{"num_bytes":4595712,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":15,"num_read_kb":25134,"num_write":22,"num_write_kb":5480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4,3,10,11,6],"acting":[1,9,4,3,10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"}]},{"pgid":"3.11","version":"116'72","reported_seq":533,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-04-19T01:05:33.682828+0000","last_change":"2024-04-19T01:05:33.683577+0000","last_active":"2024-04-19T01:05:33.682828+0000","last_peered":"2024-04-19T01:05:33.682828+0000","last_clean":"2024-04-19T01:05:33.682828+0000","last_became_active":"2024-04-19T01:05:26.626309+0000","last_became_peered":"2024-04-19T01:05:26.626309+0000","last_unstale":"2024-04-19T01:05:33.682828+0000","last_undegraded":"2024-04-19T01:05:33.682828+0000","last_fullsized":"2024-04-19T01:05:33.682828+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":5,"last_scrub":"116'72","last_scrub_stamp":"2024-04-19T01:05:30.683830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:30.683830+0000","objects_scrubbed":2,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:46.857597+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0.001293163,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,11,9,1,7],"acting":[6,4,3,11,9,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"2e","length":"2"}]},{"pgid":"3.12","version":"113'61","reported_seq":556,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-04-19T01:05:33.679292+0000","last_change":"2024-04-19T01:05:33.680599+0000","last_active":"2024-04-19T01:05:33.679292+0000","last_peered":"2024-04-19T01:05:33.679292+0000","last_clean":"2024-04-19T01:05:33.679292+0000","last_became_active":"2024-04-19T01:05:26.593802+0000","last_became_peered":"2024-04-19T01:05:26.593802+0000","last_unstale":"2024-04-19T01:05:33.679292+0000","last_undegraded":"2024-04-19T01:05:33.679292+0000","last_fullsized":"2024-04-19T01:05:33.679292+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":5,"last_scrub":"113'61","last_scrub_stamp":"2024-04-19T01:05:32.765866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:32.765866+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:45.093940+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0044409860000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,13,4,8,3,7],"acting":[11,5,13,4,8,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"2e","length":"2"}]},{"pgid":"3.13","version":"123'111","reported_seq":616,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-04-19T01:05:33.719265+0000","last_change":"2024-04-19T01:05:33.722444+0000","last_active":"2024-04-19T01:05:33.719265+0000","last_peered":"2024-04-19T01:05:33.719265+0000","last_clean":"2024-04-19T01:05:33.719265+0000","last_became_active":"2024-04-19T01:05:26.572738+0000","last_became_peered":"2024-04-19T01:05:26.572738+0000","last_unstale":"2024-04-19T01:05:33.719265+0000","last_undegraded":"2024-04-19T01:05:33.719265+0000","last_fullsized":"2024-04-19T01:05:33.719265+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":5,"last_scrub":"116'105","last_scrub_stamp":"2024-04-19T01:05:30.439101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:04:01.712349+0000","last_clean_scrub_stamp":"2024-04-19T01:05:30.439101+0000","objects_scrubbed":2,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:55.129434+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.011162013,"stat_sum":{"num_bytes":4259840,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2,3,1,4,12],"acting":[15,8,2,3,1,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"24","length":"1"},{"start":"27","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"2e","length":"2"}]}],"pool_stats":[{"poolid":3,"num_pg":36,"stat_sum":{"num_bytes":111570944,"num_objects":76,"num_object_clones":32,"num_object_copies":532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":4,"num_read":129,"num_read_kb":85801,"num_write":234,"num_write_kb":45448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":6938624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":190750720,"data_stored":275136512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3264,"ondisk_log_size":3264,"up":245,"acting":245,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":13},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":15546,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":14,"up_from":24,"seq":103079215128,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46564,"kb_used_data":11216,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94325276,"statfs":{"total":96636764160,"available":96589082624,"internally_reserved":0,"allocated":11485184,"data_stored":14226571,"data_compressed":194329,"data_compressed_allocated":397312,"data_compressed_original":794624,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,5,6,7,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":13,"up_from":24,"seq":103079215129,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49204,"kb_used_data":13860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94322636,"statfs":{"total":96636764160,"available":96586379264,"internally_reserved":0,"allocated":14192640,"data_stored":18168922,"data_compressed":192287,"data_compressed_allocated":393216,"data_compressed_original":786432,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":24,"seq":103079215130,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46992,"kb_used_data":11644,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94324848,"statfs":{"total":96636764160,"available":96588644352,"internally_reserved":0,"allocated":11923456,"data_stored":17175691,"data_compressed":194329,"data_compressed_allocated":397312,"data_compressed_original":794624,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[1,2,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313241,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48304,"kb_used_data":17120,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323536,"statfs":{"total":96636764160,"available":96587300864,"internally_reserved":0,"allocated":17530880,"data_stored":24430047,"data_compressed":198417,"data_compressed_allocated":405504,"data_compressed_original":811008,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280535,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47596,"kb_used_data":16412,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324244,"statfs":{"total":96636764160,"available":96588025856,"internally_reserved":0,"allocated":16805888,"data_stored":24679903,"data_compressed":198417,"data_compressed_allocated":405504,"data_compressed_original":811008,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,3,4,5,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280537,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49372,"kb_used_data":18184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322468,"statfs":{"total":96636764160,"available":96586207232,"internally_reserved":0,"allocated":18620416,"data_stored":27332241,"data_compressed":200468,"data_compressed_allocated":409600,"data_compressed_original":819200,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,6,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280536,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42396,"kb_used_data":11208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329444,"statfs":{"total":96636764160,"available":96593350656,"internally_reserved":0,"allocated":11476992,"data_stored":15699601,"data_compressed":200468,"data_compressed_allocated":409600,"data_compressed_original":819200,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[2,4,5,6,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280536,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48124,"kb_used_data":16936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323716,"statfs":{"total":96636764160,"available":96587485184,"internally_reserved":0,"allocated":17342464,"data_stored":23748241,"data_compressed":200468,"data_compressed_allocated":409600,"data_compressed_original":819200,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280536,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44248,"kb_used_data":13060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327592,"statfs":{"total":96636764160,"available":96591454208,"internally_reserved":0,"allocated":13373440,"data_stored":19120305,"data_compressed":208241,"data_compressed_allocated":442368,"data_compressed_original":1331200,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,6,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280537,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43792,"kb_used_data":12604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328048,"statfs":{"total":96636764160,"available":96591921152,"internally_reserved":0,"allocated":12906496,"data_stored":16567953,"data_compressed":200468,"data_compressed_allocated":409600,"data_compressed_original":819200,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,5,6,7,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280535,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68992,"kb_used_data":19756,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94302848,"statfs":{"total":96636764160,"available":96566116352,"internally_reserved":0,"allocated":20230144,"data_stored":30199985,"data_compressed":208241,"data_compressed_allocated":442368,"data_compressed_original":1331200,"omap_allocated":1596,"internal_metadata":50395588},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280535,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40000,"kb_used_data":8812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331840,"statfs":{"total":96636764160,"available":96595804160,"internally_reserved":0,"allocated":9023488,"data_stored":12377745,"data_compressed":200468,"data_compressed_allocated":409600,"data_compressed_original":819200,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,6,7,8,9,10,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280536,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39244,"kb_used_data":8056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332596,"statfs":{"total":96636764160,"available":96596578304,"internally_reserved":0,"allocated":8249344,"data_stored":10571409,"data_compressed":200468,"data_compressed_allocated":409600,"data_compressed_original":819200,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":24,"seq":103079215128,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48108,"kb_used_data":12760,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94323732,"statfs":{"total":96636764160,"available":96587501568,"internally_reserved":0,"allocated":13066240,"data_stored":17740939,"data_compressed":194329,"data_compressed_allocated":397312,"data_compressed_original":794624,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":0,"seq":103079215129,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":24,"seq":103079215128,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41024,"kb_used_data":9836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330816,"statfs":{"total":96636764160,"available":96594755584,"internally_reserved":0,"allocated":10072064,"data_stored":13051019,"data_compressed":194329,"data_compressed_allocated":397312,"data_compressed_original":794624,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5787648,"data_stored":6684672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":17240064,"data_stored":26951680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11669504,"data_stored":16674816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":10043392,"data_stored":14602240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":15908864,"data_stored":22650880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":11886592,"data_stored":17494016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15384576,"data_stored":23568384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8675328,"data_stored":11984896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":11472896,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":10526720,"data_stored":16109568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":16109568,"data_stored":23343104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":18743296,"data_stored":28573696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":7589888,"data_stored":11280384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":12808192,"data_stored":17113088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":10088448,"data_stored":13160448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":6815744,"data_stored":9474048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T01:05:35.432 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2024-04-19T01:05:35.432 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-04-19T01:05:35.434 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-19T01:05:35.475 INFO:tasks.ceph.osd.9.smithi110.stderr:2024-04-19T01:05:35.476+0000 7f23fa4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25043) UID: 0 2024-04-19T01:05:35.554 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:35.555 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:35.569 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-19T01:05:35.576 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:05:35.574+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:05:35.677 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-04-19T01:05:35.674+0000 7fe4bd939640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25022) UID: 0 2024-04-19T01:05:35.678 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.14 primary-affinity to 0.593582 (8389002) 2024-04-19T01:05:35.678 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2024-04-19T01:05:35.693 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:35.693 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:35.709 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-19T01:05:35.778 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:05:35.776+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:05:35.832 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:35.832 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:35.847 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-19T01:05:35.892 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:05:35.892+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:05:35.980 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:05:35.980+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:05:36.012 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:36.012 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:36.026 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-19T01:05:36.054 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-19T01:05:36.083 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:05:36.078+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:05:36.184 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-04-19T01:05:36.184+0000 7fc156105640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25347) UID: 0 2024-04-19T01:05:36.188 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:36.188 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:36.203 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-19T01:05:36.216 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2024-04-19T01:05:36.216 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T01:05:36.237 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2024-04-19T01:05:36.237 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T01:05:36.261 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2024-04-19T01:05:36.261 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T01:05:36.279 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2024-04-19T01:05:36.280 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T01:05:36.285 INFO:tasks.ceph.osd.5.smithi110.stderr:2024-04-19T01:05:36.284+0000 7f2726e7c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25041) UID: 0 2024-04-19T01:05:36.300 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2024-04-19T01:05:36.300 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T01:05:36.322 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2024-04-19T01:05:36.322 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T01:05:36.342 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2024-04-19T01:05:36.342 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T01:05:36.363 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2024-04-19T01:05:36.363 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T01:05:36.371 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:36.372 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:36.382 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2024-04-19T01:05:36.382 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T01:05:36.384 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-04-19T01:05:36.386 INFO:tasks.ceph.osd.10.smithi118.stderr:2024-04-19T01:05:36.386+0000 7f744a718640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25024) UID: 0 2024-04-19T01:05:36.405 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2024-04-19T01:05:36.405 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T01:05:36.426 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2024-04-19T01:05:36.426 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T01:05:36.447 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2024-04-19T01:05:36.447 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T01:05:36.467 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2024-04-19T01:05:36.468 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-19T01:05:36.487 INFO:tasks.ceph.osd.2.smithi118.stderr:2024-04-19T01:05:36.482+0000 7f127dabe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25019) UID: 0 2024-04-19T01:05:36.490 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2024-04-19T01:05:36.491 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-19T01:05:36.502 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-04-19T01:05:36.503 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T01:05:36.513 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2024-04-19T01:05:36.513 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-19T01:05:36.516 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-04-19T01:05:36.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:05:36.528+0000 7f4b077fe640 0 -- 172.21.15.33:0/2586978919 >> [v2:172.21.15.155:6807/741132602,v1:172.21.15.155:6810/741132602] conn(0x7f4b082244b0 msgr2=0x7f4b082268a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-19T01:05:36.536 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2024-04-19T01:05:36.587 INFO:tasks.ceph.osd.15.smithi155.stderr:2024-04-19T01:05:36.588+0000 7f0560ad4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25352) UID: 0 2024-04-19T01:05:36.635 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:05:36.635 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-04-19T01:05:36.635 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-04-19T01:05:36.635 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-04-19T01:05:36.635 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:05:36.650 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-04-19T01:05:36.680 INFO:teuthology.orchestra.run.smithi033.stderr:pool 'unique_pool_0' removed 2024-04-19T01:05:36.692 INFO:tasks.ceph.osd.8.smithi033.stderr:2024-04-19T01:05:36.684+0000 7f6abd57f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25029) UID: 0 2024-04-19T01:05:36.712 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-04-19T01:05:36.726 INFO:tasks.thrashosds:joining thrashosds 2024-04-19T01:05:36.818 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-19T01:05:36.819 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-04-19T01:05:36.819 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-04-19T01:05:36.819 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-04-19T01:05:36.819 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.819 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:173 3.3s0 3:c34529b5:::smithi15528762-31:head [append 0~507904] snapc 5=[5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-19T01:05:36.819 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:08.816172+0000", 2024-04-19T01:05:36.819 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 88.001135957000002, 2024-04-19T01:05:36.819 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.024750307999999999, 2024-04-19T01:05:36.819 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:36.819 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:36.819 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:36.819 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:36.820 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:36.820 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:36.820 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 173 2024-04-19T01:05:36.820 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.820 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:36.820 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.820 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:36.820 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816172+0000", 2024-04-19T01:05:36.820 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:36.820 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.820 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.820 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:36.820 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816172+0000", 2024-04-19T01:05:36.820 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:36.821 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.821 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.821 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:36.821 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816175+0000", 2024-04-19T01:05:36.821 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9589999999999997e-06 2024-04-19T01:05:36.821 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.821 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.821 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:36.821 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816766+0000", 2024-04-19T01:05:36.821 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00059020999999999998 2024-04-19T01:05:36.821 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.821 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.821 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:36.822 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816802+0000", 2024-04-19T01:05:36.822 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6140999999999998e-05 2024-04-19T01:05:36.822 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.822 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.822 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:36.822 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816820+0000", 2024-04-19T01:05:36.822 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7771999999999999e-05 2024-04-19T01:05:36.822 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.822 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.822 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:36.822 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.816990+0000", 2024-04-19T01:05:36.822 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016993299999999999 2024-04-19T01:05:36.822 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.823 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.823 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op must wait for map", 2024-04-19T01:05:36.823 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.817016+0000", 2024-04-19T01:05:36.823 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6857000000000001e-05 2024-04-19T01:05:36.823 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.823 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.823 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:36.823 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.821660+0000", 2024-04-19T01:05:36.823 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0046440520000000001 2024-04-19T01:05:36.823 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.823 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.824 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:36.824 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.821835+0000", 2024-04-19T01:05:36.824 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000174112 2024-04-19T01:05:36.824 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.824 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.824 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:36.824 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.824020+0000", 2024-04-19T01:05:36.824 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.002185523 2024-04-19T01:05:36.824 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.824 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.824 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:36.824 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.839423+0000", 2024-04-19T01:05:36.825 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.015403261999999999 2024-04-19T01:05:36.825 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.825 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.825 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:36.825 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.840758+0000", 2024-04-19T01:05:36.825 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001335145 2024-04-19T01:05:36.825 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.825 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.825 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:36.825 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.840922+0000", 2024-04-19T01:05:36.825 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000163342 2024-04-19T01:05:36.825 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:36.825 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:36.826 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:36.826 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.826 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.826 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:176 3.3s0 3:c34529b5:::smithi15528762-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-19T01:05:36.826 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:08.822193+0000", 2024-04-19T01:05:36.826 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 87.995114494000006, 2024-04-19T01:05:36.826 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.023273202, 2024-04-19T01:05:36.826 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:36.826 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:36.827 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-04-19T01:05:36.827 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:36.827 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:36.827 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:36.827 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 176 2024-04-19T01:05:36.827 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.827 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:36.827 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.827 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:36.827 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822193+0000", 2024-04-19T01:05:36.827 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:36.827 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.828 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.828 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:36.828 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822193+0000", 2024-04-19T01:05:36.828 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:36.828 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.828 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.828 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:36.828 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822194+0000", 2024-04-19T01:05:36.828 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4200000000000004e-07 2024-04-19T01:05:36.828 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.828 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.828 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:36.828 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822198+0000", 2024-04-19T01:05:36.829 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.206e-06 2024-04-19T01:05:36.829 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.829 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.829 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:36.829 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822210+0000", 2024-04-19T01:05:36.829 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.217e-05 2024-04-19T01:05:36.829 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.829 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.829 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:36.829 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.822220+0000", 2024-04-19T01:05:36.830 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0059e-05 2024-04-19T01:05:36.830 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.830 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.830 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:36.830 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.842930+0000", 2024-04-19T01:05:36.830 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.020709701 2024-04-19T01:05:36.830 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.830 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.830 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:36.830 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.843170+0000", 2024-04-19T01:05:36.830 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00024026700000000001 2024-04-19T01:05:36.830 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.831 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.831 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:36.831 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:08.845466+0000", 2024-04-19T01:05:36.831 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0022958570000000001 2024-04-19T01:05:36.831 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:36.831 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:36.831 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:36.831 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.831 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.831 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:343 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [copy-from ver 14] snapc 9=[8,7] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:05:36.832 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.511298+0000", 2024-04-19T01:05:36.832 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 78.306009496000001, 2024-04-19T01:05:36.832 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.043267551000000001, 2024-04-19T01:05:36.832 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:36.832 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:36.832 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:36.832 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:36.832 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:36.832 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:36.832 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 343 2024-04-19T01:05:36.832 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.832 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:36.832 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.833 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:36.833 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511298+0000", 2024-04-19T01:05:36.833 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:36.833 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.833 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.833 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:36.833 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511298+0000", 2024-04-19T01:05:36.833 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:36.833 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.833 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.833 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:36.833 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511302+0000", 2024-04-19T01:05:36.833 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1030000000000004e-06 2024-04-19T01:05:36.833 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.834 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.834 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:36.834 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511319+0000", 2024-04-19T01:05:36.834 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7070000000000001e-05 2024-04-19T01:05:36.834 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.834 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.834 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:36.834 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511357+0000", 2024-04-19T01:05:36.834 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8056000000000003e-05 2024-04-19T01:05:36.834 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.834 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.834 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:36.834 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511390+0000", 2024-04-19T01:05:36.835 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3213000000000002e-05 2024-04-19T01:05:36.835 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.835 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.835 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:36.835 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.516539+0000", 2024-04-19T01:05:36.835 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0051490729999999997 2024-04-19T01:05:36.835 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.835 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.835 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:36.835 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.516748+0000", 2024-04-19T01:05:36.836 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000208099 2024-04-19T01:05:36.836 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.836 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.836 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:36.836 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.541650+0000", 2024-04-19T01:05:36.836 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.024902837000000001 2024-04-19T01:05:36.836 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.836 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.836 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:36.836 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.551197+0000", 2024-04-19T01:05:36.836 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0095463240000000001 2024-04-19T01:05:36.836 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.836 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.837 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:36.837 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.554383+0000", 2024-04-19T01:05:36.837 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0031863830000000001 2024-04-19T01:05:36.837 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.837 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.837 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:36.837 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.554566+0000", 2024-04-19T01:05:36.837 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00018239299999999999 2024-04-19T01:05:36.837 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:36.837 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:36.837 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:36.837 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.837 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.838 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:344 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e48)", 2024-04-19T01:05:36.838 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:18.511452+0000", 2024-04-19T01:05:36.838 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 78.305855245999993, 2024-04-19T01:05:36.838 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.038223957000000003, 2024-04-19T01:05:36.838 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:36.838 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:36.838 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-04-19T01:05:36.838 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:36.838 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:36.838 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:36.838 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 344 2024-04-19T01:05:36.838 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.838 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:36.838 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.839 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:36.839 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511452+0000", 2024-04-19T01:05:36.839 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:36.839 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.839 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.839 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:36.839 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511452+0000", 2024-04-19T01:05:36.839 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:36.839 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.839 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.839 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:36.839 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511454+0000", 2024-04-19T01:05:36.839 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.39e-06 2024-04-19T01:05:36.839 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.840 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.840 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:36.840 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511460+0000", 2024-04-19T01:05:36.840 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.46e-06 2024-04-19T01:05:36.840 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.840 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.840 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:36.840 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511476+0000", 2024-04-19T01:05:36.840 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6180000000000001e-05 2024-04-19T01:05:36.840 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.840 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.840 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:36.840 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.511487+0000", 2024-04-19T01:05:36.840 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.049e-05 2024-04-19T01:05:36.841 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.841 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.841 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:36.841 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.517203+0000", 2024-04-19T01:05:36.841 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00571595 2024-04-19T01:05:36.841 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.841 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.841 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for blocked object", 2024-04-19T01:05:36.841 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.517341+0000", 2024-04-19T01:05:36.841 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013835099999999999 2024-04-19T01:05:36.841 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.841 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.841 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:36.841 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.549188+0000", 2024-04-19T01:05:36.841 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.031846589000000002 2024-04-19T01:05:36.842 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.842 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.842 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:36.842 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.549399+0000", 2024-04-19T01:05:36.842 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021149600000000001 2024-04-19T01:05:36.842 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.842 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.842 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:36.842 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:18.549676+0000", 2024-04-19T01:05:36.842 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000277051 2024-04-19T01:05:36.842 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:36.843 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:36.843 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:36.843 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.843 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.843 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.8s3 49/31 ECSubWrite(tid=203, reqid=client.4604.0:360, at_version=49'25, trim_to=0'0, roll_forward_to=49'24))", 2024-04-19T01:05:36.843 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:19.539927+0000", 2024-04-19T01:05:36.843 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 77.277380062999995, 2024-04-19T01:05:36.843 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.027280209, 2024-04-19T01:05:36.843 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:36.843 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:36.843 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:36.843 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:36.843 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.843 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:36.843 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.539927+0000", 2024-04-19T01:05:36.844 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:36.844 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.844 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.844 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:36.844 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.539927+0000", 2024-04-19T01:05:36.844 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:36.844 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.844 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.844 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:36.844 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.539929+0000", 2024-04-19T01:05:36.844 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3829999999999999e-06 2024-04-19T01:05:36.844 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.845 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.845 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:36.845 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540190+0000", 2024-04-19T01:05:36.845 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00026120499999999998 2024-04-19T01:05:36.845 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.845 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.845 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:36.845 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540216+0000", 2024-04-19T01:05:36.845 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5573000000000001e-05 2024-04-19T01:05:36.845 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.845 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.845 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:36.845 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540242+0000", 2024-04-19T01:05:36.846 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5939999999999999e-05 2024-04-19T01:05:36.846 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.846 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.846 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:36.846 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540387+0000", 2024-04-19T01:05:36.846 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000145983 2024-04-19T01:05:36.846 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.846 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.846 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:36.846 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.540412+0000", 2024-04-19T01:05:36.847 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.442e-05 2024-04-19T01:05:36.847 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.866 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.866 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:36.866 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.567114+0000", 2024-04-19T01:05:36.866 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.026701643000000001 2024-04-19T01:05:36.867 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.867 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.867 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:36.867 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:19.567208+0000", 2024-04-19T01:05:36.867 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4061999999999995e-05 2024-04-19T01:05:36.867 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:36.867 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:36.867 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:36.868 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.868 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.868 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.8s3 60/31 ECSubWrite(tid=225, reqid=client.4604.0:433, at_version=60'43, trim_to=0'0, roll_forward_to=59'41))", 2024-04-19T01:05:36.868 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:30.569755+0000", 2024-04-19T01:05:36.868 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 66.247552861000003, 2024-04-19T01:05:36.868 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.026016885999999999, 2024-04-19T01:05:36.868 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:36.868 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:36.868 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:36.868 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:36.868 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.868 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:36.869 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.569755+0000", 2024-04-19T01:05:36.869 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:36.869 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.869 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.869 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:36.869 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.569755+0000", 2024-04-19T01:05:36.869 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:36.869 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.869 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.869 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:36.869 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.569755+0000", 2024-04-19T01:05:36.869 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6599999999999996e-07 2024-04-19T01:05:36.869 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.869 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.869 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:36.870 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.570112+0000", 2024-04-19T01:05:36.870 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00035699100000000002 2024-04-19T01:05:36.870 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.870 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.870 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:36.870 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.570159+0000", 2024-04-19T01:05:36.870 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7312999999999999e-05 2024-04-19T01:05:36.870 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.870 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.870 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:36.870 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.570184+0000", 2024-04-19T01:05:36.870 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4593e-05 2024-04-19T01:05:36.870 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.870 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.871 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:36.871 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.570362+0000", 2024-04-19T01:05:36.871 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017814400000000001 2024-04-19T01:05:36.871 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.871 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.871 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:36.871 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.570381+0000", 2024-04-19T01:05:36.871 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8369999999999999e-05 2024-04-19T01:05:36.871 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.871 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.871 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:36.871 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.595729+0000", 2024-04-19T01:05:36.871 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.025348186000000002 2024-04-19T01:05:36.871 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.871 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.872 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:36.872 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:30.595771+0000", 2024-04-19T01:05:36.872 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2722999999999997e-05 2024-04-19T01:05:36.872 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:36.872 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:36.872 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:36.872 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.872 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.872 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.4s4 71/31 ECSubWrite(tid=89, reqid=client.4604.0:518, at_version=71'22, trim_to=0'0, roll_forward_to=52'20))", 2024-04-19T01:05:36.872 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:41.542817+0000", 2024-04-19T01:05:36.872 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 55.274490641, 2024-04-19T01:05:36.872 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.068188129, 2024-04-19T01:05:36.872 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:36.872 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:36.873 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:36.873 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:36.873 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.873 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:36.873 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.542817+0000", 2024-04-19T01:05:36.873 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:36.873 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.874 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.874 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:36.874 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.542817+0000", 2024-04-19T01:05:36.874 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:36.874 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.874 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.874 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:36.874 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.542819+0000", 2024-04-19T01:05:36.875 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.283e-06 2024-04-19T01:05:36.875 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.875 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.875 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:36.875 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.545764+0000", 2024-04-19T01:05:36.875 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0029444530000000001 2024-04-19T01:05:36.875 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.875 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.875 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:36.875 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.545797+0000", 2024-04-19T01:05:36.875 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3203e-05 2024-04-19T01:05:36.875 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.875 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.875 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:36.875 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.545821+0000", 2024-04-19T01:05:36.876 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4436000000000001e-05 2024-04-19T01:05:36.876 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.876 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.876 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:36.876 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.545998+0000", 2024-04-19T01:05:36.876 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017682099999999999 2024-04-19T01:05:36.880 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.880 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.880 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:36.880 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.546048+0000", 2024-04-19T01:05:36.881 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0423000000000002e-05 2024-04-19T01:05:36.881 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.881 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.881 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:36.881 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.610960+0000", 2024-04-19T01:05:36.881 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.064912005999999994 2024-04-19T01:05:36.881 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.881 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.881 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:36.881 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:41.611005+0000", 2024-04-19T01:05:36.881 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4503999999999999e-05 2024-04-19T01:05:36.881 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:36.881 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:36.882 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:36.882 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.882 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.882 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:612 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [append 0~704512] snapc 1b=[1b,19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-04-19T01:05:36.883 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:50.652703+0000", 2024-04-19T01:05:36.883 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 46.164604427999997, 2024-04-19T01:05:36.883 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.035173242, 2024-04-19T01:05:36.883 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:36.883 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:36.883 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:36.883 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:36.883 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:36.883 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:36.883 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 612 2024-04-19T01:05:36.883 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.883 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:36.885 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.885 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:36.885 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.652703+0000", 2024-04-19T01:05:36.885 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:36.885 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.885 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.885 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:36.886 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.652703+0000", 2024-04-19T01:05:36.886 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:36.886 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.886 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.886 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:36.886 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.652706+0000", 2024-04-19T01:05:36.886 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4070000000000002e-06 2024-04-19T01:05:36.886 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.886 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.886 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:36.886 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.653472+0000", 2024-04-19T01:05:36.886 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00076519800000000003 2024-04-19T01:05:36.886 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.887 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.887 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:36.887 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.653507+0000", 2024-04-19T01:05:36.887 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5525999999999998e-05 2024-04-19T01:05:36.887 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.887 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.887 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:36.887 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.653525+0000", 2024-04-19T01:05:36.887 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7329e-05 2024-04-19T01:05:36.887 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.887 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.887 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:36.887 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.653662+0000", 2024-04-19T01:05:36.888 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013767900000000001 2024-04-19T01:05:36.888 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.888 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.888 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op must wait for map", 2024-04-19T01:05:36.888 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.653675+0000", 2024-04-19T01:05:36.888 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2723e-05 2024-04-19T01:05:36.888 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.888 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.888 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:36.888 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.658036+0000", 2024-04-19T01:05:36.888 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0043615820000000001 2024-04-19T01:05:36.888 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.888 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.888 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:36.889 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.658195+0000", 2024-04-19T01:05:36.889 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000158448 2024-04-19T01:05:36.889 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.889 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.889 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:36.889 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.660614+0000", 2024-04-19T01:05:36.889 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0024192749999999998 2024-04-19T01:05:36.889 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.889 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.889 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:36.889 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.687106+0000", 2024-04-19T01:05:36.889 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.026491879999999999 2024-04-19T01:05:36.890 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.890 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.890 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:36.890 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.687703+0000", 2024-04-19T01:05:36.890 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00059716299999999995 2024-04-19T01:05:36.890 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.890 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.890 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:36.890 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.687876+0000", 2024-04-19T01:05:36.890 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017303199999999999 2024-04-19T01:05:36.890 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:36.890 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:36.890 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:36.890 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.891 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.891 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:613 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [append 0~655360] snapc 1b=[1b,19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-04-19T01:05:36.891 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:50.661555+0000", 2024-04-19T01:05:36.891 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 46.155752192000001, 2024-04-19T01:05:36.891 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.038292557999999997, 2024-04-19T01:05:36.891 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:36.891 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:36.891 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:36.891 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:36.891 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:36.891 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:36.891 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 613 2024-04-19T01:05:36.891 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.892 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:36.892 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.892 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:36.892 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.661555+0000", 2024-04-19T01:05:36.892 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:36.892 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.892 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.892 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:36.892 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.661555+0000", 2024-04-19T01:05:36.893 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:36.893 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.893 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.893 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:36.893 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.661559+0000", 2024-04-19T01:05:36.893 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.337e-06 2024-04-19T01:05:36.893 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.893 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.893 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:36.893 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.662402+0000", 2024-04-19T01:05:36.893 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00084358700000000003 2024-04-19T01:05:36.893 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.893 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.893 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:36.894 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.662434+0000", 2024-04-19T01:05:36.894 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2223e-05 2024-04-19T01:05:36.894 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.894 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.894 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:36.894 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.662458+0000", 2024-04-19T01:05:36.894 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3598999999999999e-05 2024-04-19T01:05:36.894 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.894 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.894 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:36.895 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.673646+0000", 2024-04-19T01:05:36.896 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.011187852999999999 2024-04-19T01:05:36.896 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.896 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.896 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:36.896 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.673820+0000", 2024-04-19T01:05:36.896 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017450199999999999 2024-04-19T01:05:36.896 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.896 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.896 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:36.896 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.675719+0000", 2024-04-19T01:05:36.896 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001898757 2024-04-19T01:05:36.896 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.896 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.896 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:36.897 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.687984+0000", 2024-04-19T01:05:36.897 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.012265009 2024-04-19T01:05:36.897 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.897 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.897 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:36.897 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.699734+0000", 2024-04-19T01:05:36.897 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.011749426 2024-04-19T01:05:36.897 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.897 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.897 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:36.897 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.699848+0000", 2024-04-19T01:05:36.897 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000114265 2024-04-19T01:05:36.897 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:36.898 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:36.898 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:36.898 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.898 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.898 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:614 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [append 0~507904] snapc 1b=[1b,19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-04-19T01:05:36.898 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:50.668341+0000", 2024-04-19T01:05:36.898 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 46.148966956999999, 2024-04-19T01:05:36.898 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.032677876000000002, 2024-04-19T01:05:36.898 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:36.898 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:36.898 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:36.898 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:36.898 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:36.898 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:36.899 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 614 2024-04-19T01:05:36.899 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.899 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:36.899 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.899 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:36.899 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.668341+0000", 2024-04-19T01:05:36.899 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:36.899 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.899 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.901 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:36.902 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.668341+0000", 2024-04-19T01:05:36.902 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:36.902 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.902 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.902 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:36.902 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.668343+0000", 2024-04-19T01:05:36.902 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2759999999999999e-06 2024-04-19T01:05:36.902 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.902 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.902 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:36.902 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.668923+0000", 2024-04-19T01:05:36.902 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00058019699999999996 2024-04-19T01:05:36.902 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.902 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.903 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:36.903 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.668956+0000", 2024-04-19T01:05:36.903 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2539000000000001e-05 2024-04-19T01:05:36.903 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.903 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.903 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:36.903 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.668977+0000", 2024-04-19T01:05:36.903 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1903e-05 2024-04-19T01:05:36.903 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.903 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.903 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:36.903 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.688297+0000", 2024-04-19T01:05:36.903 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.019319106999999999 2024-04-19T01:05:36.903 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.904 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.904 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:36.904 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.688460+0000", 2024-04-19T01:05:36.904 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016338799999999999 2024-04-19T01:05:36.904 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.904 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.904 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:36.904 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.689733+0000", 2024-04-19T01:05:36.904 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001273499 2024-04-19T01:05:36.904 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.904 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.904 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:36.904 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.699996+0000", 2024-04-19T01:05:36.904 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.010263042 2024-04-19T01:05:36.904 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.905 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.905 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:36.905 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.700905+0000", 2024-04-19T01:05:36.905 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00090875999999999999 2024-04-19T01:05:36.905 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.905 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.905 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:36.905 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.701018+0000", 2024-04-19T01:05:36.905 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000113165 2024-04-19T01:05:36.905 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:36.905 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:36.905 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:36.905 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.905 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.906 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:615 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [setxattr _header (57) in=7b] snapc 1b=[1b,19,18,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-04-19T01:05:36.906 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:50.669014+0000", 2024-04-19T01:05:36.906 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 46.148293479000003, 2024-04-19T01:05:36.906 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.032983169999999999, 2024-04-19T01:05:36.906 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:36.906 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:36.906 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:36.906 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:36.906 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:36.906 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:36.906 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 615 2024-04-19T01:05:36.906 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.906 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:36.906 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.907 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:36.907 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669014+0000", 2024-04-19T01:05:36.907 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:36.907 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.907 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.907 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:36.907 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669014+0000", 2024-04-19T01:05:36.907 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:36.907 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.907 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.907 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:36.907 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669016+0000", 2024-04-19T01:05:36.907 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3439999999999999e-06 2024-04-19T01:05:36.907 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.908 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.908 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:36.908 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669039+0000", 2024-04-19T01:05:36.908 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3008999999999999e-05 2024-04-19T01:05:36.908 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.908 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.908 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:36.908 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669057+0000", 2024-04-19T01:05:36.908 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.749e-05 2024-04-19T01:05:36.908 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.908 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.908 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:36.908 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669073+0000", 2024-04-19T01:05:36.908 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6492999999999999e-05 2024-04-19T01:05:36.908 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.909 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.909 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:36.909 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.698263+0000", 2024-04-19T01:05:36.909 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.029189397999999998 2024-04-19T01:05:36.909 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.909 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.909 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:36.910 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.698412+0000", 2024-04-19T01:05:36.910 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014930499999999999 2024-04-19T01:05:36.910 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.910 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.910 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:36.910 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.699113+0000", 2024-04-19T01:05:36.910 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00070141799999999999 2024-04-19T01:05:36.911 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.911 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.911 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:36.911 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.701045+0000", 2024-04-19T01:05:36.911 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001931667 2024-04-19T01:05:36.911 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.911 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.911 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:36.911 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.701847+0000", 2024-04-19T01:05:36.914 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00080155099999999996 2024-04-19T01:05:36.914 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.914 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.914 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:36.914 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.701997+0000", 2024-04-19T01:05:36.914 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000150495 2024-04-19T01:05:36.914 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:36.915 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:36.915 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:36.915 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.915 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.915 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:616 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e80)", 2024-04-19T01:05:36.915 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:50.669097+0000", 2024-04-19T01:05:36.915 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 46.148210943000002, 2024-04-19T01:05:36.915 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.038907733999999999, 2024-04-19T01:05:36.915 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:36.915 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:36.915 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-04-19T01:05:36.915 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:36.915 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:36.915 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:36.915 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 616 2024-04-19T01:05:36.916 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.916 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:36.916 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.916 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:36.916 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669097+0000", 2024-04-19T01:05:36.916 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:36.916 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.916 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.916 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:36.916 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669097+0000", 2024-04-19T01:05:36.916 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:36.916 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.916 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.916 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:36.917 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669098+0000", 2024-04-19T01:05:36.917 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.587e-06 2024-04-19T01:05:36.917 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.917 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.917 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:36.917 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669105+0000", 2024-04-19T01:05:36.917 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3660000000000003e-06 2024-04-19T01:05:36.917 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.917 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.917 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:36.917 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669130+0000", 2024-04-19T01:05:36.917 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4377000000000001e-05 2024-04-19T01:05:36.917 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.917 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.918 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:36.918 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.669146+0000", 2024-04-19T01:05:36.918 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6399e-05 2024-04-19T01:05:36.918 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.918 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.918 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:36.918 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.702705+0000", 2024-04-19T01:05:36.918 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.033558534000000001 2024-04-19T01:05:36.918 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.918 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.918 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:36.918 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.702962+0000", 2024-04-19T01:05:36.918 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00025750700000000003 2024-04-19T01:05:36.918 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.919 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.919 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:36.919 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:50.708004+0000", 2024-04-19T01:05:36.919 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0050419640000000003 2024-04-19T01:05:36.919 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:36.919 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:36.919 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:36.919 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.919 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.919 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.es6 86/31 ECSubWrite(tid=195, reqid=client.4604.0:653, at_version=86'39, trim_to=0'0, roll_forward_to=84'38))", 2024-04-19T01:05:36.919 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:04:56.572643+0000", 2024-04-19T01:05:36.919 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 40.244664536000002, 2024-04-19T01:05:36.919 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.035539842000000002, 2024-04-19T01:05:36.919 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:36.919 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:36.920 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:36.920 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:36.920 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.920 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:36.920 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.572643+0000", 2024-04-19T01:05:36.920 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:36.920 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.920 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.920 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:36.920 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.572643+0000", 2024-04-19T01:05:36.920 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:36.920 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.921 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.921 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:36.921 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.572644+0000", 2024-04-19T01:05:36.921 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8999999999999996e-07 2024-04-19T01:05:36.921 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.921 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.921 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:36.921 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.573792+0000", 2024-04-19T01:05:36.921 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011481810000000001 2024-04-19T01:05:36.921 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.921 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.921 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:36.921 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.573818+0000", 2024-04-19T01:05:36.921 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5814000000000001e-05 2024-04-19T01:05:36.922 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.922 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.922 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:36.922 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.573832+0000", 2024-04-19T01:05:36.922 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4056999999999999e-05 2024-04-19T01:05:36.922 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.922 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.922 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:36.922 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.573964+0000", 2024-04-19T01:05:36.922 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013278399999999999 2024-04-19T01:05:36.922 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.922 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.922 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:36.922 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.573992+0000", 2024-04-19T01:05:36.923 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7250000000000002e-05 2024-04-19T01:05:36.923 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.923 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.923 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:36.923 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.608115+0000", 2024-04-19T01:05:36.923 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.034123819999999999 2024-04-19T01:05:36.923 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.923 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.923 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:36.923 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:04:56.608183+0000", 2024-04-19T01:05:36.923 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7346000000000002e-05 2024-04-19T01:05:36.923 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:36.923 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:36.923 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:36.923 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.924 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.924 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:745 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [append 0~573440] snapc 25=[25,24,22,21,1f] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-04-19T01:05:36.924 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:12.601513+0000", 2024-04-19T01:05:36.924 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 24.215794869, 2024-04-19T01:05:36.924 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.026388179000000001, 2024-04-19T01:05:36.924 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:36.924 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:36.924 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:36.925 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:36.925 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:36.925 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:36.925 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 745 2024-04-19T01:05:36.925 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.925 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:36.925 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.925 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:36.925 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.601513+0000", 2024-04-19T01:05:36.925 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:36.925 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.925 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.925 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:36.925 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.601513+0000", 2024-04-19T01:05:36.926 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:36.926 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.926 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.926 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:36.926 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.601516+0000", 2024-04-19T01:05:36.926 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6370000000000001e-06 2024-04-19T01:05:36.926 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.926 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.927 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:36.927 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.602170+0000", 2024-04-19T01:05:36.927 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00065339399999999996 2024-04-19T01:05:36.927 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.927 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.927 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:36.928 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.602201+0000", 2024-04-19T01:05:36.928 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1356000000000003e-05 2024-04-19T01:05:36.928 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.928 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.928 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:36.929 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.602225+0000", 2024-04-19T01:05:36.929 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4348000000000001e-05 2024-04-19T01:05:36.929 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.929 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.929 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:36.929 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.602333+0000", 2024-04-19T01:05:36.929 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000108137 2024-04-19T01:05:36.929 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.929 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.929 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:36.930 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.602579+0000", 2024-04-19T01:05:36.930 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00024595500000000001 2024-04-19T01:05:36.930 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.930 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.930 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:36.931 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.604232+0000", 2024-04-19T01:05:36.931 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0016524490000000001 2024-04-19T01:05:36.931 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.931 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.931 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:36.931 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.627421+0000", 2024-04-19T01:05:36.932 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.023188937 2024-04-19T01:05:36.932 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.932 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.932 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:36.932 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.627761+0000", 2024-04-19T01:05:36.932 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00034045499999999998 2024-04-19T01:05:36.933 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.933 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.933 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:36.933 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.627901+0000", 2024-04-19T01:05:36.933 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013951100000000001 2024-04-19T01:05:36.933 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:36.933 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:36.933 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:36.933 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.933 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.933 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:746 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [append 0~622592] snapc 25=[25,24,22,21,1f] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-04-19T01:05:36.933 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:12.610368+0000", 2024-04-19T01:05:36.933 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 24.206939438999999, 2024-04-19T01:05:36.934 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.034973300999999998, 2024-04-19T01:05:36.934 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:36.934 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:36.934 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:36.934 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:36.934 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:36.934 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:36.934 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 746 2024-04-19T01:05:36.934 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.934 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:36.934 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.934 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:36.934 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.610368+0000", 2024-04-19T01:05:36.934 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:36.935 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.935 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.935 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:36.935 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.610368+0000", 2024-04-19T01:05:36.935 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:36.935 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.935 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.935 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:36.935 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.610371+0000", 2024-04-19T01:05:36.935 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2830000000000002e-06 2024-04-19T01:05:36.935 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.935 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.935 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:36.936 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.611190+0000", 2024-04-19T01:05:36.936 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000818796 2024-04-19T01:05:36.936 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.936 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.936 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:36.936 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.611225+0000", 2024-04-19T01:05:36.936 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5008000000000001e-05 2024-04-19T01:05:36.936 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.936 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.936 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:36.936 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.611245+0000", 2024-04-19T01:05:36.936 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0231e-05 2024-04-19T01:05:36.936 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.936 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.936 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:36.937 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.614527+0000", 2024-04-19T01:05:36.937 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0032817599999999999 2024-04-19T01:05:36.937 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.937 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.937 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:36.937 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.614806+0000", 2024-04-19T01:05:36.937 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00027925500000000001 2024-04-19T01:05:36.937 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.937 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.937 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:36.937 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.616838+0000", 2024-04-19T01:05:36.937 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.002031167 2024-04-19T01:05:36.937 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.937 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.937 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:36.938 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.628169+0000", 2024-04-19T01:05:36.938 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.011331831000000001 2024-04-19T01:05:36.938 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.938 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.938 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:36.938 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.645187+0000", 2024-04-19T01:05:36.938 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.017017498999999998 2024-04-19T01:05:36.938 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.938 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.938 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:36.938 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.645341+0000", 2024-04-19T01:05:36.938 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015447099999999999 2024-04-19T01:05:36.938 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:36.938 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:36.939 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:36.939 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.939 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.939 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:747 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [append 0~622592] snapc 25=[25,24,22,21,1f] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-04-19T01:05:36.939 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:12.618378+0000", 2024-04-19T01:05:36.939 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 24.198929505999999, 2024-04-19T01:05:36.939 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.031200483000000001, 2024-04-19T01:05:36.939 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:36.939 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:36.939 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:36.939 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:36.939 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:36.939 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:36.939 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 747 2024-04-19T01:05:36.940 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.940 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:36.940 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.940 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:36.940 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.618378+0000", 2024-04-19T01:05:36.940 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:36.940 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.940 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.940 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:36.940 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.618378+0000", 2024-04-19T01:05:36.940 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:36.940 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.940 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.940 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:36.940 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.618380+0000", 2024-04-19T01:05:36.941 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4779999999999998e-06 2024-04-19T01:05:36.941 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.941 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.941 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:36.941 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.619044+0000", 2024-04-19T01:05:36.941 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00066315199999999997 2024-04-19T01:05:36.941 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.941 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.941 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:36.941 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.619073+0000", 2024-04-19T01:05:36.941 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9363e-05 2024-04-19T01:05:36.941 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.941 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.941 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:36.942 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.619088+0000", 2024-04-19T01:05:36.942 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5526e-05 2024-04-19T01:05:36.942 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.942 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.942 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:36.942 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.628378+0000", 2024-04-19T01:05:36.942 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0092899550000000008 2024-04-19T01:05:36.942 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.942 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.942 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:36.942 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.628555+0000", 2024-04-19T01:05:36.942 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017693299999999999 2024-04-19T01:05:36.942 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.942 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.942 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:36.943 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.630031+0000", 2024-04-19T01:05:36.943 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0014759720000000001 2024-04-19T01:05:36.943 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.943 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.943 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:36.943 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.645501+0000", 2024-04-19T01:05:36.943 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.015469505999999999 2024-04-19T01:05:36.943 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.943 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.943 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:36.943 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.649469+0000", 2024-04-19T01:05:36.943 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.003968297 2024-04-19T01:05:36.943 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.943 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.944 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:36.944 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.649578+0000", 2024-04-19T01:05:36.944 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000109301 2024-04-19T01:05:36.944 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:36.944 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:36.944 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:36.944 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.944 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.944 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:748 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [append 0~311296] snapc 25=[25,24,22,21,1f] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-04-19T01:05:36.944 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:12.621710+0000", 2024-04-19T01:05:36.944 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 24.195597329999998, 2024-04-19T01:05:36.944 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.02824397, 2024-04-19T01:05:36.944 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:36.944 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:36.945 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:36.945 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:36.945 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:36.945 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:36.945 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 748 2024-04-19T01:05:36.945 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.945 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:36.945 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.945 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:36.945 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.621710+0000", 2024-04-19T01:05:36.945 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:36.945 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.945 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.946 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:36.946 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.621710+0000", 2024-04-19T01:05:36.946 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:36.946 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.946 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.946 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:36.946 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.621711+0000", 2024-04-19T01:05:36.946 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3319999999999999e-06 2024-04-19T01:05:36.946 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.946 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.946 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:36.946 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.622069+0000", 2024-04-19T01:05:36.946 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00035724299999999998 2024-04-19T01:05:36.946 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.947 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.947 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:36.947 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.622094+0000", 2024-04-19T01:05:36.947 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5510000000000001e-05 2024-04-19T01:05:36.947 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.947 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.947 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:36.947 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.622107+0000", 2024-04-19T01:05:36.947 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2989e-05 2024-04-19T01:05:36.947 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.947 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.947 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:36.947 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.638128+0000", 2024-04-19T01:05:36.947 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.016020319000000002 2024-04-19T01:05:36.948 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.948 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.948 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:36.948 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.638423+0000", 2024-04-19T01:05:36.948 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000295006 2024-04-19T01:05:36.948 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.948 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.948 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:36.950 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.639758+0000", 2024-04-19T01:05:36.950 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0013350949999999999 2024-04-19T01:05:36.950 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.950 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.950 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:36.950 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.645682+0000", 2024-04-19T01:05:36.950 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0059239319999999998 2024-04-19T01:05:36.950 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.951 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.951 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:36.951 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.649864+0000", 2024-04-19T01:05:36.951 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.004182346 2024-04-19T01:05:36.951 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.951 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.952 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:36.952 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.649954+0000", 2024-04-19T01:05:36.952 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.0198000000000006e-05 2024-04-19T01:05:36.952 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:36.952 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:36.952 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:36.952 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.952 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.952 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:749 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [setxattr _header (57) in=7b] snapc 25=[25,24,22,21,1f] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-04-19T01:05:36.952 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:12.622198+0000", 2024-04-19T01:05:36.952 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 24.195109753000001, 2024-04-19T01:05:36.952 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.030651616, 2024-04-19T01:05:36.952 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:36.952 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:36.952 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T01:05:36.953 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:36.953 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:36.953 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:36.953 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 749 2024-04-19T01:05:36.953 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.953 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:36.953 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.953 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:36.953 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.622198+0000", 2024-04-19T01:05:36.953 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:36.953 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.953 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.953 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:36.953 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.622198+0000", 2024-04-19T01:05:36.953 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:36.954 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.954 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.954 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:36.954 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.622199+0000", 2024-04-19T01:05:36.954 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2800000000000005e-07 2024-04-19T01:05:36.954 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.954 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.954 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:36.954 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.622207+0000", 2024-04-19T01:05:36.954 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.0420000000000003e-06 2024-04-19T01:05:36.954 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.954 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.954 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:36.954 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.622220+0000", 2024-04-19T01:05:36.955 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3709e-05 2024-04-19T01:05:36.955 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.955 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.955 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:36.955 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.622229+0000", 2024-04-19T01:05:36.955 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.6200000000000005e-06 2024-04-19T01:05:36.955 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.955 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.955 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:36.955 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.646439+0000", 2024-04-19T01:05:36.955 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.024210084 2024-04-19T01:05:36.955 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.955 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.955 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:36.955 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.646693+0000", 2024-04-19T01:05:36.956 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00025368100000000001 2024-04-19T01:05:36.956 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.956 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.956 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:36.956 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.647772+0000", 2024-04-19T01:05:36.956 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0010789479999999999 2024-04-19T01:05:36.956 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.956 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.956 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:36.956 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.648875+0000", 2024-04-19T01:05:36.956 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011036139999999999 2024-04-19T01:05:36.956 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.956 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.956 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-04-19T01:05:36.956 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.652743+0000", 2024-04-19T01:05:36.957 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0038673510000000002 2024-04-19T01:05:36.957 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.958 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.958 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:36.958 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.652849+0000", 2024-04-19T01:05:36.959 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010663899999999999 2024-04-19T01:05:36.959 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:36.959 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:36.959 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:36.959 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.959 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.959 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4604.0:750 3.3s0 3:cbbd4e2f:::smithi15528762-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e102)", 2024-04-19T01:05:36.959 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:12.622276+0000", 2024-04-19T01:05:36.959 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 24.195031299, 2024-04-19T01:05:36.959 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.033252480000000001, 2024-04-19T01:05:36.959 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:36.959 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:36.959 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-04-19T01:05:36.959 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-04-19T01:05:36.960 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4604", 2024-04-19T01:05:36.960 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.155:0/3169916025", 2024-04-19T01:05:36.960 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 750 2024-04-19T01:05:36.960 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.960 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:36.960 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.960 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:36.960 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.622276+0000", 2024-04-19T01:05:36.960 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:36.960 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.960 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.960 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:36.960 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.622276+0000", 2024-04-19T01:05:36.960 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:36.960 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.961 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.961 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:36.961 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.622277+0000", 2024-04-19T01:05:36.961 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7000000000000004e-07 2024-04-19T01:05:36.961 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.961 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.961 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:36.961 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.622280+0000", 2024-04-19T01:05:36.961 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5640000000000001e-06 2024-04-19T01:05:36.962 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.962 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.962 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:36.962 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.622294+0000", 2024-04-19T01:05:36.963 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3278000000000001e-05 2024-04-19T01:05:36.963 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.963 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.963 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:36.963 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.622302+0000", 2024-04-19T01:05:36.963 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.3760000000000008e-06 2024-04-19T01:05:36.963 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.963 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.963 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:36.963 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.648257+0000", 2024-04-19T01:05:36.963 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.025954481000000001 2024-04-19T01:05:36.963 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.964 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.964 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-04-19T01:05:36.964 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.648468+0000", 2024-04-19T01:05:36.964 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021143900000000001 2024-04-19T01:05:36.964 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.964 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.964 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:36.964 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.653397+0000", 2024-04-19T01:05:36.964 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0049289210000000002 2024-04-19T01:05:36.964 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.964 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.964 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-04-19T01:05:36.964 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.653608+0000", 2024-04-19T01:05:36.964 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021074700000000001 2024-04-19T01:05:36.964 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.965 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.965 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:36.965 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:12.655529+0000", 2024-04-19T01:05:36.965 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001921004 2024-04-19T01:05:36.965 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:36.965 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:36.965 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:36.965 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.965 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.965 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.4s4 107/31 ECSubWrite(tid=196, reqid=client.4604.0:819, at_version=107'87, trim_to=0'0, roll_forward_to=102'85))", 2024-04-19T01:05:36.965 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-04-19T01:05:17.628492+0000", 2024-04-19T01:05:36.965 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 19.188815942000002, 2024-04-19T01:05:36.965 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.033676134000000003, 2024-04-19T01:05:36.965 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-04-19T01:05:36.965 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-04-19T01:05:36.966 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", 2024-04-19T01:05:36.966 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-04-19T01:05:36.966 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.966 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-04-19T01:05:36.966 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.628492+0000", 2024-04-19T01:05:36.966 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:36.966 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.966 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.966 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-04-19T01:05:36.966 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.628492+0000", 2024-04-19T01:05:36.966 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-04-19T01:05:36.966 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.966 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.966 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-04-19T01:05:36.967 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.628492+0000", 2024-04-19T01:05:36.967 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9700000000000003e-07 2024-04-19T01:05:36.967 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.967 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.967 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-04-19T01:05:36.967 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.629345+0000", 2024-04-19T01:05:36.967 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00085334600000000005 2024-04-19T01:05:36.967 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.967 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.967 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-04-19T01:05:36.967 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.629371+0000", 2024-04-19T01:05:36.967 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6084999999999998e-05 2024-04-19T01:05:36.967 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.967 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.967 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-04-19T01:05:36.968 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.629388+0000", 2024-04-19T01:05:36.968 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6733000000000001e-05 2024-04-19T01:05:36.968 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.968 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.968 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-04-19T01:05:36.968 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.629572+0000", 2024-04-19T01:05:36.968 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00018418500000000001 2024-04-19T01:05:36.968 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.968 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.968 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", 2024-04-19T01:05:36.968 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.629604+0000", 2024-04-19T01:05:36.968 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1794000000000001e-05 2024-04-19T01:05:36.968 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.968 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.968 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", 2024-04-19T01:05:36.969 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.662101+0000", 2024-04-19T01:05:36.969 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.032496628999999999 2024-04-19T01:05:36.969 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-04-19T01:05:36.969 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-04-19T01:05:36.969 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-04-19T01:05:36.969 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-04-19T01:05:17.662168+0000", 2024-04-19T01:05:36.969 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7064999999999996e-05 2024-04-19T01:05:36.969 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:36.969 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:36.969 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:36.969 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-04-19T01:05:36.969 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-19T01:05:36.969 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-19T01:05:37.705 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-19T01:05:38.693 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2024-04-19T01:05:38.719 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-19T01:05:39.707 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2024-04-19T01:05:40.705 INFO:tasks.thrashosds.thrasher:inning osd 2024-04-19T01:05:40.705 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-04-19T01:05:40.706 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 0 2024-04-19T01:05:41.036 INFO:tasks.ceph.mon.a.smithi033.stderr:2024-04-19T01:05:41.032+0000 7f9b08596640 -1 mon.a@0(leader).osd e128 definitely_dead 0 2024-04-19T01:05:41.717 INFO:teuthology.orchestra.run.smithi033.stderr:marked in osd.0. 2024-04-19T01:05:41.742 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-04-19T01:05:41.743 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-04-19T01:05:41.887 INFO:teuthology.orchestra.run.smithi033.stdout:mon.a: {} 2024-04-19T01:05:41.887 INFO:teuthology.orchestra.run.smithi033.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-19T01:05:41.900 INFO:teuthology.orchestra.run.smithi033.stdout:mon.b: {} 2024-04-19T01:05:41.901 INFO:teuthology.orchestra.run.smithi033.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-19T01:05:41.915 INFO:teuthology.orchestra.run.smithi033.stdout:mon.c: {} 2024-04-19T01:05:41.915 INFO:teuthology.orchestra.run.smithi033.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-19T01:05:41.932 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-19T01:05:42.721 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.0 to 1 (10000) 2024-04-19T01:05:42.746 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-04-19T01:05:43.737 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-04-19T01:05:43.763 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-04-19T01:05:44.746 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.4 to 1 (10000) 2024-04-19T01:05:44.770 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-04-19T01:05:45.749 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-04-19T01:05:45.776 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-04-19T01:05:46.759 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.8 to 1 (10000) 2024-04-19T01:05:46.789 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 8 1 2024-04-19T01:05:47.769 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.8 primary-affinity to 1 (8655362) 2024-04-19T01:05:47.795 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 12 1 2024-04-19T01:05:48.780 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.12 to 1 (10000) 2024-04-19T01:05:48.804 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 12 1 2024-04-19T01:05:49.788 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.12 primary-affinity to 1 (8655362) 2024-04-19T01:05:49.819 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-19T01:05:50.798 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.1 to 1 (10000) 2024-04-19T01:05:50.823 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-04-19T01:05:51.813 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-04-19T01:05:51.839 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-04-19T01:05:52.816 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.5 to 1 (10000) 2024-04-19T01:05:52.843 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-04-19T01:05:53.825 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-04-19T01:05:53.853 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-04-19T01:05:54.835 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.9 to 1 (10000) 2024-04-19T01:05:54.861 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 9 1 2024-04-19T01:05:55.844 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.9 primary-affinity to 1 (8655362) 2024-04-19T01:05:55.872 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 13 1 2024-04-19T01:05:56.852 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.13 to 1 (10000) 2024-04-19T01:05:56.880 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 13 1 2024-04-19T01:05:57.862 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.13 primary-affinity to 1 (8655362) 2024-04-19T01:05:57.887 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-19T01:05:58.878 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.2 to 1 (10000) 2024-04-19T01:05:58.904 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-04-19T01:05:59.880 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-04-19T01:05:59.908 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-04-19T01:06:00.896 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.6 to 1 (10000) 2024-04-19T01:06:00.926 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-04-19T01:06:01.899 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-04-19T01:06:01.926 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-04-19T01:06:02.919 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.10 to 1 (10000) 2024-04-19T01:06:02.943 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 10 1 2024-04-19T01:06:03.922 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.10 primary-affinity to 1 (8655362) 2024-04-19T01:06:03.947 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 14 1 2024-04-19T01:06:04.935 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.14 to 1 (10000) 2024-04-19T01:06:04.962 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 14 1 2024-04-19T01:06:05.949 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.14 primary-affinity to 1 (8655362) 2024-04-19T01:06:05.975 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-19T01:06:06.952 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.3 to 1 (10000) 2024-04-19T01:06:06.979 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-04-19T01:06:07.961 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-04-19T01:06:07.987 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-04-19T01:06:08.971 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.7 to 1 (10000) 2024-04-19T01:06:09.001 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-04-19T01:06:09.981 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-04-19T01:06:10.008 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-04-19T01:06:10.990 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.11 to 1 (10000) 2024-04-19T01:06:11.015 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 11 1 2024-04-19T01:06:12.001 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.11 primary-affinity to 1 (8655362) 2024-04-19T01:06:12.027 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 15 1 2024-04-19T01:06:13.016 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.15 to 1 (10000) 2024-04-19T01:06:13.040 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 15 1 2024-04-19T01:06:14.018 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.15 primary-affinity to 1 (8655362) 2024-04-19T01:06:14.051 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-04-19T01:06:14.051 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-04-19T01:06:14.051 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-04-19T01:06:14.051 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-04-19T01:06:14.051 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-19T01:06:14.051 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-19T01:06:14.363 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:06:14.363 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":161,"fsid":"f11bd2d5-4456-4ffa-83c0-722dddd22ad3","created":"2024-04-19T01:03:36.766089+0000","modified":"2024-04-19T01:06:14.016677+0000","last_up_change":"2024-04-19T01:03:50.591452+0000","last_in_change":"2024-04-19T01:05:41.710130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-19T01:03:49.571609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-19T01:03:51.913782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.0199999809265137,"score_stable":6.0199999809265137,"optimal_score":0.81000000238418579,"raw_score_acting":4.880000114440918,"raw_score_stable":4.880000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"296899fd-2b2b-4071-8b1f-0f92c3d0bbc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":1633616806},{"type":"v1","addr":"172.21.15.33:6825","nonce":1633616806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":1633616806},{"type":"v1","addr":"172.21.15.33:6827","nonce":1633616806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":1633616806},{"type":"v1","addr":"172.21.15.33:6831","nonce":1633616806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":1633616806},{"type":"v1","addr":"172.21.15.33:6829","nonce":1633616806}]},"public_addr":"172.21.15.33:6825/1633616806","cluster_addr":"172.21.15.33:6827/1633616806","heartbeat_back_addr":"172.21.15.33:6831/1633616806","heartbeat_front_addr":"172.21.15.33:6829/1633616806","state":["exists","up"]},{"osd":1,"uuid":"021f7190-0ec3-4553-a44f-15095765cccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":4270148213},{"type":"v1","addr":"172.21.15.110:6808","nonce":4270148213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6811","nonce":4270148213},{"type":"v1","addr":"172.21.15.110:6814","nonce":4270148213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":4270148213},{"type":"v1","addr":"172.21.15.110:6823","nonce":4270148213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6817","nonce":4270148213},{"type":"v1","addr":"172.21.15.110:6820","nonce":4270148213}]},"public_addr":"172.21.15.110:6808/4270148213","cluster_addr":"172.21.15.110:6814/4270148213","heartbeat_back_addr":"172.21.15.110:6823/4270148213","heartbeat_front_addr":"172.21.15.110:6820/4270148213","state":["exists","up"]},{"osd":2,"uuid":"c283d0bc-5e34-4763-9019-328ee9b6c07a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":13248188},{"type":"v1","addr":"172.21.15.118:6809","nonce":13248188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":13248188},{"type":"v1","addr":"172.21.15.118:6811","nonce":13248188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":13248188},{"type":"v1","addr":"172.21.15.118:6815","nonce":13248188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":13248188},{"type":"v1","addr":"172.21.15.118:6813","nonce":13248188}]},"public_addr":"172.21.15.118:6809/13248188","cluster_addr":"172.21.15.118:6811/13248188","heartbeat_back_addr":"172.21.15.118:6815/13248188","heartbeat_front_addr":"172.21.15.118:6813/13248188","state":["exists","up"]},{"osd":3,"uuid":"d45888f2-eecd-480b-80f5-c0731a06ff3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":2658385904},{"type":"v1","addr":"172.21.15.155:6803","nonce":2658385904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":2658385904},{"type":"v1","addr":"172.21.15.155:6805","nonce":2658385904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6813","nonce":2658385904},{"type":"v1","addr":"172.21.15.155:6816","nonce":2658385904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":2658385904},{"type":"v1","addr":"172.21.15.155:6809","nonce":2658385904}]},"public_addr":"172.21.15.155:6803/2658385904","cluster_addr":"172.21.15.155:6805/2658385904","heartbeat_back_addr":"172.21.15.155:6816/2658385904","heartbeat_front_addr":"172.21.15.155:6809/2658385904","state":["exists","up"]},{"osd":4,"uuid":"d060d939-8c94-4a07-9fd5-42af0044a16f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":2389566155},{"type":"v1","addr":"172.21.15.33:6801","nonce":2389566155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":2389566155},{"type":"v1","addr":"172.21.15.33:6803","nonce":2389566155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":2389566155},{"type":"v1","addr":"172.21.15.33:6807","nonce":2389566155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":2389566155},{"type":"v1","addr":"172.21.15.33:6805","nonce":2389566155}]},"public_addr":"172.21.15.33:6801/2389566155","cluster_addr":"172.21.15.33:6803/2389566155","heartbeat_back_addr":"172.21.15.33:6807/2389566155","heartbeat_front_addr":"172.21.15.33:6805/2389566155","state":["exists","up"]},{"osd":5,"uuid":"9700d2f7-ed56-4143-ba9d-6cebe524e877","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":2821892844},{"type":"v1","addr":"172.21.15.110:6825","nonce":2821892844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":2821892844},{"type":"v1","addr":"172.21.15.110:6827","nonce":2821892844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":2821892844},{"type":"v1","addr":"172.21.15.110:6831","nonce":2821892844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":2821892844},{"type":"v1","addr":"172.21.15.110:6829","nonce":2821892844}]},"public_addr":"172.21.15.110:6825/2821892844","cluster_addr":"172.21.15.110:6827/2821892844","heartbeat_back_addr":"172.21.15.110:6831/2821892844","heartbeat_front_addr":"172.21.15.110:6829/2821892844","state":["exists","up"]},{"osd":6,"uuid":"49324695-d83b-41ad-92e1-5348e8e9149f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":273776428},{"type":"v1","addr":"172.21.15.118:6818","nonce":273776428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":273776428},{"type":"v1","addr":"172.21.15.118:6822","nonce":273776428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":273776428},{"type":"v1","addr":"172.21.15.118:6830","nonce":273776428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":273776428},{"type":"v1","addr":"172.21.15.118:6826","nonce":273776428}]},"public_addr":"172.21.15.118:6818/273776428","cluster_addr":"172.21.15.118:6822/273776428","heartbeat_back_addr":"172.21.15.118:6830/273776428","heartbeat_front_addr":"172.21.15.118:6826/273776428","state":["exists","up"]},{"osd":7,"uuid":"afef160d-0911-4406-ad49-cd6d61ee8a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1006872173},{"type":"v1","addr":"172.21.15.155:6827","nonce":1006872173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1006872173},{"type":"v1","addr":"172.21.15.155:6829","nonce":1006872173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1006872173},{"type":"v1","addr":"172.21.15.155:6833","nonce":1006872173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1006872173},{"type":"v1","addr":"172.21.15.155:6831","nonce":1006872173}]},"public_addr":"172.21.15.155:6827/1006872173","cluster_addr":"172.21.15.155:6829/1006872173","heartbeat_back_addr":"172.21.15.155:6833/1006872173","heartbeat_front_addr":"172.21.15.155:6831/1006872173","state":["exists","up"]},{"osd":8,"uuid":"298396bf-31a2-4832-91a4-55fcb050130f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":744651323},{"type":"v1","addr":"172.21.15.33:6817","nonce":744651323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":744651323},{"type":"v1","addr":"172.21.15.33:6819","nonce":744651323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":744651323},{"type":"v1","addr":"172.21.15.33:6823","nonce":744651323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":744651323},{"type":"v1","addr":"172.21.15.33:6821","nonce":744651323}]},"public_addr":"172.21.15.33:6817/744651323","cluster_addr":"172.21.15.33:6819/744651323","heartbeat_back_addr":"172.21.15.33:6823/744651323","heartbeat_front_addr":"172.21.15.33:6821/744651323","state":["exists","up"]},{"osd":9,"uuid":"c151ae6b-279d-4174-aceb-4671fbdac3b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":872519654},{"type":"v1","addr":"172.21.15.110:6802","nonce":872519654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":872519654},{"type":"v1","addr":"172.21.15.110:6807","nonce":872519654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":872519654},{"type":"v1","addr":"172.21.15.110:6819","nonce":872519654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":872519654},{"type":"v1","addr":"172.21.15.110:6813","nonce":872519654}]},"public_addr":"172.21.15.110:6802/872519654","cluster_addr":"172.21.15.110:6807/872519654","heartbeat_back_addr":"172.21.15.110:6819/872519654","heartbeat_front_addr":"172.21.15.110:6813/872519654","state":["exists","up"]},{"osd":10,"uuid":"0b292dfb-2029-4e1e-b206-ea7e0f6969ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6817","nonce":4099141917},{"type":"v1","addr":"172.21.15.118:6819","nonce":4099141917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6821","nonce":4099141917},{"type":"v1","addr":"172.21.15.118:6823","nonce":4099141917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6829","nonce":4099141917},{"type":"v1","addr":"172.21.15.118:6831","nonce":4099141917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":4099141917},{"type":"v1","addr":"172.21.15.118:6827","nonce":4099141917}]},"public_addr":"172.21.15.118:6819/4099141917","cluster_addr":"172.21.15.118:6823/4099141917","heartbeat_back_addr":"172.21.15.118:6831/4099141917","heartbeat_front_addr":"172.21.15.118:6827/4099141917","state":["exists","up"]},{"osd":11,"uuid":"10c61619-ad11-40f0-b013-9ef52bcabf59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":3981951559},{"type":"v1","addr":"172.21.15.155:6811","nonce":3981951559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":3981951559},{"type":"v1","addr":"172.21.15.155:6817","nonce":3981951559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":3981951559},{"type":"v1","addr":"172.21.15.155:6825","nonce":3981951559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3981951559},{"type":"v1","addr":"172.21.15.155:6823","nonce":3981951559}]},"public_addr":"172.21.15.155:6811/3981951559","cluster_addr":"172.21.15.155:6817/3981951559","heartbeat_back_addr":"172.21.15.155:6825/3981951559","heartbeat_front_addr":"172.21.15.155:6823/3981951559","state":["exists","up"]},{"osd":12,"uuid":"3f0d26e5-f3e2-47b6-91a8-0daa30d0544f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":4133738119},{"type":"v1","addr":"172.21.15.33:6809","nonce":4133738119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":4133738119},{"type":"v1","addr":"172.21.15.33:6811","nonce":4133738119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":4133738119},{"type":"v1","addr":"172.21.15.33:6815","nonce":4133738119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":4133738119},{"type":"v1","addr":"172.21.15.33:6813","nonce":4133738119}]},"public_addr":"172.21.15.33:6809/4133738119","cluster_addr":"172.21.15.33:6811/4133738119","heartbeat_back_addr":"172.21.15.33:6815/4133738119","heartbeat_front_addr":"172.21.15.33:6813/4133738119","state":["exists","up"]},{"osd":13,"uuid":"c8d6213f-af5e-4646-9241-6f91589b0e1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":1583232449},{"type":"v1","addr":"172.21.15.110:6803","nonce":1583232449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":1583232449},{"type":"v1","addr":"172.21.15.110:6809","nonce":1583232449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":1583232449},{"type":"v1","addr":"172.21.15.110:6821","nonce":1583232449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":1583232449},{"type":"v1","addr":"172.21.15.110:6815","nonce":1583232449}]},"public_addr":"172.21.15.110:6803/1583232449","cluster_addr":"172.21.15.110:6809/1583232449","heartbeat_back_addr":"172.21.15.110:6821/1583232449","heartbeat_front_addr":"172.21.15.110:6815/1583232449","state":["exists","up"]},{"osd":14,"uuid":"7b3d56df-62bf-4877-b984-6f17fe1fab33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":506416463},{"type":"v1","addr":"172.21.15.118:6801","nonce":506416463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":506416463},{"type":"v1","addr":"172.21.15.118:6803","nonce":506416463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":506416463},{"type":"v1","addr":"172.21.15.118:6807","nonce":506416463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":506416463},{"type":"v1","addr":"172.21.15.118:6805","nonce":506416463}]},"public_addr":"172.21.15.118:6801/506416463","cluster_addr":"172.21.15.118:6803/506416463","heartbeat_back_addr":"172.21.15.118:6807/506416463","heartbeat_front_addr":"172.21.15.118:6805/506416463","state":["exists","up"]},{"osd":15,"uuid":"6d526edb-f1d8-415a-ad55-81e810ff4944","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6807","nonce":741132602},{"type":"v1","addr":"172.21.15.155:6810","nonce":741132602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":741132602},{"type":"v1","addr":"172.21.15.155:6815","nonce":741132602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6821","nonce":741132602},{"type":"v1","addr":"172.21.15.155:6822","nonce":741132602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":741132602},{"type":"v1","addr":"172.21.15.155:6819","nonce":741132602}]},"public_addr":"172.21.15.155:6810/741132602","cluster_addr":"172.21.15.155:6815/741132602","heartbeat_back_addr":"172.21.15.155:6822/741132602","heartbeat_front_addr":"172.21.15.155:6819/741132602","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:05:00.394086+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:06:07.843641+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:06:06.531265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:05:14.387087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:05:07.754528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:05:03.768853+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:05:58.104022+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:05:03.346710+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:06:01.987377+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:06:12.890542+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:06:12.330281+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:06:10.632141+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:06:05.930654+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:05:59.307943+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:06:08.128740+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:05:06.403111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":36,"length":1},{"begin":39,"length":1},{"begin":41,"length":1},{"begin":43,"length":1},{"begin":46,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T01:06:14.382 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-19T01:06:14.383 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-19T01:06:14.692 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:06:14.693 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":161,"fsid":"f11bd2d5-4456-4ffa-83c0-722dddd22ad3","created":"2024-04-19T01:03:36.766089+0000","modified":"2024-04-19T01:06:14.016677+0000","last_up_change":"2024-04-19T01:03:50.591452+0000","last_in_change":"2024-04-19T01:05:41.710130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-19T01:03:49.571609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-19T01:03:51.913782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.0199999809265137,"score_stable":6.0199999809265137,"optimal_score":0.81000000238418579,"raw_score_acting":4.880000114440918,"raw_score_stable":4.880000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"296899fd-2b2b-4071-8b1f-0f92c3d0bbc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":1633616806},{"type":"v1","addr":"172.21.15.33:6825","nonce":1633616806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":1633616806},{"type":"v1","addr":"172.21.15.33:6827","nonce":1633616806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":1633616806},{"type":"v1","addr":"172.21.15.33:6831","nonce":1633616806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":1633616806},{"type":"v1","addr":"172.21.15.33:6829","nonce":1633616806}]},"public_addr":"172.21.15.33:6825/1633616806","cluster_addr":"172.21.15.33:6827/1633616806","heartbeat_back_addr":"172.21.15.33:6831/1633616806","heartbeat_front_addr":"172.21.15.33:6829/1633616806","state":["exists","up"]},{"osd":1,"uuid":"021f7190-0ec3-4553-a44f-15095765cccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":4270148213},{"type":"v1","addr":"172.21.15.110:6808","nonce":4270148213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6811","nonce":4270148213},{"type":"v1","addr":"172.21.15.110:6814","nonce":4270148213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":4270148213},{"type":"v1","addr":"172.21.15.110:6823","nonce":4270148213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6817","nonce":4270148213},{"type":"v1","addr":"172.21.15.110:6820","nonce":4270148213}]},"public_addr":"172.21.15.110:6808/4270148213","cluster_addr":"172.21.15.110:6814/4270148213","heartbeat_back_addr":"172.21.15.110:6823/4270148213","heartbeat_front_addr":"172.21.15.110:6820/4270148213","state":["exists","up"]},{"osd":2,"uuid":"c283d0bc-5e34-4763-9019-328ee9b6c07a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":13248188},{"type":"v1","addr":"172.21.15.118:6809","nonce":13248188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":13248188},{"type":"v1","addr":"172.21.15.118:6811","nonce":13248188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":13248188},{"type":"v1","addr":"172.21.15.118:6815","nonce":13248188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":13248188},{"type":"v1","addr":"172.21.15.118:6813","nonce":13248188}]},"public_addr":"172.21.15.118:6809/13248188","cluster_addr":"172.21.15.118:6811/13248188","heartbeat_back_addr":"172.21.15.118:6815/13248188","heartbeat_front_addr":"172.21.15.118:6813/13248188","state":["exists","up"]},{"osd":3,"uuid":"d45888f2-eecd-480b-80f5-c0731a06ff3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":2658385904},{"type":"v1","addr":"172.21.15.155:6803","nonce":2658385904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":2658385904},{"type":"v1","addr":"172.21.15.155:6805","nonce":2658385904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6813","nonce":2658385904},{"type":"v1","addr":"172.21.15.155:6816","nonce":2658385904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":2658385904},{"type":"v1","addr":"172.21.15.155:6809","nonce":2658385904}]},"public_addr":"172.21.15.155:6803/2658385904","cluster_addr":"172.21.15.155:6805/2658385904","heartbeat_back_addr":"172.21.15.155:6816/2658385904","heartbeat_front_addr":"172.21.15.155:6809/2658385904","state":["exists","up"]},{"osd":4,"uuid":"d060d939-8c94-4a07-9fd5-42af0044a16f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":2389566155},{"type":"v1","addr":"172.21.15.33:6801","nonce":2389566155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":2389566155},{"type":"v1","addr":"172.21.15.33:6803","nonce":2389566155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":2389566155},{"type":"v1","addr":"172.21.15.33:6807","nonce":2389566155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":2389566155},{"type":"v1","addr":"172.21.15.33:6805","nonce":2389566155}]},"public_addr":"172.21.15.33:6801/2389566155","cluster_addr":"172.21.15.33:6803/2389566155","heartbeat_back_addr":"172.21.15.33:6807/2389566155","heartbeat_front_addr":"172.21.15.33:6805/2389566155","state":["exists","up"]},{"osd":5,"uuid":"9700d2f7-ed56-4143-ba9d-6cebe524e877","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":2821892844},{"type":"v1","addr":"172.21.15.110:6825","nonce":2821892844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":2821892844},{"type":"v1","addr":"172.21.15.110:6827","nonce":2821892844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":2821892844},{"type":"v1","addr":"172.21.15.110:6831","nonce":2821892844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":2821892844},{"type":"v1","addr":"172.21.15.110:6829","nonce":2821892844}]},"public_addr":"172.21.15.110:6825/2821892844","cluster_addr":"172.21.15.110:6827/2821892844","heartbeat_back_addr":"172.21.15.110:6831/2821892844","heartbeat_front_addr":"172.21.15.110:6829/2821892844","state":["exists","up"]},{"osd":6,"uuid":"49324695-d83b-41ad-92e1-5348e8e9149f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":273776428},{"type":"v1","addr":"172.21.15.118:6818","nonce":273776428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":273776428},{"type":"v1","addr":"172.21.15.118:6822","nonce":273776428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":273776428},{"type":"v1","addr":"172.21.15.118:6830","nonce":273776428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":273776428},{"type":"v1","addr":"172.21.15.118:6826","nonce":273776428}]},"public_addr":"172.21.15.118:6818/273776428","cluster_addr":"172.21.15.118:6822/273776428","heartbeat_back_addr":"172.21.15.118:6830/273776428","heartbeat_front_addr":"172.21.15.118:6826/273776428","state":["exists","up"]},{"osd":7,"uuid":"afef160d-0911-4406-ad49-cd6d61ee8a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1006872173},{"type":"v1","addr":"172.21.15.155:6827","nonce":1006872173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1006872173},{"type":"v1","addr":"172.21.15.155:6829","nonce":1006872173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1006872173},{"type":"v1","addr":"172.21.15.155:6833","nonce":1006872173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1006872173},{"type":"v1","addr":"172.21.15.155:6831","nonce":1006872173}]},"public_addr":"172.21.15.155:6827/1006872173","cluster_addr":"172.21.15.155:6829/1006872173","heartbeat_back_addr":"172.21.15.155:6833/1006872173","heartbeat_front_addr":"172.21.15.155:6831/1006872173","state":["exists","up"]},{"osd":8,"uuid":"298396bf-31a2-4832-91a4-55fcb050130f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":744651323},{"type":"v1","addr":"172.21.15.33:6817","nonce":744651323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":744651323},{"type":"v1","addr":"172.21.15.33:6819","nonce":744651323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":744651323},{"type":"v1","addr":"172.21.15.33:6823","nonce":744651323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":744651323},{"type":"v1","addr":"172.21.15.33:6821","nonce":744651323}]},"public_addr":"172.21.15.33:6817/744651323","cluster_addr":"172.21.15.33:6819/744651323","heartbeat_back_addr":"172.21.15.33:6823/744651323","heartbeat_front_addr":"172.21.15.33:6821/744651323","state":["exists","up"]},{"osd":9,"uuid":"c151ae6b-279d-4174-aceb-4671fbdac3b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":872519654},{"type":"v1","addr":"172.21.15.110:6802","nonce":872519654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":872519654},{"type":"v1","addr":"172.21.15.110:6807","nonce":872519654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":872519654},{"type":"v1","addr":"172.21.15.110:6819","nonce":872519654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":872519654},{"type":"v1","addr":"172.21.15.110:6813","nonce":872519654}]},"public_addr":"172.21.15.110:6802/872519654","cluster_addr":"172.21.15.110:6807/872519654","heartbeat_back_addr":"172.21.15.110:6819/872519654","heartbeat_front_addr":"172.21.15.110:6813/872519654","state":["exists","up"]},{"osd":10,"uuid":"0b292dfb-2029-4e1e-b206-ea7e0f6969ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6817","nonce":4099141917},{"type":"v1","addr":"172.21.15.118:6819","nonce":4099141917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6821","nonce":4099141917},{"type":"v1","addr":"172.21.15.118:6823","nonce":4099141917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6829","nonce":4099141917},{"type":"v1","addr":"172.21.15.118:6831","nonce":4099141917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":4099141917},{"type":"v1","addr":"172.21.15.118:6827","nonce":4099141917}]},"public_addr":"172.21.15.118:6819/4099141917","cluster_addr":"172.21.15.118:6823/4099141917","heartbeat_back_addr":"172.21.15.118:6831/4099141917","heartbeat_front_addr":"172.21.15.118:6827/4099141917","state":["exists","up"]},{"osd":11,"uuid":"10c61619-ad11-40f0-b013-9ef52bcabf59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":3981951559},{"type":"v1","addr":"172.21.15.155:6811","nonce":3981951559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":3981951559},{"type":"v1","addr":"172.21.15.155:6817","nonce":3981951559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":3981951559},{"type":"v1","addr":"172.21.15.155:6825","nonce":3981951559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3981951559},{"type":"v1","addr":"172.21.15.155:6823","nonce":3981951559}]},"public_addr":"172.21.15.155:6811/3981951559","cluster_addr":"172.21.15.155:6817/3981951559","heartbeat_back_addr":"172.21.15.155:6825/3981951559","heartbeat_front_addr":"172.21.15.155:6823/3981951559","state":["exists","up"]},{"osd":12,"uuid":"3f0d26e5-f3e2-47b6-91a8-0daa30d0544f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":4133738119},{"type":"v1","addr":"172.21.15.33:6809","nonce":4133738119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":4133738119},{"type":"v1","addr":"172.21.15.33:6811","nonce":4133738119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":4133738119},{"type":"v1","addr":"172.21.15.33:6815","nonce":4133738119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":4133738119},{"type":"v1","addr":"172.21.15.33:6813","nonce":4133738119}]},"public_addr":"172.21.15.33:6809/4133738119","cluster_addr":"172.21.15.33:6811/4133738119","heartbeat_back_addr":"172.21.15.33:6815/4133738119","heartbeat_front_addr":"172.21.15.33:6813/4133738119","state":["exists","up"]},{"osd":13,"uuid":"c8d6213f-af5e-4646-9241-6f91589b0e1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":1583232449},{"type":"v1","addr":"172.21.15.110:6803","nonce":1583232449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":1583232449},{"type":"v1","addr":"172.21.15.110:6809","nonce":1583232449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":1583232449},{"type":"v1","addr":"172.21.15.110:6821","nonce":1583232449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":1583232449},{"type":"v1","addr":"172.21.15.110:6815","nonce":1583232449}]},"public_addr":"172.21.15.110:6803/1583232449","cluster_addr":"172.21.15.110:6809/1583232449","heartbeat_back_addr":"172.21.15.110:6821/1583232449","heartbeat_front_addr":"172.21.15.110:6815/1583232449","state":["exists","up"]},{"osd":14,"uuid":"7b3d56df-62bf-4877-b984-6f17fe1fab33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":506416463},{"type":"v1","addr":"172.21.15.118:6801","nonce":506416463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":506416463},{"type":"v1","addr":"172.21.15.118:6803","nonce":506416463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":506416463},{"type":"v1","addr":"172.21.15.118:6807","nonce":506416463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":506416463},{"type":"v1","addr":"172.21.15.118:6805","nonce":506416463}]},"public_addr":"172.21.15.118:6801/506416463","cluster_addr":"172.21.15.118:6803/506416463","heartbeat_back_addr":"172.21.15.118:6807/506416463","heartbeat_front_addr":"172.21.15.118:6805/506416463","state":["exists","up"]},{"osd":15,"uuid":"6d526edb-f1d8-415a-ad55-81e810ff4944","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6807","nonce":741132602},{"type":"v1","addr":"172.21.15.155:6810","nonce":741132602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":741132602},{"type":"v1","addr":"172.21.15.155:6815","nonce":741132602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6821","nonce":741132602},{"type":"v1","addr":"172.21.15.155:6822","nonce":741132602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":741132602},{"type":"v1","addr":"172.21.15.155:6819","nonce":741132602}]},"public_addr":"172.21.15.155:6810/741132602","cluster_addr":"172.21.15.155:6815/741132602","heartbeat_back_addr":"172.21.15.155:6822/741132602","heartbeat_front_addr":"172.21.15.155:6819/741132602","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:05:00.394086+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:06:07.843641+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:06:06.531265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:05:14.387087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:05:07.754528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:05:03.768853+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:05:58.104022+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:05:03.346710+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:06:01.987377+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:06:12.890542+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:06:12.330281+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:06:10.632141+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:06:05.930654+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:05:59.307943+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:06:08.128740+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:05:06.403111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":36,"length":1},{"begin":39,"length":1},{"begin":41,"length":1},{"begin":43,"length":1},{"begin":46,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T01:06:14.713 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-19T01:06:14.713 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-19T01:06:14.714 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-19T01:06:14.714 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-19T01:06:14.714 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-19T01:06:14.715 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-19T01:06:14.715 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-19T01:06:14.716 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-19T01:06:14.716 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-04-19T01:06:14.716 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-04-19T01:06:14.717 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2024-04-19T01:06:14.717 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2024-04-19T01:06:14.718 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 flush_pg_stats 2024-04-19T01:06:14.718 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 flush_pg_stats 2024-04-19T01:06:14.719 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 flush_pg_stats 2024-04-19T01:06:14.719 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 flush_pg_stats 2024-04-19T01:06:15.140 INFO:teuthology.orchestra.run.smithi033.stdout:103079215137 2024-04-19T01:06:15.140 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-19T01:06:15.161 INFO:teuthology.orchestra.run.smithi033.stdout:103079215138 2024-04-19T01:06:15.162 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-04-19T01:06:15.195 INFO:teuthology.orchestra.run.smithi033.stdout:94489280544 2024-04-19T01:06:15.196 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-04-19T01:06:15.204 INFO:teuthology.orchestra.run.smithi033.stdout:94489280546 2024-04-19T01:06:15.204 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-04-19T01:06:15.221 INFO:teuthology.orchestra.run.smithi033.stdout:94489280546 2024-04-19T01:06:15.222 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-19T01:06:15.237 INFO:teuthology.orchestra.run.smithi033.stdout:90194313250 2024-04-19T01:06:15.237 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-04-19T01:06:15.243 INFO:teuthology.orchestra.run.smithi033.stdout:94489280544 2024-04-19T01:06:15.243 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-04-19T01:06:15.244 INFO:teuthology.orchestra.run.smithi033.stdout:94489280545 2024-04-19T01:06:15.244 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-19T01:06:15.253 INFO:teuthology.orchestra.run.smithi033.stdout:103079215137 2024-04-19T01:06:15.254 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-19T01:06:15.268 INFO:teuthology.orchestra.run.smithi033.stdout:103079215139 2024-04-19T01:06:15.270 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-04-19T01:06:15.289 INFO:teuthology.orchestra.run.smithi033.stdout:103079215138 2024-04-19T01:06:15.289 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-19T01:06:15.300 INFO:teuthology.orchestra.run.smithi033.stdout:103079215137 2024-04-19T01:06:15.300 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-04-19T01:06:15.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-04-19T01:06:15.304+0000 7f5c44ce9640 0 --2- 172.21.15.33:0/2917717168 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c40160e80 0x7f5c4006dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-04-19T01:06:15.312 INFO:teuthology.orchestra.run.smithi033.stdout:94489280545 2024-04-19T01:06:15.313 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-04-19T01:06:15.342 INFO:teuthology.orchestra.run.smithi033.stdout:94489280544 2024-04-19T01:06:15.343 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-19T01:06:15.354 INFO:teuthology.orchestra.run.smithi033.stdout:94489280545 2024-04-19T01:06:15.355 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-19T01:06:15.436 INFO:teuthology.orchestra.run.smithi033.stdout:94489280545 2024-04-19T01:06:15.436 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-19T01:06:15.975 INFO:teuthology.orchestra.run.smithi033.stdout:103079215136 2024-04-19T01:06:16.003 INFO:teuthology.orchestra.run.smithi033.stdout:103079215137 2024-04-19T01:06:16.042 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215137 got 103079215136 for osd.2 2024-04-19T01:06:16.070 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215138 got 103079215137 for osd.13 2024-04-19T01:06:16.162 INFO:teuthology.orchestra.run.smithi033.stdout:103079215137 2024-04-19T01:06:16.205 INFO:teuthology.orchestra.run.smithi033.stdout:94489280546 2024-04-19T01:06:16.210 INFO:teuthology.orchestra.run.smithi033.stdout:94489280546 2024-04-19T01:06:16.224 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215137 got 103079215137 for osd.14 2024-04-19T01:06:16.224 DEBUG:teuthology.parallel:result is None 2024-04-19T01:06:16.258 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280546 got 94489280546 for osd.8 2024-04-19T01:06:16.258 DEBUG:teuthology.parallel:result is None 2024-04-19T01:06:16.262 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280546 got 94489280546 for osd.1 2024-04-19T01:06:16.262 DEBUG:teuthology.parallel:result is None 2024-04-19T01:06:16.271 INFO:teuthology.orchestra.run.smithi033.stdout:90194313250 2024-04-19T01:06:16.294 INFO:teuthology.orchestra.run.smithi033.stdout:94489280544 2024-04-19T01:06:16.299 INFO:teuthology.orchestra.run.smithi033.stdout:94489280545 2024-04-19T01:06:16.308 INFO:teuthology.orchestra.run.smithi033.stdout:103079215138 2024-04-19T01:06:16.334 INFO:teuthology.orchestra.run.smithi033.stdout:94489280544 2024-04-19T01:06:16.335 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313250 got 90194313250 for osd.10 2024-04-19T01:06:16.335 DEBUG:teuthology.parallel:result is None 2024-04-19T01:06:16.337 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280544 got 94489280544 for osd.12 2024-04-19T01:06:16.337 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280545 got 94489280545 for osd.4 2024-04-19T01:06:16.338 DEBUG:teuthology.parallel:result is None 2024-04-19T01:06:16.338 DEBUG:teuthology.parallel:result is None 2024-04-19T01:06:16.344 INFO:teuthology.orchestra.run.smithi033.stdout:94489280545 2024-04-19T01:06:16.350 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215138 got 103079215138 for osd.0 2024-04-19T01:06:16.350 INFO:teuthology.orchestra.run.smithi033.stdout:103079215137 2024-04-19T01:06:16.350 DEBUG:teuthology.parallel:result is None 2024-04-19T01:06:16.365 INFO:teuthology.orchestra.run.smithi033.stdout:103079215139 2024-04-19T01:06:16.373 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280544 got 94489280544 for osd.11 2024-04-19T01:06:16.373 DEBUG:teuthology.parallel:result is None 2024-04-19T01:06:16.375 INFO:teuthology.orchestra.run.smithi033.stdout:94489280544 2024-04-19T01:06:16.380 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280545 got 94489280545 for osd.3 2024-04-19T01:06:16.380 DEBUG:teuthology.parallel:result is None 2024-04-19T01:06:16.383 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215137 got 103079215137 for osd.7 2024-04-19T01:06:16.383 DEBUG:teuthology.parallel:result is None 2024-04-19T01:06:16.387 INFO:teuthology.orchestra.run.smithi033.stdout:94489280545 2024-04-19T01:06:16.391 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215139 got 103079215139 for osd.9 2024-04-19T01:06:16.391 DEBUG:teuthology.parallel:result is None 2024-04-19T01:06:16.401 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280544 got 94489280544 for osd.6 2024-04-19T01:06:16.401 DEBUG:teuthology.parallel:result is None 2024-04-19T01:06:16.407 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280545 got 94489280545 for osd.15 2024-04-19T01:06:16.407 DEBUG:teuthology.parallel:result is None 2024-04-19T01:06:16.441 INFO:teuthology.orchestra.run.smithi033.stdout:94489280545 2024-04-19T01:06:16.457 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280545 got 94489280545 for osd.5 2024-04-19T01:06:16.458 DEBUG:teuthology.parallel:result is None 2024-04-19T01:06:17.044 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-19T01:06:17.071 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-04-19T01:06:17.353 INFO:teuthology.orchestra.run.smithi033.stdout:103079215137 2024-04-19T01:06:17.371 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215137 got 103079215137 for osd.2 2024-04-19T01:06:17.371 DEBUG:teuthology.parallel:result is None 2024-04-19T01:06:17.380 INFO:teuthology.orchestra.run.smithi033.stdout:103079215138 2024-04-19T01:06:17.397 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215138 got 103079215138 for osd.13 2024-04-19T01:06:17.397 DEBUG:teuthology.parallel:result is None 2024-04-19T01:06:17.397 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-19T01:06:17.397 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T01:06:17.723 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:06:17.724 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-04-19T01:06:17.737 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":221,"stamp":"2024-04-19T01:06:17.563359+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":13,"kb":1509949440,"kb_used":563460,"kb_used_data":29828,"kb_used_omap":24,"kb_used_meta":533351,"kb_avail":1509385980,"statfs":{"total":1546188226560,"available":1545611243520,"internally_reserved":0,"allocated":30543872,"data_stored":24490250,"data_compressed":4452451,"data_compressed_allocated":9097216,"data_compressed_original":19087360,"omap_allocated":25520,"internal_metadata":546151504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001826"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":286,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.641918+0000","last_change":"2024-04-19T01:05:46.014188+0000","last_active":"2024-04-19T01:06:14.641918+0000","last_peered":"2024-04-19T01:06:14.641918+0000","last_clean":"2024-04-19T01:06:14.641918+0000","last_became_active":"2024-04-19T01:03:53.938774+0000","last_became_peered":"2024-04-19T01:03:53.938774+0000","last_unstale":"2024-04-19T01:06:14.641918+0000","last_undegraded":"2024-04-19T01:06:14.641918+0000","last_fullsized":"2024-04-19T01:06:14.641918+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:46.013839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:46.013839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:57.206833+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":243,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.361415+0000","last_change":"2024-04-19T01:06:14.361472+0000","last_active":"2024-04-19T01:06:14.361415+0000","last_peered":"2024-04-19T01:06:14.361415+0000","last_clean":"2024-04-19T01:06:14.361415+0000","last_became_active":"2024-04-19T01:06:07.457936+0000","last_became_peered":"2024-04-19T01:06:07.457936+0000","last_unstale":"2024-04-19T01:06:14.361415+0000","last_undegraded":"2024-04-19T01:06:14.361415+0000","last_fullsized":"2024-04-19T01:06:14.361415+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:06:14.361244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:06:14.361244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:07:28.897633+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13],"acting":[14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":284,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.642941+0000","last_change":"2024-04-19T01:05:21.106153+0000","last_active":"2024-04-19T01:06:14.642941+0000","last_peered":"2024-04-19T01:06:14.642941+0000","last_clean":"2024-04-19T01:06:14.642941+0000","last_became_active":"2024-04-19T01:03:53.771887+0000","last_became_peered":"2024-04-19T01:03:53.771887+0000","last_unstale":"2024-04-19T01:06:14.642941+0000","last_undegraded":"2024-04-19T01:06:14.642941+0000","last_fullsized":"2024-04-19T01:06:14.642941+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:21.105806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:21.105806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:40.011866+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":284,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.643353+0000","last_change":"2024-04-19T01:05:10.593158+0000","last_active":"2024-04-19T01:06:14.643353+0000","last_peered":"2024-04-19T01:06:14.643353+0000","last_clean":"2024-04-19T01:06:14.643353+0000","last_became_active":"2024-04-19T01:03:53.716040+0000","last_became_peered":"2024-04-19T01:03:53.716040+0000","last_unstale":"2024-04-19T01:06:14.643353+0000","last_undegraded":"2024-04-19T01:06:14.643353+0000","last_fullsized":"2024-04-19T01:06:14.643353+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:10.592906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:10.592906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:39.747266+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":294,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.641903+0000","last_change":"2024-04-19T01:06:12.724217+0000","last_active":"2024-04-19T01:06:14.641903+0000","last_peered":"2024-04-19T01:06:14.641903+0000","last_clean":"2024-04-19T01:06:14.641903+0000","last_became_active":"2024-04-19T01:03:54.011460+0000","last_became_peered":"2024-04-19T01:03:54.011460+0000","last_unstale":"2024-04-19T01:06:14.641903+0000","last_undegraded":"2024-04-19T01:06:14.641903+0000","last_fullsized":"2024-04-19T01:06:14.641903+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:06:12.723991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:06:12.723991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:07:26.541856+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":283,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.643538+0000","last_change":"2024-04-19T01:05:43.353831+0000","last_active":"2024-04-19T01:06:14.643538+0000","last_peered":"2024-04-19T01:06:14.643538+0000","last_clean":"2024-04-19T01:06:14.643538+0000","last_became_active":"2024-04-19T01:03:53.649606+0000","last_became_peered":"2024-04-19T01:03:53.649606+0000","last_unstale":"2024-04-19T01:06:14.643538+0000","last_undegraded":"2024-04-19T01:06:14.643538+0000","last_fullsized":"2024-04-19T01:06:14.643538+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:43.353589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:43.353589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:07:12.784639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":286,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.641897+0000","last_change":"2024-04-19T01:05:32.160602+0000","last_active":"2024-04-19T01:06:14.641897+0000","last_peered":"2024-04-19T01:06:14.641897+0000","last_clean":"2024-04-19T01:06:14.641897+0000","last_became_active":"2024-04-19T01:03:53.759326+0000","last_became_peered":"2024-04-19T01:03:53.759326+0000","last_unstale":"2024-04-19T01:06:14.641897+0000","last_undegraded":"2024-04-19T01:06:14.641897+0000","last_fullsized":"2024-04-19T01:06:14.641897+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:32.160242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:32.160242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:40.352873+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":290,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.024954+0000","last_change":"2024-04-19T01:05:44.095759+0000","last_active":"2024-04-19T01:06:14.024954+0000","last_peered":"2024-04-19T01:06:14.024954+0000","last_clean":"2024-04-19T01:06:14.024954+0000","last_became_active":"2024-04-19T01:05:44.095001+0000","last_became_peered":"2024-04-19T01:05:44.095001+0000","last_unstale":"2024-04-19T01:06:14.024954+0000","last_undegraded":"2024-04-19T01:06:14.024954+0000","last_fullsized":"2024-04-19T01:06:14.024954+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:07.548635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:07.548635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:16.956323+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"159'159","reported_seq":493,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.024930+0000","last_change":"2024-04-19T01:06:12.605622+0000","last_active":"2024-04-19T01:06:14.024930+0000","last_peered":"2024-04-19T01:06:14.024930+0000","last_clean":"2024-04-19T01:06:14.024930+0000","last_became_active":"2024-04-19T01:03:50.774078+0000","last_became_peered":"2024-04-19T01:03:50.774078+0000","last_unstale":"2024-04-19T01:06:14.024930+0000","last_undegraded":"2024-04-19T01:06:14.024930+0000","last_fullsized":"2024-04-19T01:06:14.024930+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'159","last_scrub_stamp":"2024-04-19T01:06:12.605370+0000","last_deep_scrub":"159'159","last_deep_scrub_stamp":"2024-04-19T01:06:12.605370+0000","last_clean_scrub_stamp":"2024-04-19T01:06:12.605370+0000","objects_scrubbed":2,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:07:19.942749+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":13},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":15546,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":14,"up_from":24,"seq":103079215138,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37204,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334636,"statfs":{"total":96636764160,"available":96598667264,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,5,6,7,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":24,"seq":103079215139,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37200,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334640,"statfs":{"total":96636764160,"available":96598671360,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[2,6,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":24,"seq":103079215140,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37204,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334636,"statfs":{"total":96636764160,"available":96598667264,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[1,2,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313250,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33044,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[5,6,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280544,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33028,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338812,"statfs":{"total":96636764160,"available":96602943488,"internally_reserved":0,"allocated":1892352,"data_stored":1456684,"data_compressed":275658,"data_compressed_allocated":561152,"data_compressed_original":1122304,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,3,4,5,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33044,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,6,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280545,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33040,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1892352,"data_stored":1456684,"data_compressed":275658,"data_compressed_allocated":561152,"data_compressed_original":1122304,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[2,4,5,6,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280545,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33044,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280545,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33096,"kb_used_data":1912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338744,"statfs":{"total":96636764160,"available":96602873856,"internally_reserved":0,"allocated":1957888,"data_stored":1995262,"data_compressed":285460,"data_compressed_allocated":598016,"data_compressed_original":1642496,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,6,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280546,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,5,6,7,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280544,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51140,"kb_used_data":1912,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320700,"statfs":{"total":96636764160,"available":96584396800,"internally_reserved":0,"allocated":1957888,"data_stored":1995262,"data_compressed":285460,"data_compressed_allocated":598016,"data_compressed_original":1642496,"omap_allocated":1596,"internal_metadata":50395588},"hb_peers":[0,5,6,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280544,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33044,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,6,7,8,9,10,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280545,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1892352,"data_stored":1456684,"data_compressed":275658,"data_compressed_allocated":561152,"data_compressed_original":1122304,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":24,"seq":103079215138,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37204,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334636,"statfs":{"total":96636764160,"available":96598667264,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215139,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33044,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":24,"seq":103079215138,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33044,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T01:06:17.739 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T01:06:18.022 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:06:18.022 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-04-19T01:06:18.039 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":221,"stamp":"2024-04-19T01:06:17.563359+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":13,"kb":1509949440,"kb_used":563460,"kb_used_data":29828,"kb_used_omap":24,"kb_used_meta":533351,"kb_avail":1509385980,"statfs":{"total":1546188226560,"available":1545611243520,"internally_reserved":0,"allocated":30543872,"data_stored":24490250,"data_compressed":4452451,"data_compressed_allocated":9097216,"data_compressed_original":19087360,"omap_allocated":25520,"internal_metadata":546151504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001826"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":286,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.641918+0000","last_change":"2024-04-19T01:05:46.014188+0000","last_active":"2024-04-19T01:06:14.641918+0000","last_peered":"2024-04-19T01:06:14.641918+0000","last_clean":"2024-04-19T01:06:14.641918+0000","last_became_active":"2024-04-19T01:03:53.938774+0000","last_became_peered":"2024-04-19T01:03:53.938774+0000","last_unstale":"2024-04-19T01:06:14.641918+0000","last_undegraded":"2024-04-19T01:06:14.641918+0000","last_fullsized":"2024-04-19T01:06:14.641918+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:46.013839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:46.013839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:57.206833+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":243,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.361415+0000","last_change":"2024-04-19T01:06:14.361472+0000","last_active":"2024-04-19T01:06:14.361415+0000","last_peered":"2024-04-19T01:06:14.361415+0000","last_clean":"2024-04-19T01:06:14.361415+0000","last_became_active":"2024-04-19T01:06:07.457936+0000","last_became_peered":"2024-04-19T01:06:07.457936+0000","last_unstale":"2024-04-19T01:06:14.361415+0000","last_undegraded":"2024-04-19T01:06:14.361415+0000","last_fullsized":"2024-04-19T01:06:14.361415+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:06:14.361244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:06:14.361244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:07:28.897633+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13],"acting":[14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":284,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.642941+0000","last_change":"2024-04-19T01:05:21.106153+0000","last_active":"2024-04-19T01:06:14.642941+0000","last_peered":"2024-04-19T01:06:14.642941+0000","last_clean":"2024-04-19T01:06:14.642941+0000","last_became_active":"2024-04-19T01:03:53.771887+0000","last_became_peered":"2024-04-19T01:03:53.771887+0000","last_unstale":"2024-04-19T01:06:14.642941+0000","last_undegraded":"2024-04-19T01:06:14.642941+0000","last_fullsized":"2024-04-19T01:06:14.642941+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:21.105806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:21.105806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:40.011866+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":284,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.643353+0000","last_change":"2024-04-19T01:05:10.593158+0000","last_active":"2024-04-19T01:06:14.643353+0000","last_peered":"2024-04-19T01:06:14.643353+0000","last_clean":"2024-04-19T01:06:14.643353+0000","last_became_active":"2024-04-19T01:03:53.716040+0000","last_became_peered":"2024-04-19T01:03:53.716040+0000","last_unstale":"2024-04-19T01:06:14.643353+0000","last_undegraded":"2024-04-19T01:06:14.643353+0000","last_fullsized":"2024-04-19T01:06:14.643353+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:10.592906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:10.592906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:39.747266+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":294,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.641903+0000","last_change":"2024-04-19T01:06:12.724217+0000","last_active":"2024-04-19T01:06:14.641903+0000","last_peered":"2024-04-19T01:06:14.641903+0000","last_clean":"2024-04-19T01:06:14.641903+0000","last_became_active":"2024-04-19T01:03:54.011460+0000","last_became_peered":"2024-04-19T01:03:54.011460+0000","last_unstale":"2024-04-19T01:06:14.641903+0000","last_undegraded":"2024-04-19T01:06:14.641903+0000","last_fullsized":"2024-04-19T01:06:14.641903+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:06:12.723991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:06:12.723991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:07:26.541856+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":283,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.643538+0000","last_change":"2024-04-19T01:05:43.353831+0000","last_active":"2024-04-19T01:06:14.643538+0000","last_peered":"2024-04-19T01:06:14.643538+0000","last_clean":"2024-04-19T01:06:14.643538+0000","last_became_active":"2024-04-19T01:03:53.649606+0000","last_became_peered":"2024-04-19T01:03:53.649606+0000","last_unstale":"2024-04-19T01:06:14.643538+0000","last_undegraded":"2024-04-19T01:06:14.643538+0000","last_fullsized":"2024-04-19T01:06:14.643538+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:43.353589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:43.353589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:07:12.784639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":286,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.641897+0000","last_change":"2024-04-19T01:05:32.160602+0000","last_active":"2024-04-19T01:06:14.641897+0000","last_peered":"2024-04-19T01:06:14.641897+0000","last_clean":"2024-04-19T01:06:14.641897+0000","last_became_active":"2024-04-19T01:03:53.759326+0000","last_became_peered":"2024-04-19T01:03:53.759326+0000","last_unstale":"2024-04-19T01:06:14.641897+0000","last_undegraded":"2024-04-19T01:06:14.641897+0000","last_fullsized":"2024-04-19T01:06:14.641897+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:32.160242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:32.160242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:40.352873+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":290,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.024954+0000","last_change":"2024-04-19T01:05:44.095759+0000","last_active":"2024-04-19T01:06:14.024954+0000","last_peered":"2024-04-19T01:06:14.024954+0000","last_clean":"2024-04-19T01:06:14.024954+0000","last_became_active":"2024-04-19T01:05:44.095001+0000","last_became_peered":"2024-04-19T01:05:44.095001+0000","last_unstale":"2024-04-19T01:06:14.024954+0000","last_undegraded":"2024-04-19T01:06:14.024954+0000","last_fullsized":"2024-04-19T01:06:14.024954+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:07.548635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:07.548635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:16.956323+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"159'159","reported_seq":493,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.024930+0000","last_change":"2024-04-19T01:06:12.605622+0000","last_active":"2024-04-19T01:06:14.024930+0000","last_peered":"2024-04-19T01:06:14.024930+0000","last_clean":"2024-04-19T01:06:14.024930+0000","last_became_active":"2024-04-19T01:03:50.774078+0000","last_became_peered":"2024-04-19T01:03:50.774078+0000","last_unstale":"2024-04-19T01:06:14.024930+0000","last_undegraded":"2024-04-19T01:06:14.024930+0000","last_fullsized":"2024-04-19T01:06:14.024930+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'159","last_scrub_stamp":"2024-04-19T01:06:12.605370+0000","last_deep_scrub":"159'159","last_deep_scrub_stamp":"2024-04-19T01:06:12.605370+0000","last_clean_scrub_stamp":"2024-04-19T01:06:12.605370+0000","objects_scrubbed":2,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:07:19.942749+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":13},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":15546,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":14,"up_from":24,"seq":103079215138,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37204,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334636,"statfs":{"total":96636764160,"available":96598667264,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,5,6,7,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":24,"seq":103079215139,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37200,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334640,"statfs":{"total":96636764160,"available":96598671360,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[2,6,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":24,"seq":103079215140,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37204,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334636,"statfs":{"total":96636764160,"available":96598667264,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[1,2,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313250,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33044,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[5,6,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280544,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33028,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338812,"statfs":{"total":96636764160,"available":96602943488,"internally_reserved":0,"allocated":1892352,"data_stored":1456684,"data_compressed":275658,"data_compressed_allocated":561152,"data_compressed_original":1122304,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,3,4,5,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33044,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,6,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280545,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33040,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1892352,"data_stored":1456684,"data_compressed":275658,"data_compressed_allocated":561152,"data_compressed_original":1122304,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[2,4,5,6,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280545,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33044,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280545,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33096,"kb_used_data":1912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338744,"statfs":{"total":96636764160,"available":96602873856,"internally_reserved":0,"allocated":1957888,"data_stored":1995262,"data_compressed":285460,"data_compressed_allocated":598016,"data_compressed_original":1642496,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,6,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280546,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,5,6,7,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280544,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51140,"kb_used_data":1912,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320700,"statfs":{"total":96636764160,"available":96584396800,"internally_reserved":0,"allocated":1957888,"data_stored":1995262,"data_compressed":285460,"data_compressed_allocated":598016,"data_compressed_original":1642496,"omap_allocated":1596,"internal_metadata":50395588},"hb_peers":[0,5,6,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280544,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33044,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,6,7,8,9,10,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280545,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1892352,"data_stored":1456684,"data_compressed":275658,"data_compressed_allocated":561152,"data_compressed_original":1122304,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":24,"seq":103079215138,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37204,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334636,"statfs":{"total":96636764160,"available":96598667264,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215139,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33044,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":24,"seq":103079215138,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33044,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T01:06:18.040 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-19T01:06:18.040 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-19T01:06:18.057 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-19T01:06:18.058 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T01:06:18.349 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:06:18.349 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-04-19T01:06:18.368 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":221,"stamp":"2024-04-19T01:06:17.563359+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":13,"kb":1509949440,"kb_used":563460,"kb_used_data":29828,"kb_used_omap":24,"kb_used_meta":533351,"kb_avail":1509385980,"statfs":{"total":1546188226560,"available":1545611243520,"internally_reserved":0,"allocated":30543872,"data_stored":24490250,"data_compressed":4452451,"data_compressed_allocated":9097216,"data_compressed_original":19087360,"omap_allocated":25520,"internal_metadata":546151504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001826"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":286,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.641918+0000","last_change":"2024-04-19T01:05:46.014188+0000","last_active":"2024-04-19T01:06:14.641918+0000","last_peered":"2024-04-19T01:06:14.641918+0000","last_clean":"2024-04-19T01:06:14.641918+0000","last_became_active":"2024-04-19T01:03:53.938774+0000","last_became_peered":"2024-04-19T01:03:53.938774+0000","last_unstale":"2024-04-19T01:06:14.641918+0000","last_undegraded":"2024-04-19T01:06:14.641918+0000","last_fullsized":"2024-04-19T01:06:14.641918+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:46.013839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:46.013839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:57.206833+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":243,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.361415+0000","last_change":"2024-04-19T01:06:14.361472+0000","last_active":"2024-04-19T01:06:14.361415+0000","last_peered":"2024-04-19T01:06:14.361415+0000","last_clean":"2024-04-19T01:06:14.361415+0000","last_became_active":"2024-04-19T01:06:07.457936+0000","last_became_peered":"2024-04-19T01:06:07.457936+0000","last_unstale":"2024-04-19T01:06:14.361415+0000","last_undegraded":"2024-04-19T01:06:14.361415+0000","last_fullsized":"2024-04-19T01:06:14.361415+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:06:14.361244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:06:14.361244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:07:28.897633+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13],"acting":[14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":284,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.642941+0000","last_change":"2024-04-19T01:05:21.106153+0000","last_active":"2024-04-19T01:06:14.642941+0000","last_peered":"2024-04-19T01:06:14.642941+0000","last_clean":"2024-04-19T01:06:14.642941+0000","last_became_active":"2024-04-19T01:03:53.771887+0000","last_became_peered":"2024-04-19T01:03:53.771887+0000","last_unstale":"2024-04-19T01:06:14.642941+0000","last_undegraded":"2024-04-19T01:06:14.642941+0000","last_fullsized":"2024-04-19T01:06:14.642941+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:21.105806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:21.105806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:40.011866+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":284,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.643353+0000","last_change":"2024-04-19T01:05:10.593158+0000","last_active":"2024-04-19T01:06:14.643353+0000","last_peered":"2024-04-19T01:06:14.643353+0000","last_clean":"2024-04-19T01:06:14.643353+0000","last_became_active":"2024-04-19T01:03:53.716040+0000","last_became_peered":"2024-04-19T01:03:53.716040+0000","last_unstale":"2024-04-19T01:06:14.643353+0000","last_undegraded":"2024-04-19T01:06:14.643353+0000","last_fullsized":"2024-04-19T01:06:14.643353+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:10.592906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:10.592906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:39.747266+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":294,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.641903+0000","last_change":"2024-04-19T01:06:12.724217+0000","last_active":"2024-04-19T01:06:14.641903+0000","last_peered":"2024-04-19T01:06:14.641903+0000","last_clean":"2024-04-19T01:06:14.641903+0000","last_became_active":"2024-04-19T01:03:54.011460+0000","last_became_peered":"2024-04-19T01:03:54.011460+0000","last_unstale":"2024-04-19T01:06:14.641903+0000","last_undegraded":"2024-04-19T01:06:14.641903+0000","last_fullsized":"2024-04-19T01:06:14.641903+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:06:12.723991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:06:12.723991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:07:26.541856+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":283,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.643538+0000","last_change":"2024-04-19T01:05:43.353831+0000","last_active":"2024-04-19T01:06:14.643538+0000","last_peered":"2024-04-19T01:06:14.643538+0000","last_clean":"2024-04-19T01:06:14.643538+0000","last_became_active":"2024-04-19T01:03:53.649606+0000","last_became_peered":"2024-04-19T01:03:53.649606+0000","last_unstale":"2024-04-19T01:06:14.643538+0000","last_undegraded":"2024-04-19T01:06:14.643538+0000","last_fullsized":"2024-04-19T01:06:14.643538+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:43.353589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:43.353589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:07:12.784639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":286,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.641897+0000","last_change":"2024-04-19T01:05:32.160602+0000","last_active":"2024-04-19T01:06:14.641897+0000","last_peered":"2024-04-19T01:06:14.641897+0000","last_clean":"2024-04-19T01:06:14.641897+0000","last_became_active":"2024-04-19T01:03:53.759326+0000","last_became_peered":"2024-04-19T01:03:53.759326+0000","last_unstale":"2024-04-19T01:06:14.641897+0000","last_undegraded":"2024-04-19T01:06:14.641897+0000","last_fullsized":"2024-04-19T01:06:14.641897+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:32.160242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:32.160242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:40.352873+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":290,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.024954+0000","last_change":"2024-04-19T01:05:44.095759+0000","last_active":"2024-04-19T01:06:14.024954+0000","last_peered":"2024-04-19T01:06:14.024954+0000","last_clean":"2024-04-19T01:06:14.024954+0000","last_became_active":"2024-04-19T01:05:44.095001+0000","last_became_peered":"2024-04-19T01:05:44.095001+0000","last_unstale":"2024-04-19T01:06:14.024954+0000","last_undegraded":"2024-04-19T01:06:14.024954+0000","last_fullsized":"2024-04-19T01:06:14.024954+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:07.548635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:07.548635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:16.956323+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"159'159","reported_seq":493,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.024930+0000","last_change":"2024-04-19T01:06:12.605622+0000","last_active":"2024-04-19T01:06:14.024930+0000","last_peered":"2024-04-19T01:06:14.024930+0000","last_clean":"2024-04-19T01:06:14.024930+0000","last_became_active":"2024-04-19T01:03:50.774078+0000","last_became_peered":"2024-04-19T01:03:50.774078+0000","last_unstale":"2024-04-19T01:06:14.024930+0000","last_undegraded":"2024-04-19T01:06:14.024930+0000","last_fullsized":"2024-04-19T01:06:14.024930+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'159","last_scrub_stamp":"2024-04-19T01:06:12.605370+0000","last_deep_scrub":"159'159","last_deep_scrub_stamp":"2024-04-19T01:06:12.605370+0000","last_clean_scrub_stamp":"2024-04-19T01:06:12.605370+0000","objects_scrubbed":2,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:07:19.942749+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":13},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":15546,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":14,"up_from":24,"seq":103079215138,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37204,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334636,"statfs":{"total":96636764160,"available":96598667264,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,5,6,7,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":24,"seq":103079215139,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37200,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334640,"statfs":{"total":96636764160,"available":96598671360,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[2,6,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":24,"seq":103079215140,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37204,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334636,"statfs":{"total":96636764160,"available":96598667264,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[1,2,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313250,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33044,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[5,6,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280544,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33028,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338812,"statfs":{"total":96636764160,"available":96602943488,"internally_reserved":0,"allocated":1892352,"data_stored":1456684,"data_compressed":275658,"data_compressed_allocated":561152,"data_compressed_original":1122304,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,3,4,5,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33044,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,6,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280545,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33040,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1892352,"data_stored":1456684,"data_compressed":275658,"data_compressed_allocated":561152,"data_compressed_original":1122304,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[2,4,5,6,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280545,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33044,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280545,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33096,"kb_used_data":1912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338744,"statfs":{"total":96636764160,"available":96602873856,"internally_reserved":0,"allocated":1957888,"data_stored":1995262,"data_compressed":285460,"data_compressed_allocated":598016,"data_compressed_original":1642496,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,6,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280546,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,5,6,7,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280544,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51140,"kb_used_data":1912,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320700,"statfs":{"total":96636764160,"available":96584396800,"internally_reserved":0,"allocated":1957888,"data_stored":1995262,"data_compressed":285460,"data_compressed_allocated":598016,"data_compressed_original":1642496,"omap_allocated":1596,"internal_metadata":50395588},"hb_peers":[0,5,6,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280544,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33044,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,6,7,8,9,10,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280545,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1892352,"data_stored":1456684,"data_compressed":275658,"data_compressed_allocated":561152,"data_compressed_original":1122304,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":24,"seq":103079215138,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37204,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334636,"statfs":{"total":96636764160,"available":96598667264,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215139,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33044,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":24,"seq":103079215138,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33044,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T01:06:18.369 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T01:06:18.673 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:06:18.674 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-04-19T01:06:18.692 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":221,"stamp":"2024-04-19T01:06:17.563359+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":13,"kb":1509949440,"kb_used":563460,"kb_used_data":29828,"kb_used_omap":24,"kb_used_meta":533351,"kb_avail":1509385980,"statfs":{"total":1546188226560,"available":1545611243520,"internally_reserved":0,"allocated":30543872,"data_stored":24490250,"data_compressed":4452451,"data_compressed_allocated":9097216,"data_compressed_original":19087360,"omap_allocated":25520,"internal_metadata":546151504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001826"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":286,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.641918+0000","last_change":"2024-04-19T01:05:46.014188+0000","last_active":"2024-04-19T01:06:14.641918+0000","last_peered":"2024-04-19T01:06:14.641918+0000","last_clean":"2024-04-19T01:06:14.641918+0000","last_became_active":"2024-04-19T01:03:53.938774+0000","last_became_peered":"2024-04-19T01:03:53.938774+0000","last_unstale":"2024-04-19T01:06:14.641918+0000","last_undegraded":"2024-04-19T01:06:14.641918+0000","last_fullsized":"2024-04-19T01:06:14.641918+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:46.013839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:46.013839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:57.206833+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":243,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.361415+0000","last_change":"2024-04-19T01:06:14.361472+0000","last_active":"2024-04-19T01:06:14.361415+0000","last_peered":"2024-04-19T01:06:14.361415+0000","last_clean":"2024-04-19T01:06:14.361415+0000","last_became_active":"2024-04-19T01:06:07.457936+0000","last_became_peered":"2024-04-19T01:06:07.457936+0000","last_unstale":"2024-04-19T01:06:14.361415+0000","last_undegraded":"2024-04-19T01:06:14.361415+0000","last_fullsized":"2024-04-19T01:06:14.361415+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:06:14.361244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:06:14.361244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:07:28.897633+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13],"acting":[14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":284,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.642941+0000","last_change":"2024-04-19T01:05:21.106153+0000","last_active":"2024-04-19T01:06:14.642941+0000","last_peered":"2024-04-19T01:06:14.642941+0000","last_clean":"2024-04-19T01:06:14.642941+0000","last_became_active":"2024-04-19T01:03:53.771887+0000","last_became_peered":"2024-04-19T01:03:53.771887+0000","last_unstale":"2024-04-19T01:06:14.642941+0000","last_undegraded":"2024-04-19T01:06:14.642941+0000","last_fullsized":"2024-04-19T01:06:14.642941+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:21.105806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:21.105806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:40.011866+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":284,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.643353+0000","last_change":"2024-04-19T01:05:10.593158+0000","last_active":"2024-04-19T01:06:14.643353+0000","last_peered":"2024-04-19T01:06:14.643353+0000","last_clean":"2024-04-19T01:06:14.643353+0000","last_became_active":"2024-04-19T01:03:53.716040+0000","last_became_peered":"2024-04-19T01:03:53.716040+0000","last_unstale":"2024-04-19T01:06:14.643353+0000","last_undegraded":"2024-04-19T01:06:14.643353+0000","last_fullsized":"2024-04-19T01:06:14.643353+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:10.592906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:10.592906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:39.747266+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":294,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.641903+0000","last_change":"2024-04-19T01:06:12.724217+0000","last_active":"2024-04-19T01:06:14.641903+0000","last_peered":"2024-04-19T01:06:14.641903+0000","last_clean":"2024-04-19T01:06:14.641903+0000","last_became_active":"2024-04-19T01:03:54.011460+0000","last_became_peered":"2024-04-19T01:03:54.011460+0000","last_unstale":"2024-04-19T01:06:14.641903+0000","last_undegraded":"2024-04-19T01:06:14.641903+0000","last_fullsized":"2024-04-19T01:06:14.641903+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:06:12.723991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:06:12.723991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:07:26.541856+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":283,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.643538+0000","last_change":"2024-04-19T01:05:43.353831+0000","last_active":"2024-04-19T01:06:14.643538+0000","last_peered":"2024-04-19T01:06:14.643538+0000","last_clean":"2024-04-19T01:06:14.643538+0000","last_became_active":"2024-04-19T01:03:53.649606+0000","last_became_peered":"2024-04-19T01:03:53.649606+0000","last_unstale":"2024-04-19T01:06:14.643538+0000","last_undegraded":"2024-04-19T01:06:14.643538+0000","last_fullsized":"2024-04-19T01:06:14.643538+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:43.353589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:43.353589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:07:12.784639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":286,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.641897+0000","last_change":"2024-04-19T01:05:32.160602+0000","last_active":"2024-04-19T01:06:14.641897+0000","last_peered":"2024-04-19T01:06:14.641897+0000","last_clean":"2024-04-19T01:06:14.641897+0000","last_became_active":"2024-04-19T01:03:53.759326+0000","last_became_peered":"2024-04-19T01:03:53.759326+0000","last_unstale":"2024-04-19T01:06:14.641897+0000","last_undegraded":"2024-04-19T01:06:14.641897+0000","last_fullsized":"2024-04-19T01:06:14.641897+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:32.160242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:32.160242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:40.352873+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":290,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.024954+0000","last_change":"2024-04-19T01:05:44.095759+0000","last_active":"2024-04-19T01:06:14.024954+0000","last_peered":"2024-04-19T01:06:14.024954+0000","last_clean":"2024-04-19T01:06:14.024954+0000","last_became_active":"2024-04-19T01:05:44.095001+0000","last_became_peered":"2024-04-19T01:05:44.095001+0000","last_unstale":"2024-04-19T01:06:14.024954+0000","last_undegraded":"2024-04-19T01:06:14.024954+0000","last_fullsized":"2024-04-19T01:06:14.024954+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:07.548635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:07.548635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:16.956323+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"159'159","reported_seq":493,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.024930+0000","last_change":"2024-04-19T01:06:12.605622+0000","last_active":"2024-04-19T01:06:14.024930+0000","last_peered":"2024-04-19T01:06:14.024930+0000","last_clean":"2024-04-19T01:06:14.024930+0000","last_became_active":"2024-04-19T01:03:50.774078+0000","last_became_peered":"2024-04-19T01:03:50.774078+0000","last_unstale":"2024-04-19T01:06:14.024930+0000","last_undegraded":"2024-04-19T01:06:14.024930+0000","last_fullsized":"2024-04-19T01:06:14.024930+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'159","last_scrub_stamp":"2024-04-19T01:06:12.605370+0000","last_deep_scrub":"159'159","last_deep_scrub_stamp":"2024-04-19T01:06:12.605370+0000","last_clean_scrub_stamp":"2024-04-19T01:06:12.605370+0000","objects_scrubbed":2,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:07:19.942749+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":13},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":15546,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":14,"up_from":24,"seq":103079215138,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37204,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334636,"statfs":{"total":96636764160,"available":96598667264,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,5,6,7,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":24,"seq":103079215139,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37200,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334640,"statfs":{"total":96636764160,"available":96598671360,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[2,6,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":24,"seq":103079215140,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37204,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334636,"statfs":{"total":96636764160,"available":96598667264,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[1,2,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313250,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33044,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[5,6,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280544,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33028,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338812,"statfs":{"total":96636764160,"available":96602943488,"internally_reserved":0,"allocated":1892352,"data_stored":1456684,"data_compressed":275658,"data_compressed_allocated":561152,"data_compressed_original":1122304,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,3,4,5,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33044,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,6,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280545,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33040,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1892352,"data_stored":1456684,"data_compressed":275658,"data_compressed_allocated":561152,"data_compressed_original":1122304,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[2,4,5,6,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280545,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33044,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280545,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33096,"kb_used_data":1912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338744,"statfs":{"total":96636764160,"available":96602873856,"internally_reserved":0,"allocated":1957888,"data_stored":1995262,"data_compressed":285460,"data_compressed_allocated":598016,"data_compressed_original":1642496,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,6,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280546,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,5,6,7,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280544,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51140,"kb_used_data":1912,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320700,"statfs":{"total":96636764160,"available":96584396800,"internally_reserved":0,"allocated":1957888,"data_stored":1995262,"data_compressed":285460,"data_compressed_allocated":598016,"data_compressed_original":1642496,"omap_allocated":1596,"internal_metadata":50395588},"hb_peers":[0,5,6,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280544,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33044,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,6,7,8,9,10,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280545,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1892352,"data_stored":1456684,"data_compressed":275658,"data_compressed_allocated":561152,"data_compressed_original":1122304,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":24,"seq":103079215138,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37204,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334636,"statfs":{"total":96636764160,"available":96598667264,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215139,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33044,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":24,"seq":103079215138,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33044,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T01:06:18.693 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-19T01:06:18.693 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T01:06:18.983 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:06:18.984 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-04-19T01:06:19.000 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":221,"stamp":"2024-04-19T01:06:17.563359+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":13,"kb":1509949440,"kb_used":563460,"kb_used_data":29828,"kb_used_omap":24,"kb_used_meta":533351,"kb_avail":1509385980,"statfs":{"total":1546188226560,"available":1545611243520,"internally_reserved":0,"allocated":30543872,"data_stored":24490250,"data_compressed":4452451,"data_compressed_allocated":9097216,"data_compressed_original":19087360,"omap_allocated":25520,"internal_metadata":546151504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001826"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":286,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.641918+0000","last_change":"2024-04-19T01:05:46.014188+0000","last_active":"2024-04-19T01:06:14.641918+0000","last_peered":"2024-04-19T01:06:14.641918+0000","last_clean":"2024-04-19T01:06:14.641918+0000","last_became_active":"2024-04-19T01:03:53.938774+0000","last_became_peered":"2024-04-19T01:03:53.938774+0000","last_unstale":"2024-04-19T01:06:14.641918+0000","last_undegraded":"2024-04-19T01:06:14.641918+0000","last_fullsized":"2024-04-19T01:06:14.641918+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:46.013839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:46.013839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:57.206833+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":243,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.361415+0000","last_change":"2024-04-19T01:06:14.361472+0000","last_active":"2024-04-19T01:06:14.361415+0000","last_peered":"2024-04-19T01:06:14.361415+0000","last_clean":"2024-04-19T01:06:14.361415+0000","last_became_active":"2024-04-19T01:06:07.457936+0000","last_became_peered":"2024-04-19T01:06:07.457936+0000","last_unstale":"2024-04-19T01:06:14.361415+0000","last_undegraded":"2024-04-19T01:06:14.361415+0000","last_fullsized":"2024-04-19T01:06:14.361415+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:06:14.361244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:06:14.361244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:07:28.897633+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13],"acting":[14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":284,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.642941+0000","last_change":"2024-04-19T01:05:21.106153+0000","last_active":"2024-04-19T01:06:14.642941+0000","last_peered":"2024-04-19T01:06:14.642941+0000","last_clean":"2024-04-19T01:06:14.642941+0000","last_became_active":"2024-04-19T01:03:53.771887+0000","last_became_peered":"2024-04-19T01:03:53.771887+0000","last_unstale":"2024-04-19T01:06:14.642941+0000","last_undegraded":"2024-04-19T01:06:14.642941+0000","last_fullsized":"2024-04-19T01:06:14.642941+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:21.105806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:21.105806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:40.011866+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":284,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.643353+0000","last_change":"2024-04-19T01:05:10.593158+0000","last_active":"2024-04-19T01:06:14.643353+0000","last_peered":"2024-04-19T01:06:14.643353+0000","last_clean":"2024-04-19T01:06:14.643353+0000","last_became_active":"2024-04-19T01:03:53.716040+0000","last_became_peered":"2024-04-19T01:03:53.716040+0000","last_unstale":"2024-04-19T01:06:14.643353+0000","last_undegraded":"2024-04-19T01:06:14.643353+0000","last_fullsized":"2024-04-19T01:06:14.643353+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:10.592906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:10.592906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:39.747266+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":294,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.641903+0000","last_change":"2024-04-19T01:06:12.724217+0000","last_active":"2024-04-19T01:06:14.641903+0000","last_peered":"2024-04-19T01:06:14.641903+0000","last_clean":"2024-04-19T01:06:14.641903+0000","last_became_active":"2024-04-19T01:03:54.011460+0000","last_became_peered":"2024-04-19T01:03:54.011460+0000","last_unstale":"2024-04-19T01:06:14.641903+0000","last_undegraded":"2024-04-19T01:06:14.641903+0000","last_fullsized":"2024-04-19T01:06:14.641903+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:06:12.723991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:06:12.723991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:07:26.541856+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":283,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.643538+0000","last_change":"2024-04-19T01:05:43.353831+0000","last_active":"2024-04-19T01:06:14.643538+0000","last_peered":"2024-04-19T01:06:14.643538+0000","last_clean":"2024-04-19T01:06:14.643538+0000","last_became_active":"2024-04-19T01:03:53.649606+0000","last_became_peered":"2024-04-19T01:03:53.649606+0000","last_unstale":"2024-04-19T01:06:14.643538+0000","last_undegraded":"2024-04-19T01:06:14.643538+0000","last_fullsized":"2024-04-19T01:06:14.643538+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:43.353589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:43.353589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:07:12.784639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":286,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.641897+0000","last_change":"2024-04-19T01:05:32.160602+0000","last_active":"2024-04-19T01:06:14.641897+0000","last_peered":"2024-04-19T01:06:14.641897+0000","last_clean":"2024-04-19T01:06:14.641897+0000","last_became_active":"2024-04-19T01:03:53.759326+0000","last_became_peered":"2024-04-19T01:03:53.759326+0000","last_unstale":"2024-04-19T01:06:14.641897+0000","last_undegraded":"2024-04-19T01:06:14.641897+0000","last_fullsized":"2024-04-19T01:06:14.641897+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:32.160242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:32.160242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:40.352873+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":290,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.024954+0000","last_change":"2024-04-19T01:05:44.095759+0000","last_active":"2024-04-19T01:06:14.024954+0000","last_peered":"2024-04-19T01:06:14.024954+0000","last_clean":"2024-04-19T01:06:14.024954+0000","last_became_active":"2024-04-19T01:05:44.095001+0000","last_became_peered":"2024-04-19T01:05:44.095001+0000","last_unstale":"2024-04-19T01:06:14.024954+0000","last_undegraded":"2024-04-19T01:06:14.024954+0000","last_fullsized":"2024-04-19T01:06:14.024954+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:07.548635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:07.548635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:16.956323+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"159'159","reported_seq":493,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-04-19T01:06:14.024930+0000","last_change":"2024-04-19T01:06:12.605622+0000","last_active":"2024-04-19T01:06:14.024930+0000","last_peered":"2024-04-19T01:06:14.024930+0000","last_clean":"2024-04-19T01:06:14.024930+0000","last_became_active":"2024-04-19T01:03:50.774078+0000","last_became_peered":"2024-04-19T01:03:50.774078+0000","last_unstale":"2024-04-19T01:06:14.024930+0000","last_undegraded":"2024-04-19T01:06:14.024930+0000","last_fullsized":"2024-04-19T01:06:14.024930+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'159","last_scrub_stamp":"2024-04-19T01:06:12.605370+0000","last_deep_scrub":"159'159","last_deep_scrub_stamp":"2024-04-19T01:06:12.605370+0000","last_clean_scrub_stamp":"2024-04-19T01:06:12.605370+0000","objects_scrubbed":2,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:07:19.942749+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":13},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":15546,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":14,"up_from":24,"seq":103079215138,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37204,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334636,"statfs":{"total":96636764160,"available":96598667264,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,5,6,7,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":24,"seq":103079215139,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37200,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334640,"statfs":{"total":96636764160,"available":96598671360,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[2,6,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":24,"seq":103079215140,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37204,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334636,"statfs":{"total":96636764160,"available":96598667264,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[1,2,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313250,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33044,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[5,6,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280544,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33028,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338812,"statfs":{"total":96636764160,"available":96602943488,"internally_reserved":0,"allocated":1892352,"data_stored":1456684,"data_compressed":275658,"data_compressed_allocated":561152,"data_compressed_original":1122304,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,3,4,5,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33044,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,6,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280545,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33040,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1892352,"data_stored":1456684,"data_compressed":275658,"data_compressed_allocated":561152,"data_compressed_original":1122304,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[2,4,5,6,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280545,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33044,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280545,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33096,"kb_used_data":1912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338744,"statfs":{"total":96636764160,"available":96602873856,"internally_reserved":0,"allocated":1957888,"data_stored":1995262,"data_compressed":285460,"data_compressed_allocated":598016,"data_compressed_original":1642496,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,6,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280546,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,5,6,7,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280544,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51140,"kb_used_data":1912,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320700,"statfs":{"total":96636764160,"available":96584396800,"internally_reserved":0,"allocated":1957888,"data_stored":1995262,"data_compressed":285460,"data_compressed_allocated":598016,"data_compressed_original":1642496,"omap_allocated":1596,"internal_metadata":50395588},"hb_peers":[0,5,6,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280544,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33044,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,6,7,8,9,10,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280545,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1892352,"data_stored":1456684,"data_compressed":275658,"data_compressed_allocated":561152,"data_compressed_original":1122304,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":24,"seq":103079215138,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37204,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334636,"statfs":{"total":96636764160,"available":96598667264,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215139,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33044,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":24,"seq":103079215138,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33044,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1904640,"data_stored":1466334,"data_compressed":277687,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T01:06:19.001 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-19T01:06:19.309 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:06:19.309 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":162,"fsid":"f11bd2d5-4456-4ffa-83c0-722dddd22ad3","created":"2024-04-19T01:03:36.766089+0000","modified":"2024-04-19T01:06:19.045575+0000","last_up_change":"2024-04-19T01:03:50.591452+0000","last_in_change":"2024-04-19T01:05:41.710130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-19T01:03:49.571609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-19T01:03:51.913782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.0199999809265137,"score_stable":6.0199999809265137,"optimal_score":0.81000000238418579,"raw_score_acting":4.880000114440918,"raw_score_stable":4.880000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"296899fd-2b2b-4071-8b1f-0f92c3d0bbc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":1633616806},{"type":"v1","addr":"172.21.15.33:6825","nonce":1633616806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":1633616806},{"type":"v1","addr":"172.21.15.33:6827","nonce":1633616806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":1633616806},{"type":"v1","addr":"172.21.15.33:6831","nonce":1633616806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":1633616806},{"type":"v1","addr":"172.21.15.33:6829","nonce":1633616806}]},"public_addr":"172.21.15.33:6825/1633616806","cluster_addr":"172.21.15.33:6827/1633616806","heartbeat_back_addr":"172.21.15.33:6831/1633616806","heartbeat_front_addr":"172.21.15.33:6829/1633616806","state":["exists","up"]},{"osd":1,"uuid":"021f7190-0ec3-4553-a44f-15095765cccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":4270148213},{"type":"v1","addr":"172.21.15.110:6808","nonce":4270148213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6811","nonce":4270148213},{"type":"v1","addr":"172.21.15.110:6814","nonce":4270148213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":4270148213},{"type":"v1","addr":"172.21.15.110:6823","nonce":4270148213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6817","nonce":4270148213},{"type":"v1","addr":"172.21.15.110:6820","nonce":4270148213}]},"public_addr":"172.21.15.110:6808/4270148213","cluster_addr":"172.21.15.110:6814/4270148213","heartbeat_back_addr":"172.21.15.110:6823/4270148213","heartbeat_front_addr":"172.21.15.110:6820/4270148213","state":["exists","up"]},{"osd":2,"uuid":"c283d0bc-5e34-4763-9019-328ee9b6c07a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":13248188},{"type":"v1","addr":"172.21.15.118:6809","nonce":13248188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":13248188},{"type":"v1","addr":"172.21.15.118:6811","nonce":13248188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":13248188},{"type":"v1","addr":"172.21.15.118:6815","nonce":13248188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":13248188},{"type":"v1","addr":"172.21.15.118:6813","nonce":13248188}]},"public_addr":"172.21.15.118:6809/13248188","cluster_addr":"172.21.15.118:6811/13248188","heartbeat_back_addr":"172.21.15.118:6815/13248188","heartbeat_front_addr":"172.21.15.118:6813/13248188","state":["exists","up"]},{"osd":3,"uuid":"d45888f2-eecd-480b-80f5-c0731a06ff3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":2658385904},{"type":"v1","addr":"172.21.15.155:6803","nonce":2658385904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":2658385904},{"type":"v1","addr":"172.21.15.155:6805","nonce":2658385904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6813","nonce":2658385904},{"type":"v1","addr":"172.21.15.155:6816","nonce":2658385904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":2658385904},{"type":"v1","addr":"172.21.15.155:6809","nonce":2658385904}]},"public_addr":"172.21.15.155:6803/2658385904","cluster_addr":"172.21.15.155:6805/2658385904","heartbeat_back_addr":"172.21.15.155:6816/2658385904","heartbeat_front_addr":"172.21.15.155:6809/2658385904","state":["exists","up"]},{"osd":4,"uuid":"d060d939-8c94-4a07-9fd5-42af0044a16f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":2389566155},{"type":"v1","addr":"172.21.15.33:6801","nonce":2389566155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":2389566155},{"type":"v1","addr":"172.21.15.33:6803","nonce":2389566155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":2389566155},{"type":"v1","addr":"172.21.15.33:6807","nonce":2389566155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":2389566155},{"type":"v1","addr":"172.21.15.33:6805","nonce":2389566155}]},"public_addr":"172.21.15.33:6801/2389566155","cluster_addr":"172.21.15.33:6803/2389566155","heartbeat_back_addr":"172.21.15.33:6807/2389566155","heartbeat_front_addr":"172.21.15.33:6805/2389566155","state":["exists","up"]},{"osd":5,"uuid":"9700d2f7-ed56-4143-ba9d-6cebe524e877","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":2821892844},{"type":"v1","addr":"172.21.15.110:6825","nonce":2821892844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":2821892844},{"type":"v1","addr":"172.21.15.110:6827","nonce":2821892844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":2821892844},{"type":"v1","addr":"172.21.15.110:6831","nonce":2821892844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":2821892844},{"type":"v1","addr":"172.21.15.110:6829","nonce":2821892844}]},"public_addr":"172.21.15.110:6825/2821892844","cluster_addr":"172.21.15.110:6827/2821892844","heartbeat_back_addr":"172.21.15.110:6831/2821892844","heartbeat_front_addr":"172.21.15.110:6829/2821892844","state":["exists","up"]},{"osd":6,"uuid":"49324695-d83b-41ad-92e1-5348e8e9149f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":273776428},{"type":"v1","addr":"172.21.15.118:6818","nonce":273776428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":273776428},{"type":"v1","addr":"172.21.15.118:6822","nonce":273776428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":273776428},{"type":"v1","addr":"172.21.15.118:6830","nonce":273776428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":273776428},{"type":"v1","addr":"172.21.15.118:6826","nonce":273776428}]},"public_addr":"172.21.15.118:6818/273776428","cluster_addr":"172.21.15.118:6822/273776428","heartbeat_back_addr":"172.21.15.118:6830/273776428","heartbeat_front_addr":"172.21.15.118:6826/273776428","state":["exists","up"]},{"osd":7,"uuid":"afef160d-0911-4406-ad49-cd6d61ee8a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1006872173},{"type":"v1","addr":"172.21.15.155:6827","nonce":1006872173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1006872173},{"type":"v1","addr":"172.21.15.155:6829","nonce":1006872173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1006872173},{"type":"v1","addr":"172.21.15.155:6833","nonce":1006872173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1006872173},{"type":"v1","addr":"172.21.15.155:6831","nonce":1006872173}]},"public_addr":"172.21.15.155:6827/1006872173","cluster_addr":"172.21.15.155:6829/1006872173","heartbeat_back_addr":"172.21.15.155:6833/1006872173","heartbeat_front_addr":"172.21.15.155:6831/1006872173","state":["exists","up"]},{"osd":8,"uuid":"298396bf-31a2-4832-91a4-55fcb050130f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":744651323},{"type":"v1","addr":"172.21.15.33:6817","nonce":744651323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":744651323},{"type":"v1","addr":"172.21.15.33:6819","nonce":744651323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":744651323},{"type":"v1","addr":"172.21.15.33:6823","nonce":744651323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":744651323},{"type":"v1","addr":"172.21.15.33:6821","nonce":744651323}]},"public_addr":"172.21.15.33:6817/744651323","cluster_addr":"172.21.15.33:6819/744651323","heartbeat_back_addr":"172.21.15.33:6823/744651323","heartbeat_front_addr":"172.21.15.33:6821/744651323","state":["exists","up"]},{"osd":9,"uuid":"c151ae6b-279d-4174-aceb-4671fbdac3b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":872519654},{"type":"v1","addr":"172.21.15.110:6802","nonce":872519654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":872519654},{"type":"v1","addr":"172.21.15.110:6807","nonce":872519654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":872519654},{"type":"v1","addr":"172.21.15.110:6819","nonce":872519654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":872519654},{"type":"v1","addr":"172.21.15.110:6813","nonce":872519654}]},"public_addr":"172.21.15.110:6802/872519654","cluster_addr":"172.21.15.110:6807/872519654","heartbeat_back_addr":"172.21.15.110:6819/872519654","heartbeat_front_addr":"172.21.15.110:6813/872519654","state":["exists","up"]},{"osd":10,"uuid":"0b292dfb-2029-4e1e-b206-ea7e0f6969ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6817","nonce":4099141917},{"type":"v1","addr":"172.21.15.118:6819","nonce":4099141917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6821","nonce":4099141917},{"type":"v1","addr":"172.21.15.118:6823","nonce":4099141917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6829","nonce":4099141917},{"type":"v1","addr":"172.21.15.118:6831","nonce":4099141917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":4099141917},{"type":"v1","addr":"172.21.15.118:6827","nonce":4099141917}]},"public_addr":"172.21.15.118:6819/4099141917","cluster_addr":"172.21.15.118:6823/4099141917","heartbeat_back_addr":"172.21.15.118:6831/4099141917","heartbeat_front_addr":"172.21.15.118:6827/4099141917","state":["exists","up"]},{"osd":11,"uuid":"10c61619-ad11-40f0-b013-9ef52bcabf59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":3981951559},{"type":"v1","addr":"172.21.15.155:6811","nonce":3981951559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":3981951559},{"type":"v1","addr":"172.21.15.155:6817","nonce":3981951559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":3981951559},{"type":"v1","addr":"172.21.15.155:6825","nonce":3981951559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3981951559},{"type":"v1","addr":"172.21.15.155:6823","nonce":3981951559}]},"public_addr":"172.21.15.155:6811/3981951559","cluster_addr":"172.21.15.155:6817/3981951559","heartbeat_back_addr":"172.21.15.155:6825/3981951559","heartbeat_front_addr":"172.21.15.155:6823/3981951559","state":["exists","up"]},{"osd":12,"uuid":"3f0d26e5-f3e2-47b6-91a8-0daa30d0544f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":4133738119},{"type":"v1","addr":"172.21.15.33:6809","nonce":4133738119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":4133738119},{"type":"v1","addr":"172.21.15.33:6811","nonce":4133738119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":4133738119},{"type":"v1","addr":"172.21.15.33:6815","nonce":4133738119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":4133738119},{"type":"v1","addr":"172.21.15.33:6813","nonce":4133738119}]},"public_addr":"172.21.15.33:6809/4133738119","cluster_addr":"172.21.15.33:6811/4133738119","heartbeat_back_addr":"172.21.15.33:6815/4133738119","heartbeat_front_addr":"172.21.15.33:6813/4133738119","state":["exists","up"]},{"osd":13,"uuid":"c8d6213f-af5e-4646-9241-6f91589b0e1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":1583232449},{"type":"v1","addr":"172.21.15.110:6803","nonce":1583232449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":1583232449},{"type":"v1","addr":"172.21.15.110:6809","nonce":1583232449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":1583232449},{"type":"v1","addr":"172.21.15.110:6821","nonce":1583232449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":1583232449},{"type":"v1","addr":"172.21.15.110:6815","nonce":1583232449}]},"public_addr":"172.21.15.110:6803/1583232449","cluster_addr":"172.21.15.110:6809/1583232449","heartbeat_back_addr":"172.21.15.110:6821/1583232449","heartbeat_front_addr":"172.21.15.110:6815/1583232449","state":["exists","up"]},{"osd":14,"uuid":"7b3d56df-62bf-4877-b984-6f17fe1fab33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":506416463},{"type":"v1","addr":"172.21.15.118:6801","nonce":506416463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":506416463},{"type":"v1","addr":"172.21.15.118:6803","nonce":506416463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":506416463},{"type":"v1","addr":"172.21.15.118:6807","nonce":506416463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":506416463},{"type":"v1","addr":"172.21.15.118:6805","nonce":506416463}]},"public_addr":"172.21.15.118:6801/506416463","cluster_addr":"172.21.15.118:6803/506416463","heartbeat_back_addr":"172.21.15.118:6807/506416463","heartbeat_front_addr":"172.21.15.118:6805/506416463","state":["exists","up"]},{"osd":15,"uuid":"6d526edb-f1d8-415a-ad55-81e810ff4944","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6807","nonce":741132602},{"type":"v1","addr":"172.21.15.155:6810","nonce":741132602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":741132602},{"type":"v1","addr":"172.21.15.155:6815","nonce":741132602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6821","nonce":741132602},{"type":"v1","addr":"172.21.15.155:6822","nonce":741132602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":741132602},{"type":"v1","addr":"172.21.15.155:6819","nonce":741132602}]},"public_addr":"172.21.15.155:6810/741132602","cluster_addr":"172.21.15.155:6815/741132602","heartbeat_back_addr":"172.21.15.155:6822/741132602","heartbeat_front_addr":"172.21.15.155:6819/741132602","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:05:00.394086+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:06:07.843641+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:06:06.531265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:06:18.377804+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:05:07.754528+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:05:03.768853+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:05:58.104022+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:06:18.042920+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:06:01.987377+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:06:12.890542+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:06:12.330281+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:06:10.632141+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:06:05.930654+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:05:59.307943+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:06:08.128740+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:05:06.403111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":36,"length":1},{"begin":39,"length":1},{"begin":41,"length":1},{"begin":43,"length":1},{"begin":46,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T01:06:20.330 INFO:tasks.ceph:Scrubbing osd.0 2024-04-19T01:06:20.330 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-19T01:06:20.495 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:06:20.496 INFO:teuthology.orchestra.run.smithi033.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-19T01:06:20.496 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:06:20.512 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-19T01:06:20.810 INFO:teuthology.orchestra.run.smithi033.stderr:instructed osd(s) 0 to deep-scrub 2024-04-19T01:06:20.828 INFO:tasks.ceph:Scrubbing osd.4 2024-04-19T01:06:20.829 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-19T01:06:20.981 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:06:20.981 INFO:teuthology.orchestra.run.smithi033.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-19T01:06:20.981 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:06:20.997 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-19T01:06:21.279 INFO:teuthology.orchestra.run.smithi033.stderr:instructed osd(s) 4 to deep-scrub 2024-04-19T01:06:21.296 INFO:tasks.ceph:Scrubbing osd.8 2024-04-19T01:06:21.297 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 config set osd_debug_deep_scrub_sleep 0 2024-04-19T01:06:21.440 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:06:21.440 INFO:teuthology.orchestra.run.smithi033.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-19T01:06:21.440 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:06:21.455 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 8 2024-04-19T01:06:21.778 INFO:teuthology.orchestra.run.smithi033.stderr:instructed osd(s) 8 to deep-scrub 2024-04-19T01:06:21.797 INFO:tasks.ceph:Scrubbing osd.12 2024-04-19T01:06:21.797 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 config set osd_debug_deep_scrub_sleep 0 2024-04-19T01:06:21.952 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:06:21.953 INFO:teuthology.orchestra.run.smithi033.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-19T01:06:21.953 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:06:21.968 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 12 2024-04-19T01:06:22.258 INFO:teuthology.orchestra.run.smithi033.stderr:instructed osd(s) 12 to deep-scrub 2024-04-19T01:06:22.275 INFO:tasks.ceph:Scrubbing osd.1 2024-04-19T01:06:22.276 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-19T01:06:22.419 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:06:22.419 INFO:teuthology.orchestra.run.smithi033.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-19T01:06:22.419 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:06:22.434 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-19T01:06:22.756 INFO:teuthology.orchestra.run.smithi033.stderr:instructed osd(s) 1 to deep-scrub 2024-04-19T01:06:22.775 INFO:tasks.ceph:Scrubbing osd.5 2024-04-19T01:06:22.775 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-19T01:06:23.135 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:06:23.135 INFO:teuthology.orchestra.run.smithi033.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-19T01:06:23.135 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:06:23.159 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-19T01:06:23.459 INFO:teuthology.orchestra.run.smithi033.stderr:instructed osd(s) 5 to deep-scrub 2024-04-19T01:06:23.477 INFO:tasks.ceph:Scrubbing osd.9 2024-04-19T01:06:23.477 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 config set osd_debug_deep_scrub_sleep 0 2024-04-19T01:06:23.623 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:06:23.623 INFO:teuthology.orchestra.run.smithi033.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-19T01:06:23.623 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:06:23.638 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 9 2024-04-19T01:06:23.926 INFO:teuthology.orchestra.run.smithi033.stderr:instructed osd(s) 9 to deep-scrub 2024-04-19T01:06:23.946 INFO:tasks.ceph:Scrubbing osd.13 2024-04-19T01:06:23.946 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 config set osd_debug_deep_scrub_sleep 0 2024-04-19T01:06:24.104 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:06:24.104 INFO:teuthology.orchestra.run.smithi033.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-19T01:06:24.104 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:06:24.121 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 13 2024-04-19T01:06:24.427 INFO:teuthology.orchestra.run.smithi033.stderr:instructed osd(s) 13 to deep-scrub 2024-04-19T01:06:24.447 INFO:tasks.ceph:Scrubbing osd.2 2024-04-19T01:06:24.447 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-19T01:06:24.608 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:06:24.608 INFO:teuthology.orchestra.run.smithi033.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-19T01:06:24.609 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:06:24.625 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-19T01:06:24.920 INFO:teuthology.orchestra.run.smithi033.stderr:instructed osd(s) 2 to deep-scrub 2024-04-19T01:06:24.940 INFO:tasks.ceph:Scrubbing osd.6 2024-04-19T01:06:24.940 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-04-19T01:06:25.095 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:06:25.096 INFO:teuthology.orchestra.run.smithi033.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-19T01:06:25.096 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:06:25.112 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-04-19T01:06:25.406 INFO:teuthology.orchestra.run.smithi033.stderr:instructed osd(s) 6 to deep-scrub 2024-04-19T01:06:25.426 INFO:tasks.ceph:Scrubbing osd.10 2024-04-19T01:06:25.426 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 config set osd_debug_deep_scrub_sleep 0 2024-04-19T01:06:25.582 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:06:25.582 INFO:teuthology.orchestra.run.smithi033.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-19T01:06:25.582 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:06:25.600 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 10 2024-04-19T01:06:25.895 INFO:teuthology.orchestra.run.smithi033.stderr:instructed osd(s) 10 to deep-scrub 2024-04-19T01:06:25.916 INFO:tasks.ceph:Scrubbing osd.14 2024-04-19T01:06:25.916 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 config set osd_debug_deep_scrub_sleep 0 2024-04-19T01:06:26.076 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:06:26.076 INFO:teuthology.orchestra.run.smithi033.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-19T01:06:26.076 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:06:26.095 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 14 2024-04-19T01:06:26.382 INFO:teuthology.orchestra.run.smithi033.stderr:instructed osd(s) 14 to deep-scrub 2024-04-19T01:06:26.400 INFO:tasks.ceph:Scrubbing osd.3 2024-04-19T01:06:26.401 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-19T01:06:26.556 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:06:26.556 INFO:teuthology.orchestra.run.smithi033.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-19T01:06:26.556 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:06:26.571 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-19T01:06:26.862 INFO:teuthology.orchestra.run.smithi033.stderr:instructed osd(s) 3 to deep-scrub 2024-04-19T01:06:26.881 INFO:tasks.ceph:Scrubbing osd.7 2024-04-19T01:06:26.881 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-04-19T01:06:27.040 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:06:27.041 INFO:teuthology.orchestra.run.smithi033.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-19T01:06:27.041 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:06:27.056 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-04-19T01:06:27.336 INFO:teuthology.orchestra.run.smithi033.stderr:instructed osd(s) 7 to deep-scrub 2024-04-19T01:06:27.353 INFO:tasks.ceph:Scrubbing osd.11 2024-04-19T01:06:27.353 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 config set osd_debug_deep_scrub_sleep 0 2024-04-19T01:06:27.496 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:06:27.497 INFO:teuthology.orchestra.run.smithi033.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-19T01:06:27.497 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:06:27.512 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 11 2024-04-19T01:06:27.796 INFO:teuthology.orchestra.run.smithi033.stderr:instructed osd(s) 11 to deep-scrub 2024-04-19T01:06:27.814 INFO:tasks.ceph:Scrubbing osd.15 2024-04-19T01:06:27.814 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 config set osd_debug_deep_scrub_sleep 0 2024-04-19T01:06:27.969 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-19T01:06:27.969 INFO:teuthology.orchestra.run.smithi033.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-19T01:06:27.969 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-19T01:06:27.986 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 15 2024-04-19T01:06:28.267 INFO:teuthology.orchestra.run.smithi033.stderr:instructed osd(s) 15 to deep-scrub 2024-04-19T01:06:28.285 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T01:06:28.575 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:06:28.576 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-04-19T01:06:28.594 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":228,"stamp":"2024-04-19T01:06:27.566356+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":13,"kb":1509949440,"kb_used":563636,"kb_used_data":30080,"kb_used_omap":24,"kb_used_meta":533351,"kb_avail":1509385804,"statfs":{"total":1546188226560,"available":1545611063296,"internally_reserved":0,"allocated":30801920,"data_stored":24694456,"data_compressed":4495130,"data_compressed_allocated":9183232,"data_compressed_original":19259392,"omap_allocated":25520,"internal_metadata":546151504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.491920"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":297,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-04-19T01:06:23.638137+0000","last_change":"2024-04-19T01:06:23.638158+0000","last_active":"2024-04-19T01:06:23.638137+0000","last_peered":"2024-04-19T01:06:23.638137+0000","last_clean":"2024-04-19T01:06:23.638137+0000","last_became_active":"2024-04-19T01:03:53.938774+0000","last_became_peered":"2024-04-19T01:03:53.938774+0000","last_unstale":"2024-04-19T01:06:23.638137+0000","last_undegraded":"2024-04-19T01:06:23.638137+0000","last_fullsized":"2024-04-19T01:06:23.638137+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:06:23.638065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:06:23.638065+0000","last_clean_scrub_stamp":"2024-04-19T01:06:23.638065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:07:38.819562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":256,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-19T01:06:26.427321+0000","last_change":"2024-04-19T01:06:26.427376+0000","last_active":"2024-04-19T01:06:26.427321+0000","last_peered":"2024-04-19T01:06:26.427321+0000","last_clean":"2024-04-19T01:06:26.427321+0000","last_became_active":"2024-04-19T01:06:07.457936+0000","last_became_peered":"2024-04-19T01:06:07.457936+0000","last_unstale":"2024-04-19T01:06:26.427321+0000","last_undegraded":"2024-04-19T01:06:26.427321+0000","last_fullsized":"2024-04-19T01:06:26.427321+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:06:26.427149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:06:26.427149+0000","last_clean_scrub_stamp":"2024-04-19T01:06:26.427149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:07:32.108208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13],"acting":[14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":297,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-19T01:06:26.088058+0000","last_change":"2024-04-19T01:06:24.157982+0000","last_active":"2024-04-19T01:06:26.088058+0000","last_peered":"2024-04-19T01:06:26.088058+0000","last_clean":"2024-04-19T01:06:26.088058+0000","last_became_active":"2024-04-19T01:03:53.771887+0000","last_became_peered":"2024-04-19T01:03:53.771887+0000","last_unstale":"2024-04-19T01:06:26.088058+0000","last_undegraded":"2024-04-19T01:06:26.088058+0000","last_fullsized":"2024-04-19T01:06:26.088058+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:06:24.157731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:06:24.157731+0000","last_clean_scrub_stamp":"2024-04-19T01:06:24.157731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:07:33.611012+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":286,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-04-19T01:06:19.258373+0000","last_change":"2024-04-19T01:05:10.593158+0000","last_active":"2024-04-19T01:06:19.258373+0000","last_peered":"2024-04-19T01:06:19.258373+0000","last_clean":"2024-04-19T01:06:19.258373+0000","last_became_active":"2024-04-19T01:03:53.716040+0000","last_became_peered":"2024-04-19T01:03:53.716040+0000","last_unstale":"2024-04-19T01:06:19.258373+0000","last_undegraded":"2024-04-19T01:06:19.258373+0000","last_fullsized":"2024-04-19T01:06:19.258373+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:10.592906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:10.592906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:06:39.747266+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":297,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-04-19T01:06:19.086066+0000","last_change":"2024-04-19T01:06:12.724217+0000","last_active":"2024-04-19T01:06:19.086066+0000","last_peered":"2024-04-19T01:06:19.086066+0000","last_clean":"2024-04-19T01:06:19.086066+0000","last_became_active":"2024-04-19T01:03:54.011460+0000","last_became_peered":"2024-04-19T01:03:54.011460+0000","last_unstale":"2024-04-19T01:06:19.086066+0000","last_undegraded":"2024-04-19T01:06:19.086066+0000","last_fullsized":"2024-04-19T01:06:19.086066+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:06:12.723991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:06:12.723991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":285,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-04-19T01:06:19.259515+0000","last_change":"2024-04-19T01:05:43.353831+0000","last_active":"2024-04-19T01:06:19.259515+0000","last_peered":"2024-04-19T01:06:19.259515+0000","last_clean":"2024-04-19T01:06:19.259515+0000","last_became_active":"2024-04-19T01:03:53.649606+0000","last_became_peered":"2024-04-19T01:03:53.649606+0000","last_unstale":"2024-04-19T01:06:19.259515+0000","last_undegraded":"2024-04-19T01:06:19.259515+0000","last_fullsized":"2024-04-19T01:06:19.259515+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:43.353589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:05:43.353589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:07:12.784639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":297,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-04-19T01:06:22.622076+0000","last_change":"2024-04-19T01:06:22.622113+0000","last_active":"2024-04-19T01:06:22.622076+0000","last_peered":"2024-04-19T01:06:22.622076+0000","last_clean":"2024-04-19T01:06:22.622076+0000","last_became_active":"2024-04-19T01:03:53.759326+0000","last_became_peered":"2024-04-19T01:03:53.759326+0000","last_unstale":"2024-04-19T01:06:22.622076+0000","last_undegraded":"2024-04-19T01:06:22.622076+0000","last_fullsized":"2024-04-19T01:06:22.622076+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:06:22.621979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:06:22.621979+0000","last_clean_scrub_stamp":"2024-04-19T01:06:22.621979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:07:45.347624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":300,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-04-19T01:06:19.596216+0000","last_change":"2024-04-19T01:06:19.596326+0000","last_active":"2024-04-19T01:06:19.596216+0000","last_peered":"2024-04-19T01:06:19.596216+0000","last_clean":"2024-04-19T01:06:19.596216+0000","last_became_active":"2024-04-19T01:05:44.095001+0000","last_became_peered":"2024-04-19T01:05:44.095001+0000","last_unstale":"2024-04-19T01:06:19.596216+0000","last_undegraded":"2024-04-19T01:06:19.596216+0000","last_fullsized":"2024-04-19T01:06:19.596216+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:06:19.596003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:03:52.624648+0000","last_clean_scrub_stamp":"2024-04-19T01:06:19.596003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:07:28.979858+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"159'159","reported_seq":495,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-04-19T01:06:19.259769+0000","last_change":"2024-04-19T01:06:12.605622+0000","last_active":"2024-04-19T01:06:19.259769+0000","last_peered":"2024-04-19T01:06:19.259769+0000","last_clean":"2024-04-19T01:06:19.259769+0000","last_became_active":"2024-04-19T01:03:50.774078+0000","last_became_peered":"2024-04-19T01:03:50.774078+0000","last_unstale":"2024-04-19T01:06:19.259769+0000","last_undegraded":"2024-04-19T01:06:19.259769+0000","last_fullsized":"2024-04-19T01:06:19.259769+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'159","last_scrub_stamp":"2024-04-19T01:06:12.605370+0000","last_deep_scrub":"159'159","last_deep_scrub_stamp":"2024-04-19T01:06:12.605370+0000","last_clean_scrub_stamp":"2024-04-19T01:06:12.605370+0000","objects_scrubbed":2,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:07:19.942749+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":13},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":15546,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":14,"up_from":24,"seq":103079215140,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37224,"kb_used_data":1884,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334616,"statfs":{"total":96636764160,"available":96598646784,"internally_reserved":0,"allocated":1929216,"data_stored":1485768,"data_compressed":281752,"data_compressed_allocated":573440,"data_compressed_original":1146880,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,5,6,7,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":24,"seq":103079215141,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37212,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334628,"statfs":{"total":96636764160,"available":96598659072,"internally_reserved":0,"allocated":1916928,"data_stored":1476076,"data_compressed":279720,"data_compressed_allocated":569344,"data_compressed_original":1138688,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[2,6,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":24,"seq":103079215142,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37224,"kb_used_data":1884,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334616,"statfs":{"total":96636764160,"available":96598646784,"internally_reserved":0,"allocated":1929216,"data_stored":1485768,"data_compressed":281752,"data_compressed_allocated":573440,"data_compressed_original":1146880,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[1,2,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313252,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33052,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338788,"statfs":{"total":96636764160,"available":96602918912,"internally_reserved":0,"allocated":1916928,"data_stored":1476076,"data_compressed":279720,"data_compressed_allocated":569344,"data_compressed_original":1138688,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[5,6,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33052,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338788,"statfs":{"total":96636764160,"available":96602918912,"internally_reserved":0,"allocated":1916928,"data_stored":1476076,"data_compressed":279720,"data_compressed_allocated":569344,"data_compressed_original":1138688,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,3,4,5,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280548,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33052,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338788,"statfs":{"total":96636764160,"available":96602918912,"internally_reserved":0,"allocated":1916928,"data_stored":1476076,"data_compressed":279720,"data_compressed_allocated":569344,"data_compressed_original":1138688,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,6,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280547,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33052,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338788,"statfs":{"total":96636764160,"available":96602918912,"internally_reserved":0,"allocated":1916928,"data_stored":1476076,"data_compressed":279720,"data_compressed_allocated":569344,"data_compressed_original":1138688,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[2,4,5,6,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280547,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33052,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338788,"statfs":{"total":96636764160,"available":96602918912,"internally_reserved":0,"allocated":1916928,"data_stored":1476076,"data_compressed":279720,"data_compressed_allocated":569344,"data_compressed_original":1138688,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280547,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33104,"kb_used_data":1924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338736,"statfs":{"total":96636764160,"available":96602865664,"internally_reserved":0,"allocated":1970176,"data_stored":2005004,"data_compressed":287493,"data_compressed_allocated":602112,"data_compressed_original":1650688,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,6,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280548,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33048,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338792,"statfs":{"total":96636764160,"available":96602923008,"internally_reserved":0,"allocated":1916928,"data_stored":1476076,"data_compressed":279720,"data_compressed_allocated":569344,"data_compressed_original":1138688,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,5,6,7,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280546,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51148,"kb_used_data":1924,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320692,"statfs":{"total":96636764160,"available":96584388608,"internally_reserved":0,"allocated":1970176,"data_stored":2005004,"data_compressed":287493,"data_compressed_allocated":602112,"data_compressed_original":1650688,"omap_allocated":1596,"internal_metadata":50395588},"hb_peers":[0,5,6,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280546,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33052,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338788,"statfs":{"total":96636764160,"available":96602918912,"internally_reserved":0,"allocated":1916928,"data_stored":1476076,"data_compressed":279720,"data_compressed_allocated":569344,"data_compressed_original":1138688,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,6,7,8,9,10,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280547,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33052,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338788,"statfs":{"total":96636764160,"available":96602918912,"internally_reserved":0,"allocated":1916928,"data_stored":1476076,"data_compressed":279720,"data_compressed_allocated":569344,"data_compressed_original":1138688,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":24,"seq":103079215140,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37212,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334628,"statfs":{"total":96636764160,"available":96598659072,"internally_reserved":0,"allocated":1916928,"data_stored":1476076,"data_compressed":279720,"data_compressed_allocated":569344,"data_compressed_original":1138688,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215141,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33048,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338792,"statfs":{"total":96636764160,"available":96602923008,"internally_reserved":0,"allocated":1916928,"data_stored":1476076,"data_compressed":279720,"data_compressed_allocated":569344,"data_compressed_original":1138688,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":24,"seq":103079215140,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33052,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338788,"statfs":{"total":96636764160,"available":96602918912,"internally_reserved":0,"allocated":1916928,"data_stored":1476076,"data_compressed":279720,"data_compressed_allocated":569344,"data_compressed_original":1138688,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T01:06:28.602 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-04-19T01:05:10.592906+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=10, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=6, tm_sec=19, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:06:28.602 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-19T01:06:12.723991+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=6, tm_sec=12, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=6, tm_sec=19, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:06:28.602 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-19T01:05:43.353589+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=43, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=6, tm_sec=19, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:06:28.603 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-19T01:06:19.596003+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=6, tm_sec=19, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=6, tm_sec=19, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:06:28.603 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-19T01:06:12.605370+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=6, tm_sec=12, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=6, tm_sec=19, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:06:28.603 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-19T01:06:48.604 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T01:06:48.922 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-19T01:06:48.923 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-04-19T01:06:48.940 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":241,"stamp":"2024-04-19T01:06:47.572315+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":13,"kb":1509949440,"kb_used":564392,"kb_used_data":30824,"kb_used_omap":24,"kb_used_meta":533351,"kb_avail":1509385048,"statfs":{"total":1546188226560,"available":1545610289152,"internally_reserved":0,"allocated":31563776,"data_stored":25295360,"data_compressed":4620986,"data_compressed_allocated":9437184,"data_compressed_original":19767296,"omap_allocated":25520,"internal_metadata":546151504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.939260"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":305,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-04-19T01:06:36.391335+0000","last_change":"2024-04-19T01:06:23.638158+0000","last_active":"2024-04-19T01:06:36.391335+0000","last_peered":"2024-04-19T01:06:36.391335+0000","last_clean":"2024-04-19T01:06:36.391335+0000","last_became_active":"2024-04-19T01:03:53.938774+0000","last_became_peered":"2024-04-19T01:03:53.938774+0000","last_unstale":"2024-04-19T01:06:36.391335+0000","last_undegraded":"2024-04-19T01:06:36.391335+0000","last_fullsized":"2024-04-19T01:06:36.391335+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:06:23.638065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:06:23.638065+0000","last_clean_scrub_stamp":"2024-04-19T01:06:23.638065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:07:38.819562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":262,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-04-19T01:06:36.451613+0000","last_change":"2024-04-19T01:06:26.427376+0000","last_active":"2024-04-19T01:06:36.451613+0000","last_peered":"2024-04-19T01:06:36.451613+0000","last_clean":"2024-04-19T01:06:36.451613+0000","last_became_active":"2024-04-19T01:06:07.457936+0000","last_became_peered":"2024-04-19T01:06:07.457936+0000","last_unstale":"2024-04-19T01:06:36.451613+0000","last_undegraded":"2024-04-19T01:06:36.451613+0000","last_fullsized":"2024-04-19T01:06:36.451613+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:06:26.427149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:06:26.427149+0000","last_clean_scrub_stamp":"2024-04-19T01:06:26.427149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:07:32.108208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13],"acting":[14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":303,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-04-19T01:06:36.450855+0000","last_change":"2024-04-19T01:06:24.157982+0000","last_active":"2024-04-19T01:06:36.450855+0000","last_peered":"2024-04-19T01:06:36.450855+0000","last_clean":"2024-04-19T01:06:36.450855+0000","last_became_active":"2024-04-19T01:03:53.771887+0000","last_became_peered":"2024-04-19T01:03:53.771887+0000","last_unstale":"2024-04-19T01:06:36.450855+0000","last_undegraded":"2024-04-19T01:06:36.450855+0000","last_fullsized":"2024-04-19T01:06:36.450855+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:06:24.157731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:06:24.157731+0000","last_clean_scrub_stamp":"2024-04-19T01:06:24.157731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:07:33.611012+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":303,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-04-19T01:06:36.450674+0000","last_change":"2024-04-19T01:06:25.472137+0000","last_active":"2024-04-19T01:06:36.450674+0000","last_peered":"2024-04-19T01:06:36.450674+0000","last_clean":"2024-04-19T01:06:36.450674+0000","last_became_active":"2024-04-19T01:03:53.716040+0000","last_became_peered":"2024-04-19T01:03:53.716040+0000","last_unstale":"2024-04-19T01:06:36.450674+0000","last_undegraded":"2024-04-19T01:06:36.450674+0000","last_fullsized":"2024-04-19T01:06:36.450674+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:06:25.471888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:06:25.471888+0000","last_clean_scrub_stamp":"2024-04-19T01:06:25.471888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:07:46.007239+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":313,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-04-19T01:06:36.391330+0000","last_change":"2024-04-19T01:06:24.674923+0000","last_active":"2024-04-19T01:06:36.391330+0000","last_peered":"2024-04-19T01:06:36.391330+0000","last_clean":"2024-04-19T01:06:36.391330+0000","last_became_active":"2024-04-19T01:03:54.011460+0000","last_became_peered":"2024-04-19T01:03:54.011460+0000","last_unstale":"2024-04-19T01:06:36.391330+0000","last_undegraded":"2024-04-19T01:06:36.391330+0000","last_fullsized":"2024-04-19T01:06:36.391330+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:06:24.674809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:06:24.674809+0000","last_clean_scrub_stamp":"2024-04-19T01:06:24.674809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:07:30.208537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":302,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-04-19T01:06:35.636996+0000","last_change":"2024-04-19T01:06:28.479488+0000","last_active":"2024-04-19T01:06:35.636996+0000","last_peered":"2024-04-19T01:06:35.636996+0000","last_clean":"2024-04-19T01:06:35.636996+0000","last_became_active":"2024-04-19T01:03:53.649606+0000","last_became_peered":"2024-04-19T01:03:53.649606+0000","last_unstale":"2024-04-19T01:06:35.636996+0000","last_undegraded":"2024-04-19T01:06:35.636996+0000","last_fullsized":"2024-04-19T01:06:35.636996+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:06:28.479262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:06:28.479262+0000","last_clean_scrub_stamp":"2024-04-19T01:06:28.479262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:07:30.418750+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":305,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-04-19T01:06:36.391342+0000","last_change":"2024-04-19T01:06:22.622113+0000","last_active":"2024-04-19T01:06:36.391342+0000","last_peered":"2024-04-19T01:06:36.391342+0000","last_clean":"2024-04-19T01:06:36.391342+0000","last_became_active":"2024-04-19T01:03:53.759326+0000","last_became_peered":"2024-04-19T01:03:53.759326+0000","last_unstale":"2024-04-19T01:06:36.391342+0000","last_undegraded":"2024-04-19T01:06:36.391342+0000","last_fullsized":"2024-04-19T01:06:36.391342+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:06:22.621979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:06:22.621979+0000","last_clean_scrub_stamp":"2024-04-19T01:06:22.621979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:07:45.347624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":317,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-04-19T01:06:36.452625+0000","last_change":"2024-04-19T01:06:29.470158+0000","last_active":"2024-04-19T01:06:36.452625+0000","last_peered":"2024-04-19T01:06:36.452625+0000","last_clean":"2024-04-19T01:06:36.452625+0000","last_became_active":"2024-04-19T01:05:44.095001+0000","last_became_peered":"2024-04-19T01:05:44.095001+0000","last_unstale":"2024-04-19T01:06:36.452625+0000","last_undegraded":"2024-04-19T01:06:36.452625+0000","last_fullsized":"2024-04-19T01:06:36.452625+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:06:29.469908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:06:29.469908+0000","last_clean_scrub_stamp":"2024-04-19T01:06:29.469908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:07:38.972332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"159'159","reported_seq":512,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-04-19T01:06:36.452595+0000","last_change":"2024-04-19T01:06:28.514476+0000","last_active":"2024-04-19T01:06:36.452595+0000","last_peered":"2024-04-19T01:06:36.452595+0000","last_clean":"2024-04-19T01:06:36.452595+0000","last_became_active":"2024-04-19T01:03:50.774078+0000","last_became_peered":"2024-04-19T01:03:50.774078+0000","last_unstale":"2024-04-19T01:06:36.452595+0000","last_undegraded":"2024-04-19T01:06:36.452595+0000","last_fullsized":"2024-04-19T01:06:36.452595+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'159","last_scrub_stamp":"2024-04-19T01:06:28.514210+0000","last_deep_scrub":"159'159","last_deep_scrub_stamp":"2024-04-19T01:06:28.514210+0000","last_clean_scrub_stamp":"2024-04-19T01:06:28.514210+0000","objects_scrubbed":2,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:07:28.692218+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":13},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":15546,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":14,"up_from":24,"seq":103079215144,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37260,"kb_used_data":1920,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334580,"statfs":{"total":96636764160,"available":96598609920,"internally_reserved":0,"allocated":1966080,"data_stored":1514844,"data_compressed":287840,"data_compressed_allocated":585728,"data_compressed_original":1171456,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,5,6,7,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":24,"seq":103079215145,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37260,"kb_used_data":1920,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334580,"statfs":{"total":96636764160,"available":96598609920,"internally_reserved":0,"allocated":1966080,"data_stored":1514844,"data_compressed":287840,"data_compressed_allocated":585728,"data_compressed_original":1171456,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[2,6,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":24,"seq":103079215146,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37260,"kb_used_data":1920,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334580,"statfs":{"total":96636764160,"available":96598609920,"internally_reserved":0,"allocated":1966080,"data_stored":1514844,"data_compressed":287840,"data_compressed_allocated":585728,"data_compressed_original":1171456,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[1,2,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313256,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33100,"kb_used_data":1920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338740,"statfs":{"total":96636764160,"available":96602869760,"internally_reserved":0,"allocated":1966080,"data_stored":1514844,"data_compressed":287840,"data_compressed_allocated":585728,"data_compressed_original":1171456,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[5,6,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280550,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33100,"kb_used_data":1920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338740,"statfs":{"total":96636764160,"available":96602869760,"internally_reserved":0,"allocated":1966080,"data_stored":1514844,"data_compressed":287840,"data_compressed_allocated":585728,"data_compressed_original":1171456,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,3,4,5,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280552,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33100,"kb_used_data":1920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338740,"statfs":{"total":96636764160,"available":96602869760,"internally_reserved":0,"allocated":1966080,"data_stored":1514844,"data_compressed":287840,"data_compressed_allocated":585728,"data_compressed_original":1171456,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,6,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280551,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33100,"kb_used_data":1920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338740,"statfs":{"total":96636764160,"available":96602869760,"internally_reserved":0,"allocated":1966080,"data_stored":1514844,"data_compressed":287840,"data_compressed_allocated":585728,"data_compressed_original":1171456,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[2,4,5,6,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280551,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33100,"kb_used_data":1920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338740,"statfs":{"total":96636764160,"available":96602869760,"internally_reserved":0,"allocated":1966080,"data_stored":1514844,"data_compressed":287840,"data_compressed_allocated":585728,"data_compressed_original":1171456,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280551,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33152,"kb_used_data":1972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338688,"statfs":{"total":96636764160,"available":96602816512,"internally_reserved":0,"allocated":2019328,"data_stored":2043772,"data_compressed":295613,"data_compressed_allocated":618496,"data_compressed_original":1683456,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,6,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280552,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33100,"kb_used_data":1920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338740,"statfs":{"total":96636764160,"available":96602869760,"internally_reserved":0,"allocated":1966080,"data_stored":1514844,"data_compressed":287840,"data_compressed_allocated":585728,"data_compressed_original":1171456,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,5,6,7,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280550,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51200,"kb_used_data":1972,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320640,"statfs":{"total":96636764160,"available":96584335360,"internally_reserved":0,"allocated":2019328,"data_stored":2043772,"data_compressed":295613,"data_compressed_allocated":618496,"data_compressed_original":1683456,"omap_allocated":1596,"internal_metadata":50395588},"hb_peers":[0,5,6,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280550,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33100,"kb_used_data":1920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338740,"statfs":{"total":96636764160,"available":96602869760,"internally_reserved":0,"allocated":1966080,"data_stored":1514844,"data_compressed":287840,"data_compressed_allocated":585728,"data_compressed_original":1171456,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,6,7,8,9,10,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280551,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33100,"kb_used_data":1920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338740,"statfs":{"total":96636764160,"available":96602869760,"internally_reserved":0,"allocated":1966080,"data_stored":1514844,"data_compressed":287840,"data_compressed_allocated":585728,"data_compressed_original":1171456,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":24,"seq":103079215144,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37260,"kb_used_data":1920,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334580,"statfs":{"total":96636764160,"available":96598609920,"internally_reserved":0,"allocated":1966080,"data_stored":1514844,"data_compressed":287840,"data_compressed_allocated":585728,"data_compressed_original":1171456,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":24,"seq":103079215145,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33100,"kb_used_data":1920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338740,"statfs":{"total":96636764160,"available":96602869760,"internally_reserved":0,"allocated":1966080,"data_stored":1514844,"data_compressed":287840,"data_compressed_allocated":585728,"data_compressed_original":1171456,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":24,"seq":103079215144,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33100,"kb_used_data":1920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338740,"statfs":{"total":96636764160,"available":96602869760,"internally_reserved":0,"allocated":1966080,"data_stored":1514844,"data_compressed":287840,"data_compressed_allocated":585728,"data_compressed_original":1171456,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7773,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T01:06:48.942 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-19T01:06:49.272 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-19T01:06:49.272 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-19T01:06:49.272 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-19T01:06:49.272 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:06:49.345 INFO:tasks.ceph.osd.0:Stopped 2024-04-19T01:06:49.345 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-19T01:06:49.345 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:06:49.419 INFO:tasks.ceph.osd.4:Stopped 2024-04-19T01:06:49.420 DEBUG:tasks.ceph.osd.8:waiting for process to exit 2024-04-19T01:06:49.420 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:06:49.498 INFO:tasks.ceph.osd.8:Stopped 2024-04-19T01:06:49.498 DEBUG:tasks.ceph.osd.12:waiting for process to exit 2024-04-19T01:06:49.498 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:06:49.584 INFO:tasks.ceph.osd.12:Stopped 2024-04-19T01:06:49.584 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-19T01:06:49.584 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:06:49.669 INFO:tasks.ceph.osd.1:Stopped 2024-04-19T01:06:49.669 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-19T01:06:49.669 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:06:49.767 INFO:tasks.ceph.osd.5:Stopped 2024-04-19T01:06:49.767 DEBUG:tasks.ceph.osd.9:waiting for process to exit 2024-04-19T01:06:49.767 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:06:49.846 INFO:tasks.ceph.osd.9:Stopped 2024-04-19T01:06:49.847 DEBUG:tasks.ceph.osd.13:waiting for process to exit 2024-04-19T01:06:49.847 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:06:55.191 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.13 is failed for ~0s 2024-04-19T01:06:55.949 INFO:tasks.ceph.osd.13:Stopped 2024-04-19T01:06:55.949 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-19T01:06:55.950 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:06:56.036 INFO:tasks.ceph.osd.2:Stopped 2024-04-19T01:06:56.037 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-04-19T01:06:56.037 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:06:56.120 INFO:tasks.ceph.osd.6:Stopped 2024-04-19T01:06:56.120 DEBUG:tasks.ceph.osd.10:waiting for process to exit 2024-04-19T01:06:56.120 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:06:56.210 INFO:tasks.ceph.osd.10:Stopped 2024-04-19T01:06:56.211 DEBUG:tasks.ceph.osd.14:waiting for process to exit 2024-04-19T01:06:56.211 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:07:01.097 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.14 is failed for ~0s 2024-04-19T01:07:01.097 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.13 has been restored 2024-04-19T01:07:02.312 INFO:tasks.ceph.osd.14:Stopped 2024-04-19T01:07:02.313 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-19T01:07:02.313 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:07:02.394 INFO:tasks.ceph.osd.3:Stopped 2024-04-19T01:07:02.395 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-04-19T01:07:02.395 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:07:02.479 INFO:tasks.ceph.osd.7:Stopped 2024-04-19T01:07:02.479 DEBUG:tasks.ceph.osd.11:waiting for process to exit 2024-04-19T01:07:02.479 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:07:02.565 INFO:tasks.ceph.osd.11:Stopped 2024-04-19T01:07:02.565 DEBUG:tasks.ceph.osd.15:waiting for process to exit 2024-04-19T01:07:02.565 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:07:02.622 INFO:tasks.ceph.osd.15:Stopped 2024-04-19T01:07:02.622 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-19T01:07:02.623 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-04-19T01:07:02.623 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:07:02.652 INFO:tasks.ceph.mgr.y:Stopped 2024-04-19T01:07:02.652 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-19T01:07:02.652 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:07:02.684 INFO:tasks.ceph.mgr.x:Stopped 2024-04-19T01:07:02.685 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-19T01:07:02.685 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-19T01:07:02.685 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:07:02.708 INFO:tasks.ceph.mon.a:Stopped 2024-04-19T01:07:02.708 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-19T01:07:02.708 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:07:02.733 INFO:tasks.ceph.mon.b:Stopped 2024-04-19T01:07:02.733 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-19T01:07:02.733 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:07:02.759 INFO:tasks.ceph.mon.c:Stopped 2024-04-19T01:07:02.759 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-19T01:07:02.760 DEBUG:teuthology.orchestra.run.smithi033:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'slow request' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-04-19T01:07:02.779 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi033.front.sepia.ceph.com 2024-04-19T01:07:02.779 DEBUG:teuthology.orchestra.run.smithi033:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-19T01:07:03.035 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi033.front.sepia.ceph.com 2024-04-19T01:07:03.036 DEBUG:teuthology.orchestra.run.smithi033:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-19T01:07:03.191 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-8 on ubuntu@smithi033.front.sepia.ceph.com 2024-04-19T01:07:03.191 DEBUG:teuthology.orchestra.run.smithi033:> sync && sudo umount -f /var/lib/ceph/osd/ceph-8 2024-04-19T01:07:03.298 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-12 on ubuntu@smithi033.front.sepia.ceph.com 2024-04-19T01:07:03.298 DEBUG:teuthology.orchestra.run.smithi033:> sync && sudo umount -f /var/lib/ceph/osd/ceph-12 2024-04-19T01:07:03.434 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi110.front.sepia.ceph.com 2024-04-19T01:07:03.434 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-19T01:07:03.642 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi110.front.sepia.ceph.com 2024-04-19T01:07:03.642 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-19T01:07:03.823 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-9 on ubuntu@smithi110.front.sepia.ceph.com 2024-04-19T01:07:03.824 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-9 2024-04-19T01:07:04.023 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-13 on ubuntu@smithi110.front.sepia.ceph.com 2024-04-19T01:07:04.023 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-13 2024-04-19T01:07:04.188 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi118.front.sepia.ceph.com 2024-04-19T01:07:04.188 DEBUG:teuthology.orchestra.run.smithi118:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-19T01:07:04.477 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi118.front.sepia.ceph.com 2024-04-19T01:07:04.478 DEBUG:teuthology.orchestra.run.smithi118:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-04-19T01:07:04.687 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-10 on ubuntu@smithi118.front.sepia.ceph.com 2024-04-19T01:07:04.687 DEBUG:teuthology.orchestra.run.smithi118:> sync && sudo umount -f /var/lib/ceph/osd/ceph-10 2024-04-19T01:07:04.872 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-14 on ubuntu@smithi118.front.sepia.ceph.com 2024-04-19T01:07:04.872 DEBUG:teuthology.orchestra.run.smithi118:> sync && sudo umount -f /var/lib/ceph/osd/ceph-14 2024-04-19T01:07:05.063 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi155.front.sepia.ceph.com 2024-04-19T01:07:05.064 DEBUG:teuthology.orchestra.run.smithi155:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-19T01:07:05.284 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi155.front.sepia.ceph.com 2024-04-19T01:07:05.285 DEBUG:teuthology.orchestra.run.smithi155:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-04-19T01:07:05.482 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-11 on ubuntu@smithi155.front.sepia.ceph.com 2024-04-19T01:07:05.483 DEBUG:teuthology.orchestra.run.smithi155:> sync && sudo umount -f /var/lib/ceph/osd/ceph-11 2024-04-19T01:07:05.674 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-15 on ubuntu@smithi155.front.sepia.ceph.com 2024-04-19T01:07:05.674 DEBUG:teuthology.orchestra.run.smithi155:> sync && sudo umount -f /var/lib/ceph/osd/ceph-15 2024-04-19T01:07:05.840 INFO:tasks.ceph:Archiving mon data... 2024-04-19T01:07:05.842 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663223/data/mon.a.tgz 2024-04-19T01:07:05.842 DEBUG:teuthology.orchestra.run.smithi033:> mktemp 2024-04-19T01:07:05.848 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/tmp.kUlW51saGV 2024-04-19T01:07:05.849 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.kUlW51saGV 2024-04-19T01:07:06.010 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0666 /tmp/tmp.kUlW51saGV 2024-04-19T01:07:06.078 DEBUG:teuthology.orchestra.remote:smithi033:/tmp/tmp.kUlW51saGV is 469KB 2024-04-19T01:07:06.100 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.14 has been restored 2024-04-19T01:07:06.147 DEBUG:teuthology.orchestra.run.smithi033:> rm -fr /tmp/tmp.kUlW51saGV 2024-04-19T01:07:06.153 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663223/data/mon.b.tgz 2024-04-19T01:07:06.153 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2024-04-19T01:07:06.159 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.iCEGa1x5li 2024-04-19T01:07:06.159 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.iCEGa1x5li 2024-04-19T01:07:06.355 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.iCEGa1x5li 2024-04-19T01:07:06.420 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.iCEGa1x5li is 651KB 2024-04-19T01:07:06.441 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.iCEGa1x5li 2024-04-19T01:07:06.447 DEBUG:teuthology.misc:Transferring archived files from smithi118:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663223/data/mon.c.tgz 2024-04-19T01:07:06.447 DEBUG:teuthology.orchestra.run.smithi118:> mktemp 2024-04-19T01:07:06.453 INFO:teuthology.orchestra.run.smithi118.stdout:/tmp/tmp.2qCmzydo5u 2024-04-19T01:07:06.453 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.2qCmzydo5u 2024-04-19T01:07:06.651 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0666 /tmp/tmp.2qCmzydo5u 2024-04-19T01:07:06.716 DEBUG:teuthology.orchestra.remote:smithi118:/tmp/tmp.2qCmzydo5u is 651KB 2024-04-19T01:07:06.785 DEBUG:teuthology.orchestra.run.smithi118:> rm -fr /tmp/tmp.2qCmzydo5u 2024-04-19T01:07:06.790 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-19T01:07:06.790 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-19T01:07:06.793 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-19T01:07:06.796 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-19T01:07:06.832 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-19T01:07:06.855 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-19T01:07:06.861 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-19T01:07:06.891 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-19T01:07:06.899 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-19T01:07:06.899 INFO:tasks.ceph:Archiving crash dumps... 2024-04-19T01:07:06.900 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663223/remote/smithi033/crash 2024-04-19T01:07:06.902 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-19T01:07:06.916 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663223/remote/smithi110/crash 2024-04-19T01:07:06.917 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-19T01:07:06.932 DEBUG:teuthology.misc:Transferring archived files from smithi118:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663223/remote/smithi118/crash 2024-04-19T01:07:06.933 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-19T01:07:06.947 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663223/remote/smithi155/crash 2024-04-19T01:07:06.948 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-19T01:07:06.963 INFO:tasks.ceph:Compressing logs... 2024-04-19T01:07:06.964 DEBUG:teuthology.orchestra.run.smithi033:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-19T01:07:06.967 DEBUG:teuthology.orchestra.run.smithi110:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-19T01:07:06.980 DEBUG:teuthology.orchestra.run.smithi118:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-19T01:07:06.989 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35894.log 2024-04-19T01:07:06.989 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31790.log 2024-04-19T01:07:06.989 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.35894.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35894.log.gz 2024-04-19T01:07:06.990 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37636.log 2024-04-19T01:07:06.990 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31790.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31790.log.gz 2024-04-19T01:07:06.990 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37740.log 2024-04-19T01:07:06.992 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33091.log 2024-04-19T01:07:06.992 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.37636.log: /var/log/ceph/ceph-client.admin.37740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37584.log 2024-04-19T01:07:06.992 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37740.log.gz 2024-04-19T01:07:06.992 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37636.log.gz 2024-04-19T01:07:06.992 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.33091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31247.log 2024-04-19T01:07:06.992 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33091.log.gz 2024-04-19T01:07:06.992 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.37584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37324.log 2024-04-19T01:07:06.995 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37584.log.gz 2024-04-19T01:07:06.995 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30881.log 2024-04-19T01:07:06.995 INFO:teuthology.orchestra.run.smithi033.stderr: 24.6% -- replaced with /var/log/ceph/ceph-client.admin.31247.log.gz 2024-04-19T01:07:06.995 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.37324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28979.log 2024-04-19T01:07:06.995 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37324.log.gz 2024-04-19T01:07:06.995 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.30881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28992.log 2024-04-19T01:07:06.996 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30881.log.gz 2024-04-19T01:07:06.996 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.28979.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28979.log.gz 2024-04-19T01:07:06.996 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28495.log 2024-04-19T01:07:06.996 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31437.log 2024-04-19T01:07:06.996 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.28992.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28992.log.gz 2024-04-19T01:07:06.996 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.28495.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28495.log.gz 2024-04-19T01:07:06.996 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37272.log 2024-04-19T01:07:06.996 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29587.log 2024-04-19T01:07:06.996 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31437.log.gz 2024-04-19T01:07:07.004 DEBUG:teuthology.orchestra.run.smithi155:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-19T01:07:07.014 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-19T01:07:07.014 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-19T01:07:07.014 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-19T01:07:07.014 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-19T01:07:07.015 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.9.log 2024-04-19T01:07:07.015 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.13.log 2024-04-19T01:07:07.015 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-19T01:07:07.015 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-osd.13.log: 86.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-19T01:07:07.015 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-mon.b.log: 89.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-19T01:07:07.025 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-19T01:07:07.026 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-04-19T01:07:07.026 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.14.log 2024-04-19T01:07:07.026 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-19T01:07:07.026 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-osd.14.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-19T01:07:07.026 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-19T01:07:07.026 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.10.log 2024-04-19T01:07:07.026 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-osd.10.log: 86.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-19T01:07:07.026 INFO:teuthology.orchestra.run.smithi118.stderr: 89.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-19T01:07:07.055 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.28762.log 2024-04-19T01:07:07.055 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25096.log 2024-04-19T01:07:07.056 INFO:teuthology.orchestra.run.smithi155.stderr:gzip/var/log/ceph/ceph-client.0.28762.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.25199.log 2024-04-19T01:07:07.056 INFO:teuthology.orchestra.run.smithi155.stderr: 74.2% -- replaced with /var/log/ceph/ceph-client.0.28762.log.gz 2024-04-19T01:07:07.056 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.25096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25000.log 2024-04-19T01:07:07.056 INFO:teuthology.orchestra.run.smithi155.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25096.log.gz 2024-04-19T01:07:07.056 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.25199.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.25199.log.gz 2024-04-19T01:07:07.056 INFO:teuthology.orchestra.run.smithi155.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.24976.log 2024-04-19T01:07:07.056 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.25000.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25000.log.gz 2024-04-19T01:07:07.056 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24952.log 2024-04-19T01:07:07.056 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.24976.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-04-19T01:07:07.056 INFO:teuthology.orchestra.run.smithi155.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24976.log.gz 2024-04-19T01:07:07.056 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.24952.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24952.log.gz 2024-04-19T01:07:07.056 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25175.log 2024-04-19T01:07:07.056 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25127.log 2024-04-19T01:07:07.056 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.25175.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-19T01:07:07.057 INFO:teuthology.orchestra.run.smithi155.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25175.log.gz 2024-04-19T01:07:07.057 INFO:teuthology.orchestra.run.smithi155.stderr:gzip/var/log/ceph/ceph-client.admin.25127.log: -5 --verbose -- /var/log/ceph/ceph-osd.11.log 2024-04-19T01:07:07.057 INFO:teuthology.orchestra.run.smithi155.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25127.log.gz 2024-04-19T01:07:07.057 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25296.log 2024-04-19T01:07:07.057 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25024.log 2024-04-19T01:07:07.057 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.25296.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.15.log 2024-04-19T01:07:07.057 INFO:teuthology.orchestra.run.smithi155.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25296.log.gz 2024-04-19T01:07:07.057 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.25024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25072.log 2024-04-19T01:07:07.057 INFO:teuthology.orchestra.run.smithi155.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25024.log.gz 2024-04-19T01:07:07.057 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-osd.15.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-19T01:07:07.057 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.25072.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25072.log.gz 2024-04-19T01:07:07.057 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25272.log 2024-04-19T01:07:07.057 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25048.log 2024-04-19T01:07:07.057 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.25272.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25272.log.gz 2024-04-19T01:07:07.057 INFO:teuthology.orchestra.run.smithi155.stderr: --verbose -- /var/log/ceph/ceph-client.admin.25223.log 2024-04-19T01:07:07.058 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.25048.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25048.log.gz 2024-04-19T01:07:07.058 INFO:teuthology.orchestra.run.smithi155.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.25247.log 2024-04-19T01:07:07.058 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.25223.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25223.log.gz 2024-04-19T01:07:07.058 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25151.log 2024-04-19T01:07:07.058 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.25247.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25247.log.gz 2024-04-19T01:07:07.058 INFO:teuthology.orchestra.run.smithi155.stderr: -- /var/log/ceph/ceph-client.admin.25320.log 2024-04-19T01:07:07.058 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.25151.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25151.log.gz 2024-04-19T01:07:07.058 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.25320.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25320.log.gz 2024-04-19T01:07:07.114 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.37272.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37272.log.gz 2024-04-19T01:07:07.114 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-04-19T01:07:07.115 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.29587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29026.log 2024-04-19T01:07:07.115 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29587.log.gz 2024-04-19T01:07:07.115 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32280.log 2024-04-19T01:07:07.115 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.29026.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.20146.log 2024-04-19T01:07:07.115 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29026.log.gz 2024-04-19T01:07:07.115 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.32280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30027.log 0.0% 2024-04-19T01:07:07.115 INFO:teuthology.orchestra.run.smithi033.stderr: -- replaced with /var/log/ceph/ceph-client.admin.32280.log.gz 2024-04-19T01:07:07.115 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph.tmp-client.admin.20146.log: 0.0%gzip -- replaced with /var/log/ceph/ceph.tmp-client.admin.20146.log.gz 2024-04-19T01:07:07.115 INFO:teuthology.orchestra.run.smithi033.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.35582.log 2024-04-19T01:07:07.115 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38000.log 2024-04-19T01:07:07.115 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.30027.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30027.log.gz 2024-04-19T01:07:07.115 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.35582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32659.log 2024-04-19T01:07:07.115 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35582.log.gz 2024-04-19T01:07:07.115 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.38000.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38000.log.gz 2024-04-19T01:07:07.116 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30766.log 2024-04-19T01:07:07.116 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.32659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28368.log 2024-04-19T01:07:07.116 INFO:teuthology.orchestra.run.smithi033.stderr: 24.6% -- replaced with /var/log/ceph/ceph-client.admin.32659.log.gz 2024-04-19T01:07:07.116 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.30766.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-19T01:07:07.116 INFO:teuthology.orchestra.run.smithi033.stderr: 24.0% -- replaced with /var/log/ceph/ceph-client.admin.30766.log.gz 2024-04-19T01:07:07.116 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.28368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35103.log 2024-04-19T01:07:07.116 INFO:teuthology.orchestra.run.smithi033.stderr: 28.8% -- replaced with /var/log/ceph/ceph-client.admin.28368.log.gz 2024-04-19T01:07:07.116 INFO:teuthology.orchestra.run.smithi033.stderr: 93.1% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-04-19T01:07:07.116 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31880.log 2024-04-19T01:07:07.116 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.35103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28956.log 2024-04-19T01:07:07.116 INFO:teuthology.orchestra.run.smithi033.stderr: 24.2% -- replaced with /var/log/ceph/ceph-client.admin.35103.log.gz 2024-04-19T01:07:07.116 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36595.log 2024-04-19T01:07:07.116 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31880.log.gz 2024-04-19T01:07:07.116 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.28956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34302.log 2024-04-19T01:07:07.117 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28956.log.gz 2024-04-19T01:07:07.117 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.36595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35530.log 2024-04-19T01:07:07.117 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36595.log.gz 2024-04-19T01:07:07.117 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.34302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37090.log 2024-04-19T01:07:07.117 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34302.log.gz 2024-04-19T01:07:07.117 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.35530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36203.log 2024-04-19T01:07:07.117 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35530.log.gz 2024-04-19T01:07:07.117 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.37090.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37090.log.gz 2024-04-19T01:07:07.117 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37168.log 2024-04-19T01:07:07.117 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.36203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30481.log 2024-04-19T01:07:07.117 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36203.log.gz 2024-04-19T01:07:07.117 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.37168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34245.log 2024-04-19T01:07:07.117 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37168.log.gz 2024-04-19T01:07:07.117 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.30481.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30481.log.gz 2024-04-19T01:07:07.117 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34676.log 2024-04-19T01:07:07.118 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.34245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29838.log 2024-04-19T01:07:07.118 INFO:teuthology.orchestra.run.smithi033.stderr: 24.2% -- replaced with /var/log/ceph/ceph-client.admin.34245.log.gz 2024-04-19T01:07:07.118 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.34676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31330.log 2024-04-19T01:07:07.118 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34676.log.gz 2024-04-19T01:07:07.118 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.29838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33735.log 2024-04-19T01:07:07.118 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29838.log.gz 2024-04-19T01:07:07.118 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36641.log 2024-04-19T01:07:07.118 INFO:teuthology.orchestra.run.smithi033.stderr: 24.6% -- replaced with /var/log/ceph/ceph-client.admin.31330.log.gz 2024-04-19T01:07:07.118 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.33735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37194.log 2024-04-19T01:07:07.118 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33735.log.gz 2024-04-19T01:07:07.118 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.36641.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36641.log.gz 2024-04-19T01:07:07.118 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36188.log 2024-04-19T01:07:07.118 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.37194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33488.log 2024-04-19T01:07:07.118 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37194.log.gz 2024-04-19T01:07:07.119 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.36188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35842.log 2024-04-19T01:07:07.119 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36188.log.gz 2024-04-19T01:07:07.119 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.33488.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33488.log.gz 2024-04-19T01:07:07.119 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36189.log 2024-04-19T01:07:07.119 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.35842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36193.log 2024-04-19T01:07:07.119 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35842.log.gz 2024-04-19T01:07:07.119 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.36189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37766.log 2024-04-19T01:07:07.119 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36189.log.gz 2024-04-19T01:07:07.119 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.36193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37402.log 2024-04-19T01:07:07.119 INFO:teuthology.orchestra.run.smithi033.stderr: 28.0% -- replaced with /var/log/ceph/ceph-client.admin.36193.log.gz 2024-04-19T01:07:07.119 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.37766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31647.log 2024-04-19T01:07:07.119 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37766.log.gz 2024-04-19T01:07:07.119 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.37402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37454.log 2024-04-19T01:07:07.119 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37402.log.gz 2024-04-19T01:07:07.120 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37818.log 2024-04-19T01:07:07.120 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31647.log.gz 2024-04-19T01:07:07.120 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.37454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31962.log 2024-04-19T01:07:07.120 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37454.log.gz 2024-04-19T01:07:07.120 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.37818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28603.log 2024-04-19T01:07:07.120 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37818.log.gz 2024-04-19T01:07:07.120 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37142.log 2024-04-19T01:07:07.120 INFO:teuthology.orchestra.run.smithi033.stderr: 25.0% -- replaced with /var/log/ceph/ceph-client.admin.31962.log.gz 2024-04-19T01:07:07.120 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.28603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29759.log 2024-04-19T01:07:07.120 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28603.log.gz 2024-04-19T01:07:07.120 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.37142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31820.log 2024-04-19T01:07:07.120 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37142.log.gz 2024-04-19T01:07:07.120 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.29759.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29759.log.gz 2024-04-19T01:07:07.120 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38076.log 2024-04-19T01:07:07.121 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30927.log 2024-04-19T01:07:07.121 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31820.log.gz 2024-04-19T01:07:07.121 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.38076.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.8.log 2024-04-19T01:07:07.121 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38076.log.gz 2024-04-19T01:07:07.121 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.30927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35738.log 2024-04-19T01:07:07.121 INFO:teuthology.orchestra.run.smithi033.stderr: 24.8% -- replaced with /var/log/ceph/ceph-client.admin.30927.log.gz 2024-04-19T01:07:07.121 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29011.log 2024-04-19T01:07:07.121 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.35738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31677.log 2024-04-19T01:07:07.121 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35738.log.gz 2024-04-19T01:07:07.121 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.29011.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29011.log.gz 2024-04-19T01:07:07.121 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36575.log 2024-04-19T01:07:07.121 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29944.log 2024-04-19T01:07:07.121 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31677.log.gz 2024-04-19T01:07:07.121 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.36575.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36575.log.gz 2024-04-19T01:07:07.122 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30507.log 2024-04-19T01:07:07.122 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.29944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37532.log 2024-04-19T01:07:07.122 INFO:teuthology.orchestra.run.smithi033.stderr: 24.6% -- replaced with /var/log/ceph/ceph-client.admin.29944.log.gz 2024-04-19T01:07:07.122 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.30507.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30507.log.gz 2024-04-19T01:07:07.122 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37116.log 2024-04-19T01:07:07.122 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.37532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28469.log 2024-04-19T01:07:07.122 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37532.log.gz 2024-04-19T01:07:07.122 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.37116.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37116.log.gz 2024-04-19T01:07:07.122 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33033.log 2024-04-19T01:07:07.122 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.28469.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.28469.log.gz 2024-04-19T01:07:07.122 INFO:teuthology.orchestra.run.smithi033.stderr: --verbose -- /var/log/ceph/ceph-client.admin.34425.log 2024-04-19T01:07:07.122 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.33033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35374.log 2024-04-19T01:07:07.122 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33033.log.gz 2024-04-19T01:07:07.123 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.34425.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34425.log.gz 2024-04-19T01:07:07.123 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-19T01:07:07.123 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.35374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28599.log 2024-04-19T01:07:07.123 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35374.log.gz 2024-04-19T01:07:07.123 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32333.log 2024-04-19T01:07:07.123 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.28599.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.28599.log.gz /var/log/ceph/ceph-client.admin.33403.log 2024-04-19T01:07:07.123 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-04-19T01:07:07.123 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.32333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28600.log 2024-04-19T01:07:07.123 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32333.log.gz 2024-04-19T01:07:07.123 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.33403.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33403.log.gz 2024-04-19T01:07:07.123 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35426.log 2024-04-19T01:07:07.123 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.28600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36199.log 2024-04-19T01:07:07.123 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28600.log.gz 2024-04-19T01:07:07.123 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.35426.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.35426.log.gz /var/log/ceph/ceph-client.admin.32185.log 2024-04-19T01:07:07.123 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-04-19T01:07:07.124 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.36199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37844.log 2024-04-19T01:07:07.124 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36199.log.gz 2024-04-19T01:07:07.124 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.32185.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32185.log.gz 2024-04-19T01:07:07.124 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30350.log 2024-04-19T01:07:07.124 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34218.log 2024-04-19T01:07:07.124 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.37844.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37844.log.gz 2024-04-19T01:07:07.124 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.30350.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30350.log.gz 2024-04-19T01:07:07.124 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36024.log 2024-04-19T01:07:07.124 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.34218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33280.log 2024-04-19T01:07:07.124 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34218.log.gz 2024-04-19T01:07:07.124 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.36024.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36024.log.gz 2024-04-19T01:07:07.124 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29022.log 2024-04-19T01:07:07.124 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.33280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33814.log 2024-04-19T01:07:07.124 INFO:teuthology.orchestra.run.smithi033.stderr: 24.3% -- replaced with /var/log/ceph/ceph-client.admin.33280.log.gz 2024-04-19T01:07:07.124 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.29022.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29022.log.gz 2024-04-19T01:07:07.125 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30569.log 2024-04-19T01:07:07.125 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35556.log 2024-04-19T01:07:07.125 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.33814.log: /var/log/ceph/ceph-client.admin.30569.log: 24.6% -- replaced with /var/log/ceph/ceph-client.admin.33814.log.gz 2024-04-19T01:07:07.125 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33003.log 2024-04-19T01:07:07.125 INFO:teuthology.orchestra.run.smithi033.stderr: 24.6% -- replaced with /var/log/ceph/ceph-client.admin.30569.log.gz 2024-04-19T01:07:07.125 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.35556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36196.log 2024-04-19T01:07:07.125 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35556.log.gz 2024-04-19T01:07:07.125 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.33003.log: gzip 23.2% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.33003.log.gz -- 2024-04-19T01:07:07.125 INFO:teuthology.orchestra.run.smithi033.stderr: /var/log/ceph/ceph-client.admin.28399.log 2024-04-19T01:07:07.125 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36076.log 2024-04-19T01:07:07.125 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.36196.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36196.log.gz 2024-04-19T01:07:07.125 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.28399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34974.log 2024-04-19T01:07:07.125 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28399.log.gz 2024-04-19T01:07:07.125 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.36076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31390.log 2024-04-19T01:07:07.126 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36076.log.gz 2024-04-19T01:07:07.126 INFO:teuthology.orchestra.run.smithi033.stderr:gzip/var/log/ceph/ceph-client.admin.34974.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.32978.log 2024-04-19T01:07:07.126 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34974.log.gz 2024-04-19T01:07:07.126 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34115.log 2024-04-19T01:07:07.126 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31390.log.gz 2024-04-19T01:07:07.126 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.32978.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32978.log.gz 2024-04-19T01:07:07.126 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34191.log 2024-04-19T01:07:07.126 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36208.log 2024-04-19T01:07:07.126 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.34115.log: /var/log/ceph/ceph-client.admin.34191.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34115.log.gz 2024-04-19T01:07:07.126 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34191.log.gz 2024-04-19T01:07:07.126 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31304.log 2024-04-19T01:07:07.126 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31595.log 2024-04-19T01:07:07.126 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.36208.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36208.log.gz 2024-04-19T01:07:07.126 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31304.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31304.log.gz 2024-04-19T01:07:07.126 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32930.log 2024-04-19T01:07:07.127 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38026.log 2024-04-19T01:07:07.127 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31595.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31595.log.gz 2024-04-19T01:07:07.127 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.32930.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32930.log.gz 2024-04-19T01:07:07.127 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38052.log 2024-04-19T01:07:07.127 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34165.log 2024-04-19T01:07:07.127 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.38026.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38026.log.gz 2024-04-19T01:07:07.127 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.38052.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38052.log.gz 2024-04-19T01:07:07.127 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36194.log 2024-04-19T01:07:07.127 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32045.log 2024-04-19T01:07:07.127 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.34165.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34165.log.gz 2024-04-19T01:07:07.127 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.36194.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36194.log.gz 2024-04-19T01:07:07.127 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30986.log 2024-04-19T01:07:07.127 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.32045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29483.log 2024-04-19T01:07:07.127 INFO:teuthology.orchestra.run.smithi033.stderr: 24.2% -- replaced with /var/log/ceph/ceph-client.admin.32045.log.gz 2024-04-19T01:07:07.128 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.30986.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30986.log.gz 2024-04-19T01:07:07.128 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28959.log 2024-04-19T01:07:07.128 INFO:teuthology.orchestra.run.smithi033.stderr:gzip/var/log/ceph/ceph-client.admin.29483.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.29066.log 2024-04-19T01:07:07.128 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29483.log.gz 2024-04-19T01:07:07.128 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.28959.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.28959.log.gz 2024-04-19T01:07:07.128 INFO:teuthology.orchestra.run.smithi033.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.30851.log 2024-04-19T01:07:07.128 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.29066.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29066.log.gzgzip 2024-04-19T01:07:07.128 INFO:teuthology.orchestra.run.smithi033.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.36990.log 2024-04-19T01:07:07.128 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.30851.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30851.log.gz 2024-04-19T01:07:07.128 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32634.log 2024-04-19T01:07:07.128 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35634.log 2024-04-19T01:07:07.128 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.36990.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36990.log.gz 2024-04-19T01:07:07.128 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.32634.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32634.log.gz 2024-04-19T01:07:07.128 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34982.log 2024-04-19T01:07:07.129 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.35634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28521.log 2024-04-19T01:07:07.129 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35634.log.gz 2024-04-19T01:07:07.129 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.34982.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34982.log.gz 2024-04-19T01:07:07.129 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28601.log 2024-04-19T01:07:07.129 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.28521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37038.log 2024-04-19T01:07:07.129 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28521.log.gz 2024-04-19T01:07:07.129 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.28601.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28601.log.gz 2024-04-19T01:07:07.129 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32019.log 2024-04-19T01:07:07.129 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34487.log 2024-04-19T01:07:07.129 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.37038.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37038.log.gz 2024-04-19T01:07:07.129 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.32019.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32019.log.gz 2024-04-19T01:07:07.129 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34327.log 2024-04-19T01:07:07.129 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29586.log 2024-04-19T01:07:07.129 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.34487.log: 25.1% -- replaced with /var/log/ceph/ceph-client.admin.34487.log.gz 2024-04-19T01:07:07.129 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.34327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34978.log 2024-04-19T01:07:07.130 INFO:teuthology.orchestra.run.smithi033.stderr: 24.0% -- replaced with /var/log/ceph/ceph-client.admin.34327.log.gz 2024-04-19T01:07:07.130 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31744.log 2024-04-19T01:07:07.130 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.29586.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29586.log.gz 2024-04-19T01:07:07.130 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.34978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34568.log 2024-04-19T01:07:07.130 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34978.log.gz 2024-04-19T01:07:07.130 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35400.log 2024-04-19T01:07:07.130 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31744.log.gz 2024-04-19T01:07:07.130 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31490.log 2024-04-19T01:07:07.130 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.34568.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34568.log.gz 2024-04-19T01:07:07.130 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.35400.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35400.log.gz 2024-04-19T01:07:07.130 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35050.log 2024-04-19T01:07:07.130 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35660.log 2024-04-19T01:07:07.130 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31490.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31490.log.gz 2024-04-19T01:07:07.130 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.35050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36207.log 2024-04-19T01:07:07.131 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35050.log.gz 2024-04-19T01:07:07.131 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37480.log 2024-04-19T01:07:07.131 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.35660.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35660.log.gz 2024-04-19T01:07:07.131 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.36207.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36207.log.gz 2024-04-19T01:07:07.131 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33598.log 2024-04-19T01:07:07.131 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28984.log 2024-04-19T01:07:07.131 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.37480.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37480.log.gz 2024-04-19T01:07:07.131 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.33598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29539.log 2024-04-19T01:07:07.131 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33598.log.gz 2024-04-19T01:07:07.131 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35076.log 2024-04-19T01:07:07.131 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.28984.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28984.log.gz 2024-04-19T01:07:07.131 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.29539.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29539.log.gz 2024-04-19T01:07:07.131 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28593.log 2024-04-19T01:07:07.131 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30346.log 2024-04-19T01:07:07.131 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.35076.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35076.log.gz 2024-04-19T01:07:07.132 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.28593.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.28593.log.gz -5 2024-04-19T01:07:07.132 INFO:teuthology.orchestra.run.smithi033.stderr: --verbose -- /var/log/ceph/ceph-client.admin.30716.log 2024-04-19T01:07:07.132 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36622.log 2024-04-19T01:07:07.132 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.30346.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30346.log.gz 2024-04-19T01:07:07.132 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.30716.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30716.log.gz 2024-04-19T01:07:07.132 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30177.log 2024-04-19T01:07:07.132 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.36622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29843.log 2024-04-19T01:07:07.132 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36622.log.gz 2024-04-19T01:07:07.132 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.30177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24919.log 2024-04-19T01:07:07.132 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30177.log.gz 2024-04-19T01:07:07.132 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37974.log 2024-04-19T01:07:07.132 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.29843.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29843.log.gz 2024-04-19T01:07:07.132 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.24919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37064.log 2024-04-19T01:07:07.132 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24919.log.gz 2024-04-19T01:07:07.133 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36620.log 2024-04-19T01:07:07.133 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.37974.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37974.log.gz 2024-04-19T01:07:07.133 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.37064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30608.log 2024-04-19T01:07:07.133 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37064.log.gz 2024-04-19T01:07:07.133 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35764.log 2024-04-19T01:07:07.133 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.36620.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36620.log.gz 2024-04-19T01:07:07.133 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.30608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31391.log 2024-04-19T01:07:07.133 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30608.log.gz 2024-04-19T01:07:07.133 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37506.log 2024-04-19T01:07:07.133 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.35764.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35764.log.gz 2024-04-19T01:07:07.133 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31391.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31391.log.gz 2024-04-19T01:07:07.133 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36102.log 2024-04-19T01:07:07.133 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28592.log 2024-04-19T01:07:07.133 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.37506.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37506.log.gz 2024-04-19T01:07:07.133 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.36102.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36102.log.gz 2024-04-19T01:07:07.134 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32835.log 2024-04-19T01:07:07.134 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.28592.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28592.log.gz 2024-04-19T01:07:07.134 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28595.log 2024-04-19T01:07:07.134 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37350.log 2024-04-19T01:07:07.134 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.32835.log: 24.4% -- replaced with /var/log/ceph/ceph-client.admin.32835.log.gz 2024-04-19T01:07:07.134 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.28595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30237.log 2024-04-19T01:07:07.134 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28595.log.gz 2024-04-19T01:07:07.134 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.37350.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37350.log.gz 2024-04-19T01:07:07.134 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30070.log 2024-04-19T01:07:07.134 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31850.log 2024-04-19T01:07:07.134 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.30237.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30237.log.gz 2024-04-19T01:07:07.134 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.30070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33982.log 2024-04-19T01:07:07.134 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30070.log.gz 2024-04-19T01:07:07.134 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31850.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.30262.log -- replaced with /var/log/ceph/ceph-client.admin.31850.log.gz 2024-04-19T01:07:07.135 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-04-19T01:07:07.135 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33913.log 2024-04-19T01:07:07.135 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.33982.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33982.log.gz 2024-04-19T01:07:07.135 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.30262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36673.log 2024-04-19T01:07:07.135 INFO:teuthology.orchestra.run.smithi033.stderr: 24.3% -- replaced with /var/log/ceph/ceph-client.admin.30262.log.gz 2024-04-19T01:07:07.135 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.33913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33138.log 2024-04-19T01:07:07.135 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33913.log.gz 2024-04-19T01:07:07.135 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.36673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33590.log 2024-04-19T01:07:07.135 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36673.log.gz 2024-04-19T01:07:07.135 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.33138.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33138.log.gz 2024-04-19T01:07:07.135 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29003.log 2024-04-19T01:07:07.135 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35790.log 2024-04-19T01:07:07.135 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.33590.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33590.log.gz 2024-04-19T01:07:07.135 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.29003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33865.log 2024-04-19T01:07:07.135 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29003.log.gz 2024-04-19T01:07:07.136 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.35790.log: gzip -5 --verbose 0.0% -- /var/log/ceph/ceph-client.admin.36608.log -- replaced with /var/log/ceph/ceph-client.admin.35790.log.gz 2024-04-19T01:07:07.136 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-04-19T01:07:07.136 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31660.log 2024-04-19T01:07:07.136 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.33865.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33865.log.gz 2024-04-19T01:07:07.136 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.36608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29039.log 2024-04-19T01:07:07.136 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36608.log.gz 2024-04-19T01:07:07.136 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29583.log 2024-04-19T01:07:07.136 INFO:teuthology.orchestra.run.smithi033.stderr: 24.6% -- replaced with /var/log/ceph/ceph-client.admin.31660.log.gz 2024-04-19T01:07:07.136 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35998.log 2024-04-19T01:07:07.136 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.29039.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29039.log.gz 2024-04-19T01:07:07.136 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.29583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32777.log 2024-04-19T01:07:07.136 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29583.log.gz 2024-04-19T01:07:07.136 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.35998.log: gzip -5 --verbose -- 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35998.log.gz /var/log/ceph/ceph-client.admin.29457.log 2024-04-19T01:07:07.136 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-04-19T01:07:07.136 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32332.log 2024-04-19T01:07:07.137 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.32777.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32777.log.gz 2024-04-19T01:07:07.137 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.29457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32708.log 2024-04-19T01:07:07.137 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29457.log.gz 2024-04-19T01:07:07.137 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.32332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32503.log 2024-04-19T01:07:07.137 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32332.log.gz 2024-04-19T01:07:07.137 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37428.log 2024-04-19T01:07:07.137 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.32708.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32708.log.gz 2024-04-19T01:07:07.137 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.32503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28582.log 2024-04-19T01:07:07.137 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32503.log.gz 2024-04-19T01:07:07.137 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.37428.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.37428.log.gz 2024-04-19T01:07:07.137 INFO:teuthology.orchestra.run.smithi033.stderr: /var/log/ceph/ceph-client.admin.35296.log 2024-04-19T01:07:07.137 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36587.log 2024-04-19T01:07:07.137 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.28582.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28582.log.gz 2024-04-19T01:07:07.137 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.35296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36534.log 2024-04-19T01:07:07.137 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35296.log.gz 2024-04-19T01:07:07.138 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.36587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36614.log 2024-04-19T01:07:07.138 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36587.log.gz 2024-04-19T01:07:07.138 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34087.log 2024-04-19T01:07:07.138 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.36534.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36534.log.gz 2024-04-19T01:07:07.138 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.36614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36212.log 2024-04-19T01:07:07.138 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36614.log.gz 2024-04-19T01:07:07.138 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.34087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32499.log 2024-04-19T01:07:07.138 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34087.log.gz 2024-04-19T01:07:07.138 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.36212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33087.log 2024-04-19T01:07:07.138 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36212.log.gz 2024-04-19T01:07:07.138 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.32499.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32499.log.gz 2024-04-19T01:07:07.138 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37792.log 2024-04-19T01:07:07.138 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30682.log 2024-04-19T01:07:07.139 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.33087.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33087.log.gz 2024-04-19T01:07:07.139 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.37792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28594.log 2024-04-19T01:07:07.139 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37792.log.gz 2024-04-19T01:07:07.139 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.30682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31038.log 2024-04-19T01:07:07.139 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30682.log.gz 2024-04-19T01:07:07.139 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.28594.log: /var/log/ceph/ceph-client.admin.31543.log 2024-04-19T01:07:07.139 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28594.log.gz 2024-04-19T01:07:07.139 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37610.log 2024-04-19T01:07:07.139 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31038.log.gz 2024-04-19T01:07:07.139 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30423.log 2024-04-19T01:07:07.139 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31543.log.gz 2024-04-19T01:07:07.139 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28607.log 2024-04-19T01:07:07.139 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.37610.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37610.log.gz 2024-04-19T01:07:07.139 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.30423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28604.log 2024-04-19T01:07:07.140 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30423.log.gz 2024-04-19T01:07:07.140 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.28607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35322.log 2024-04-19T01:07:07.140 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28607.log.gz 2024-04-19T01:07:07.140 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.28604.log: -- /var/log/ceph/ceph-client.admin.31068.log 2024-04-19T01:07:07.140 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28604.log.gz 2024-04-19T01:07:07.140 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.35322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31093.log 2024-04-19T01:07:07.140 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35322.log.gz 2024-04-19T01:07:07.140 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31068.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-19T01:07:07.140 INFO:teuthology.orchestra.run.smithi033.stderr: 26.0% -- replaced with /var/log/ceph/ceph-client.admin.31068.log.gz 2024-04-19T01:07:07.140 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32831.log 2024-04-19T01:07:07.140 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31093.log.gz 2024-04-19T01:07:07.140 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28443.log 2024-04-19T01:07:07.140 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.32831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37714.log 2024-04-19T01:07:07.140 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32831.log.gz 2024-04-19T01:07:07.140 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.28443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35504.log 2024-04-19T01:07:07.141 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28443.log.gz 2024-04-19T01:07:07.141 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.37714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30068.log 2024-04-19T01:07:07.141 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37714.log.gz 2024-04-19T01:07:07.141 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.35504.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35504.log.gz 2024-04-19T01:07:07.141 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25152.log 2024-04-19T01:07:07.141 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.30068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34834.log 2024-04-19T01:07:07.141 INFO:teuthology.orchestra.run.smithi033.stderr: 86.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-19T01:07:07.141 INFO:teuthology.orchestra.run.smithi033.stderr: 24.6% -- replaced with /var/log/ceph/ceph-client.admin.30068.log.gz 2024-04-19T01:07:07.141 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.25152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34891.log 2024-04-19T01:07:07.141 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25152.log.gz 2024-04-19T01:07:07.141 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.34834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32129.log 2024-04-19T01:07:07.141 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34834.log.gz 2024-04-19T01:07:07.141 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.34891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28995.log 2024-04-19T01:07:07.141 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34891.log.gz 2024-04-19T01:07:07.142 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.32129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36176.log 2024-04-19T01:07:07.142 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32129.log.gz 2024-04-19T01:07:07.142 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33731.log 2024-04-19T01:07:07.142 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.28995.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28995.log.gz 2024-04-19T01:07:07.142 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.36176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36214.log 2024-04-19T01:07:07.142 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36176.log.gz 2024-04-19T01:07:07.142 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.33731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28606.log 2024-04-19T01:07:07.142 INFO:teuthology.orchestra.run.smithi033.stderr: 24.6% -- replaced with /var/log/ceph/ceph-client.admin.33731.log.gz 2024-04-19T01:07:07.142 INFO:teuthology.orchestra.run.smithi033.stderr:gzip/var/log/ceph/ceph-client.admin.36214.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.32473.log 2024-04-19T01:07:07.142 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36214.log.gz 2024-04-19T01:07:07.142 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.28606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29431.log 2024-04-19T01:07:07.142 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28606.log.gz 2024-04-19T01:07:07.142 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.32473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30398.log 2024-04-19T01:07:07.143 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32473.log.gz 2024-04-19T01:07:07.143 INFO:teuthology.orchestra.run.smithi033.stderr:gzip/var/log/ceph/ceph-client.admin.29431.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.29837.log 2024-04-19T01:07:07.143 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29431.log.gz 2024-04-19T01:07:07.143 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.30398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29712.log 2024-04-19T01:07:07.143 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30398.log.gz 2024-04-19T01:07:07.143 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.29837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37870.log 2024-04-19T01:07:07.143 INFO:teuthology.orchestra.run.smithi033.stderr: 24.6% -- replaced with /var/log/ceph/ceph-client.admin.29837.log.gz 2024-04-19T01:07:07.143 INFO:teuthology.orchestra.run.smithi033.stderr:gzip/var/log/ceph/ceph-client.admin.29712.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.28941.log 2024-04-19T01:07:07.143 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29712.log.gz 2024-04-19T01:07:07.143 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.37870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28976.log 2024-04-19T01:07:07.143 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37870.log.gz 2024-04-19T01:07:07.143 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.28941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29509.log 2024-04-19T01:07:07.143 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28941.log.gz 2024-04-19T01:07:07.143 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.28976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36986.log 2024-04-19T01:07:07.143 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28976.log.gz 2024-04-19T01:07:07.144 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.29509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32926.log 2024-04-19T01:07:07.144 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29509.log.gz 2024-04-19T01:07:07.144 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.36986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30308.log 2024-04-19T01:07:07.144 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36986.log.gz 2024-04-19T01:07:07.144 INFO:teuthology.orchestra.run.smithi033.stderr:gzip/var/log/ceph/ceph-client.admin.32926.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.36128.log 2024-04-19T01:07:07.144 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32926.log.gz 2024-04-19T01:07:07.144 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.30308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33431.log 2024-04-19T01:07:07.144 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30308.log.gz 2024-04-19T01:07:07.144 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.36128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35187.log 2024-04-19T01:07:07.144 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36128.log.gz 2024-04-19T01:07:07.144 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37948.log 2024-04-19T01:07:07.144 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.33431.log: 24.6% -- replaced with /var/log/ceph/ceph-client.admin.33431.log.gz 2024-04-19T01:07:07.144 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.35187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35239.log 2024-04-19T01:07:07.144 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35187.log.gz 2024-04-19T01:07:07.144 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.37948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31086.log 2024-04-19T01:07:07.145 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37948.log.gz 2024-04-19T01:07:07.145 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5/var/log/ceph/ceph-client.admin.35239.log: --verbose -- /var/log/ceph/ceph-client.admin.32181.log 2024-04-19T01:07:07.145 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35239.log.gz 2024-04-19T01:07:07.145 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32608.log 2024-04-19T01:07:07.145 INFO:teuthology.orchestra.run.smithi033.stderr: 24.8% -- replaced with /var/log/ceph/ceph-client.admin.31086.log.gz 2024-04-19T01:07:07.145 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.32181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35452.log 2024-04-19T01:07:07.145 INFO:teuthology.orchestra.run.smithi033.stderr: 23.9% -- replaced with /var/log/ceph/ceph-client.admin.32181.log.gz 2024-04-19T01:07:07.145 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.32608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31908.log 2024-04-19T01:07:07.145 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32608.log.gz 2024-04-19T01:07:07.145 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.35452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28570.log 2024-04-19T01:07:07.145 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35452.log.gz 2024-04-19T01:07:07.145 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32133.log 2024-04-19T01:07:07.145 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31908.log.gz 2024-04-19T01:07:07.145 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.28570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35920.log 2024-04-19T01:07:07.145 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28570.log.gz 2024-04-19T01:07:07.146 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.32133.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-19T01:07:07.146 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32133.log.gz 2024-04-19T01:07:07.146 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.35920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33353.log 2024-04-19T01:07:07.146 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35920.log.gz 2024-04-19T01:07:07.146 INFO:teuthology.orchestra.run.smithi033.stderr:gzip/var/log/ceph/ceph.audit.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.33165.log 2024-04-19T01:07:07.146 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.33353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36205.log 2024-04-19T01:07:07.146 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33353.log.gz 2024-04-19T01:07:07.146 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.33165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29405.log 2024-04-19T01:07:07.146 INFO:teuthology.orchestra.run.smithi033.stderr: 24.2% -- replaced with /var/log/ceph/ceph-client.admin.33165.log.gz 2024-04-19T01:07:07.146 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.36205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32429.log 2024-04-19T01:07:07.146 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36205.log.gz 2024-04-19T01:07:07.146 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.29405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30741.log 2024-04-19T01:07:07.146 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29405.log.gz 2024-04-19T01:07:07.146 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.32429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30173.log 2024-04-19T01:07:07.147 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32429.log.gz 2024-04-19T01:07:07.147 INFO:teuthology.orchestra.run.smithi033.stderr: 89.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-19T01:07:07.147 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.30741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34032.log 2024-04-19T01:07:07.147 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30741.log.gz 2024-04-19T01:07:07.147 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.30173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33226.log 2024-04-19T01:07:07.147 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30173.log.gz 2024-04-19T01:07:07.147 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.34032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36192.log 2024-04-19T01:07:07.147 INFO:teuthology.orchestra.run.smithi033.stderr: 25.4% -- replaced with /var/log/ceph/ceph-client.admin.34032.log.gz 2024-04-19T01:07:07.147 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.33226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34682.log 2024-04-19T01:07:07.147 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33226.log.gz 2024-04-19T01:07:07.147 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.36192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36737.log 2024-04-19T01:07:07.147 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36192.log.gz 2024-04-19T01:07:07.147 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.34682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31464.log 2024-04-19T01:07:07.147 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34682.log.gz 2024-04-19T01:07:07.148 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35686.log 2024-04-19T01:07:07.148 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.36737.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36737.log.gz 2024-04-19T01:07:07.148 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35348.log 2024-04-19T01:07:07.148 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31464.log.gz 2024-04-19T01:07:07.148 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.35686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32709.log 2024-04-19T01:07:07.148 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35686.log.gz 2024-04-19T01:07:07.148 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.12.log 2024-04-19T01:07:07.148 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.35348.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35348.log.gz 2024-04-19T01:07:07.148 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.32709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36588.log 2024-04-19T01:07:07.148 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32709.log.gz 2024-04-19T01:07:07.148 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-osd.12.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29581.log 2024-04-19T01:07:07.148 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.36588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31621.log 2024-04-19T01:07:07.148 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36588.log.gz 2024-04-19T01:07:07.148 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.29581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37922.log 2024-04-19T01:07:07.148 INFO:teuthology.orchestra.run.smithi033.stderr: 23.7% -- replaced with /var/log/ceph/ceph-client.admin.29581.log.gz 2024-04-19T01:07:07.149 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36050.log 2024-04-19T01:07:07.149 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31621.log.gz 2024-04-19T01:07:07.149 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.37922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35161.log 2024-04-19T01:07:07.149 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37922.log.gz 2024-04-19T01:07:07.149 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.36050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35478.log 2024-04-19T01:07:07.149 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36050.log.gz 2024-04-19T01:07:07.149 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.35161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34592.log 2024-04-19T01:07:07.149 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35161.log.gz 2024-04-19T01:07:07.149 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.35478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37376.log 2024-04-19T01:07:07.149 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35478.log.gz 2024-04-19T01:07:07.149 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.34592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36603.log 2024-04-19T01:07:07.149 INFO:teuthology.orchestra.run.smithi033.stderr: 24.3% -- replaced with /var/log/ceph/ceph-client.admin.34592.log.gz 2024-04-19T01:07:07.149 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.37376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31569.log 2024-04-19T01:07:07.149 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37376.log.gz 2024-04-19T01:07:07.150 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.36603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35816.log 2024-04-19T01:07:07.150 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36603.log.gz 2024-04-19T01:07:07.150 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33516.log 2024-04-19T01:07:07.150 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31569.log.gz 2024-04-19T01:07:07.150 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.35816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34488.log 2024-04-19T01:07:07.150 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35816.log.gz 2024-04-19T01:07:07.150 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35712.log 2024-04-19T01:07:07.150 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.33516.log: 24.2% -- replaced with /var/log/ceph/ceph-client.admin.33516.log.gz 2024-04-19T01:07:07.150 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.34488.log: -- /var/log/ceph/ceph-client.admin.36209.log 2024-04-19T01:07:07.150 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34488.log.gz 2024-04-19T01:07:07.150 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.35712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28993.log 2024-04-19T01:07:07.150 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35712.log.gz 2024-04-19T01:07:07.150 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.36209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37688.log 2024-04-19T01:07:07.150 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36209.log.gz 2024-04-19T01:07:07.150 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.28993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34763.log 2024-04-19T01:07:07.151 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28993.log.gz 2024-04-19T01:07:07.151 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.37688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31176.log 2024-04-19T01:07:07.151 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37688.log.gz 2024-04-19T01:07:07.151 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29004.log 2024-04-19T01:07:07.151 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.34763.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34763.log.gz 2024-04-19T01:07:07.151 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35270.log 2024-04-19T01:07:07.151 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31176.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31176.log.gz 2024-04-19T01:07:07.151 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.29004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33646.log 2024-04-19T01:07:07.151 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29004.log.gz 2024-04-19T01:07:07.151 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.35270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35868.log 2024-04-19T01:07:07.151 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35270.log.gz 2024-04-19T01:07:07.151 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.33646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35213.log 2024-04-19T01:07:07.151 INFO:teuthology.orchestra.run.smithi033.stderr: 24.7% -- replaced with /var/log/ceph/ceph-client.admin.33646.log.gz 2024-04-19T01:07:07.151 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.35868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32388.log 2024-04-19T01:07:07.152 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35868.log.gz 2024-04-19T01:07:07.152 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.35213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34433.log 2024-04-19T01:07:07.152 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35213.log.gz 2024-04-19T01:07:07.152 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.32388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35946.log 2024-04-19T01:07:07.152 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32388.log.gz 2024-04-19T01:07:07.152 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.34433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30877.log 2024-04-19T01:07:07.152 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34433.log.gz 2024-04-19T01:07:07.152 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.35946.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-19T01:07:07.152 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35946.log.gz 2024-04-19T01:07:07.152 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.30877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31516.log 2024-04-19T01:07:07.152 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30877.log.gz 2024-04-19T01:07:07.152 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37220.log 2024-04-19T01:07:07.152 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-osd.4.log: /var/log/ceph/ceph-client.admin.31516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28596.log 2024-04-19T01:07:07.152 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31516.log.gz 2024-04-19T01:07:07.153 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.37220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37896.log 2024-04-19T01:07:07.153 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37220.log.gz 2024-04-19T01:07:07.153 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.28596.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28596.log.gz 2024-04-19T01:07:07.153 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36597.log 2024-04-19T01:07:07.153 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30002.log 2024-04-19T01:07:07.153 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.37896.log: 0.0%/var/log/ceph/ceph-client.admin.36597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37558.log 2024-04-19T01:07:07.153 INFO:teuthology.orchestra.run.smithi033.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37896.log.gz 2024-04-19T01:07:07.153 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36597.log.gz 2024-04-19T01:07:07.153 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.30002.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30002.log.gz 2024-04-19T01:07:07.153 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31012.log 2024-04-19T01:07:07.153 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.37558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29379.log 2024-04-19T01:07:07.153 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37558.log.gz 2024-04-19T01:07:07.153 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30511.log 2024-04-19T01:07:07.153 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31012.log.gz 2024-04-19T01:07:07.153 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34651.log 2024-04-19T01:07:07.154 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.29379.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29379.log.gz 2024-04-19T01:07:07.154 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.30511.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30511.log.gz 2024-04-19T01:07:07.154 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32103.log 2024-04-19T01:07:07.154 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30809.log 2024-04-19T01:07:07.154 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.34651.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34651.log.gz 2024-04-19T01:07:07.154 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36175.log 2024-04-19T01:07:07.154 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.32103.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32103.log.gz 2024-04-19T01:07:07.154 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.30809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25040.log 2024-04-19T01:07:07.154 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30809.log.gz 2024-04-19T01:07:07.154 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34917.log 2024-04-19T01:07:07.154 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.36175.log: /var/log/ceph/ceph-client.admin.25040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33377.log 2024-04-19T01:07:07.154 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36175.log.gz 2024-04-19T01:07:07.154 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25040.log.gz 2024-04-19T01:07:07.154 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.34917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36566.log 2024-04-19T01:07:07.155 INFO:teuthology.orchestra.run.smithi033.stderr: 24.7% -- replaced with /var/log/ceph/ceph-client.admin.34917.log.gz 2024-04-19T01:07:07.155 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.33377.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33377.log.gz 2024-04-19T01:07:07.155 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37662.log 2024-04-19T01:07:07.155 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28586.log 2024-04-19T01:07:07.155 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.36566.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32547.log 2024-04-19T01:07:07.155 INFO:teuthology.orchestra.run.smithi033.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36566.log.gz/var/log/ceph/ceph-client.admin.37662.log: 2024-04-19T01:07:07.155 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37662.log.gz 2024-04-19T01:07:07.155 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.28586.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28586.log.gz 2024-04-19T01:07:07.155 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35608.log 2024-04-19T01:07:07.155 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.32547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37246.log 2024-04-19T01:07:07.155 INFO:teuthology.orchestra.run.smithi033.stderr: 24.6% -- replaced with /var/log/ceph/ceph-client.admin.32547.log.gz 2024-04-19T01:07:07.155 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.35608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36525.log 2024-04-19T01:07:07.155 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35608.log.gz 2024-04-19T01:07:07.155 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34141.log 2024-04-19T01:07:07.155 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.37246.log: /var/log/ceph/ceph-client.admin.36525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28588.log 2024-04-19T01:07:07.156 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37246.log.gz 2024-04-19T01:07:07.156 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36525.log.gz 2024-04-19T01:07:07.156 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.34141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35972.log 2024-04-19T01:07:07.156 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34141.log.gz 2024-04-19T01:07:07.156 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.28588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29760.log 2024-04-19T01:07:07.156 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28588.log.gz 2024-04-19T01:07:07.156 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.35972.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35972.log.gz 2024-04-19T01:07:07.156 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37298.log 2024-04-19T01:07:07.156 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.29760.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29760.log.gz 2024-04-19T01:07:07.156 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.37298.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37298.log.gz 2024-04-19T01:07:07.157 INFO:teuthology.orchestra.run.smithi118.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-19T01:07:07.157 INFO:teuthology.orchestra.run.smithi110.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-19T01:07:07.157 INFO:teuthology.orchestra.run.smithi155.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-19T01:07:07.313 INFO:teuthology.orchestra.run.smithi033.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-19T01:07:07.409 INFO:teuthology.orchestra.run.smithi155.stderr: 91.3% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-19T01:07:07.512 INFO:teuthology.orchestra.run.smithi110.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-19T01:07:07.528 INFO:teuthology.orchestra.run.smithi155.stderr: 91.6% -- replaced with /var/log/ceph/ceph-osd.15.log.gz 2024-04-19T01:07:07.596 INFO:teuthology.orchestra.run.smithi033.stderr: 91.8% -- replaced with /var/log/ceph/ceph-osd.12.log.gz 2024-04-19T01:07:07.603 INFO:teuthology.orchestra.run.smithi118.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-04-19T01:07:07.629 INFO:teuthology.orchestra.run.smithi110.stderr: 91.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-19T01:07:07.743 INFO:teuthology.orchestra.run.smithi118.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.10.log.gz 2024-04-19T01:07:07.786 INFO:teuthology.orchestra.run.smithi033.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-19T01:07:07.791 INFO:teuthology.orchestra.run.smithi033.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.8.log.gz 2024-04-19T01:07:07.819 INFO:teuthology.orchestra.run.smithi155.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-04-19T01:07:07.838 INFO:teuthology.orchestra.run.smithi033.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-19T01:07:07.839 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-04-19T01:07:07.839 INFO:teuthology.orchestra.run.smithi033.stderr:real 0m0.871s 2024-04-19T01:07:07.839 INFO:teuthology.orchestra.run.smithi033.stderr:user 0m3.250s 2024-04-19T01:07:07.839 INFO:teuthology.orchestra.run.smithi033.stderr:sys 0m0.190s 2024-04-19T01:07:07.840 INFO:teuthology.orchestra.run.smithi118.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-19T01:07:07.876 INFO:teuthology.orchestra.run.smithi110.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.9.log.gz 2024-04-19T01:07:07.888 INFO:teuthology.orchestra.run.smithi155.stderr: 91.8% -- replaced with /var/log/ceph/ceph-osd.11.log.gz 2024-04-19T01:07:07.890 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-04-19T01:07:07.890 INFO:teuthology.orchestra.run.smithi155.stderr:real 0m0.875s 2024-04-19T01:07:07.890 INFO:teuthology.orchestra.run.smithi155.stderr:user 0m2.471s 2024-04-19T01:07:07.890 INFO:teuthology.orchestra.run.smithi155.stderr:sys 0m0.131s 2024-04-19T01:07:07.963 INFO:teuthology.orchestra.run.smithi110.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.13.log.gz 2024-04-19T01:07:07.964 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-04-19T01:07:07.964 INFO:teuthology.orchestra.run.smithi110.stderr:real 0m0.982s 2024-04-19T01:07:07.964 INFO:teuthology.orchestra.run.smithi110.stderr:user 0m2.928s 2024-04-19T01:07:07.965 INFO:teuthology.orchestra.run.smithi110.stderr:sys 0m0.164s 2024-04-19T01:07:07.970 INFO:teuthology.orchestra.run.smithi118.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.14.log.gz 2024-04-19T01:07:07.972 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-04-19T01:07:07.972 INFO:teuthology.orchestra.run.smithi118.stderr:real 0m0.972s 2024-04-19T01:07:07.972 INFO:teuthology.orchestra.run.smithi118.stderr:user 0m3.101s 2024-04-19T01:07:07.972 INFO:teuthology.orchestra.run.smithi118.stderr:sys 0m0.125s 2024-04-19T01:07:07.972 INFO:tasks.ceph:Archiving logs... 2024-04-19T01:07:07.972 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663223/remote/smithi033/log 2024-04-19T01:07:07.974 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-19T01:07:08.634 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663223/remote/smithi110/log 2024-04-19T01:07:08.635 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-19T01:07:09.047 DEBUG:teuthology.misc:Transferring archived files from smithi118:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663223/remote/smithi118/log 2024-04-19T01:07:09.048 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-19T01:07:09.502 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663223/remote/smithi155/log 2024-04-19T01:07:09.504 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-19T01:07:09.995 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-19T01:07:10.004 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-19T01:07:10.005 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-19T01:07:10.007 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-19T01:07:10.010 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-19T01:07:10.012 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-19T01:07:10.120 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-19T01:07:10.130 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-19T01:07:10.130 DEBUG:teuthology.orchestra.run.smithi033:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-19T01:07:10.137 DEBUG:teuthology.orchestra.run.smithi110:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-19T01:07:10.139 DEBUG:teuthology.orchestra.run.smithi118:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-19T01:07:10.142 DEBUG:teuthology.orchestra.run.smithi155:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-19T01:07:10.149 INFO:teuthology.orchestra.run.smithi033.stdout: remote refid st t when poll reach delay offset jitter 2024-04-19T01:07:10.149 INFO:teuthology.orchestra.run.smithi033.stdout:============================================================================== 2024-04-19T01:07:10.149 INFO:teuthology.orchestra.run.smithi033.stdout:+hv01.front.sepi 67.205.162.81 3 u 10 64 177 0.094 -3.044 1.746 2024-04-19T01:07:10.149 INFO:teuthology.orchestra.run.smithi033.stdout:+hv02.front.sepi 63.231.80.2 3 u 8 64 177 0.075 +2.091 2.341 2024-04-19T01:07:10.150 INFO:teuthology.orchestra.run.smithi033.stdout:*hv03.front.sepi 74.6.168.72 3 u 9 64 177 0.093 +0.753 2.407 2024-04-19T01:07:10.150 INFO:teuthology.orchestra.run.smithi033.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:07:10.151 INFO:teuthology.orchestra.run.smithi110.stdout: remote refid st t when poll reach delay offset jitter 2024-04-19T01:07:10.151 INFO:teuthology.orchestra.run.smithi110.stdout:============================================================================== 2024-04-19T01:07:10.151 INFO:teuthology.orchestra.run.smithi110.stdout:+hv01.front.sepi 67.205.162.81 3 u 6 64 177 0.102 -2.360 1.087 2024-04-19T01:07:10.151 INFO:teuthology.orchestra.run.smithi110.stdout:+hv02.front.sepi 63.231.80.2 3 u 5 64 177 0.060 -1.025 1.546 2024-04-19T01:07:10.151 INFO:teuthology.orchestra.run.smithi110.stdout:*hv03.front.sepi 74.6.168.72 3 u 11 64 177 0.079 -2.033 1.271 2024-04-19T01:07:10.151 INFO:teuthology.orchestra.run.smithi110.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:07:10.153 INFO:teuthology.orchestra.run.smithi155.stdout: remote refid st t when poll reach delay offset jitter 2024-04-19T01:07:10.153 INFO:teuthology.orchestra.run.smithi155.stdout:============================================================================== 2024-04-19T01:07:10.153 INFO:teuthology.orchestra.run.smithi155.stdout:*hv01.front.sepi 67.205.162.81 3 u 13 64 177 0.100 -1.740 0.482 2024-04-19T01:07:10.153 INFO:teuthology.orchestra.run.smithi155.stdout:+hv02.front.sepi 63.231.80.2 3 u 10 64 177 0.101 +0.537 0.475 2024-04-19T01:07:10.154 INFO:teuthology.orchestra.run.smithi155.stdout:+hv03.front.sepi 74.6.168.72 3 u 3 64 177 0.103 -0.807 0.478 2024-04-19T01:07:10.154 INFO:teuthology.orchestra.run.smithi155.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:07:10.154 INFO:teuthology.orchestra.run.smithi118.stdout: remote refid st t when poll reach delay offset jitter 2024-04-19T01:07:10.154 INFO:teuthology.orchestra.run.smithi118.stdout:============================================================================== 2024-04-19T01:07:10.154 INFO:teuthology.orchestra.run.smithi118.stdout:*hv01.front.sepi 67.205.162.81 3 u 5 64 177 0.143 -1.983 0.771 2024-04-19T01:07:10.154 INFO:teuthology.orchestra.run.smithi118.stdout:+hv02.front.sepi 63.231.80.2 3 u 15 64 177 0.109 +0.329 0.760 2024-04-19T01:07:10.154 INFO:teuthology.orchestra.run.smithi118.stdout:+hv03.front.sepi 74.6.168.72 3 u 3 64 177 0.109 -1.045 0.783 2024-04-19T01:07:10.154 INFO:teuthology.orchestra.run.smithi118.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:07:10.156 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-19T01:07:10.209 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-19T01:07:10.209 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-19T01:07:10.221 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-19T01:07:10.232 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-19T01:07:10.244 INFO:teuthology.task.internal:Duration was 843.258793 seconds 2024-04-19T01:07:10.244 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-19T01:07:10.257 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-19T01:07:10.257 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-19T01:07:10.259 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-19T01:07:10.262 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-19T01:07:10.264 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-19T01:07:10.302 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-19T01:07:10.303 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi033.front.sepia.ceph.com 2024-04-19T01:07:10.303 DEBUG:teuthology.orchestra.run.smithi033:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-19T01:07:10.350 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi110.front.sepia.ceph.com 2024-04-19T01:07:10.350 DEBUG:teuthology.orchestra.run.smithi110:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-19T01:07:10.361 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi118.front.sepia.ceph.com 2024-04-19T01:07:10.361 DEBUG:teuthology.orchestra.run.smithi118:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-19T01:07:10.372 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi155.front.sepia.ceph.com 2024-04-19T01:07:10.372 DEBUG:teuthology.orchestra.run.smithi155:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-19T01:07:10.383 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-19T01:07:10.383 DEBUG:teuthology.orchestra.run.smithi033:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-19T01:07:10.395 DEBUG:teuthology.orchestra.run.smithi110:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-19T01:07:10.409 DEBUG:teuthology.orchestra.run.smithi118:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-19T01:07:10.416 DEBUG:teuthology.orchestra.run.smithi155:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-19T01:07:10.453 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-19T01:07:10.453 DEBUG:teuthology.orchestra.run.smithi033:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-19T01:07:10.467 DEBUG:teuthology.orchestra.run.smithi110:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-19T01:07:10.476 DEBUG:teuthology.orchestra.run.smithi118:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-19T01:07:10.484 DEBUG:teuthology.orchestra.run.smithi155:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-19T01:07:10.549 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-19T01:07:10.559 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-19T01:07:10.560 DEBUG:teuthology.orchestra.run.smithi033:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-19T01:07:10.572 DEBUG:teuthology.orchestra.run.smithi110:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-19T01:07:10.585 DEBUG:teuthology.orchestra.run.smithi118:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-19T01:07:10.598 DEBUG:teuthology.orchestra.run.smithi155:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-19T01:07:10.608 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-19T01:07:10.619 DEBUG:teuthology.orchestra.run.smithi033:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-19T01:07:10.622 DEBUG:teuthology.orchestra.run.smithi110:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-19T01:07:10.631 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern = core 2024-04-19T01:07:10.632 DEBUG:teuthology.orchestra.run.smithi118:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-19T01:07:10.640 DEBUG:teuthology.orchestra.run.smithi155:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-19T01:07:10.641 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern = core 2024-04-19T01:07:10.650 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern = core 2024-04-19T01:07:10.662 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern = core 2024-04-19T01:07:10.685 DEBUG:teuthology.orchestra.run.smithi033:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-19T01:07:10.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T01:07:10.694 DEBUG:teuthology.orchestra.run.smithi110:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-19T01:07:10.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T01:07:10.716 DEBUG:teuthology.orchestra.run.smithi118:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-19T01:07:10.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T01:07:10.721 DEBUG:teuthology.orchestra.run.smithi155:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-19T01:07:10.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T01:07:10.731 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-19T01:07:10.742 INFO:teuthology.task.internal:Transferring archived files... 2024-04-19T01:07:10.743 DEBUG:teuthology.misc:Transferring archived files from smithi033:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663223/remote/smithi033 2024-04-19T01:07:10.743 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-19T01:07:10.766 DEBUG:teuthology.misc:Transferring archived files from smithi110:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663223/remote/smithi110 2024-04-19T01:07:10.766 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-19T01:07:10.788 DEBUG:teuthology.misc:Transferring archived files from smithi118:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663223/remote/smithi118 2024-04-19T01:07:10.788 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-19T01:07:10.819 DEBUG:teuthology.misc:Transferring archived files from smithi155:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663223/remote/smithi155 2024-04-19T01:07:10.819 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-19T01:07:10.841 INFO:teuthology.task.internal:Removing archive directory... 2024-04-19T01:07:10.841 DEBUG:teuthology.orchestra.run.smithi033:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-19T01:07:10.843 DEBUG:teuthology.orchestra.run.smithi110:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-19T01:07:10.846 DEBUG:teuthology.orchestra.run.smithi118:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-19T01:07:10.856 DEBUG:teuthology.orchestra.run.smithi155:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-19T01:07:10.885 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-19T01:07:10.898 INFO:teuthology.task.internal:Not uploading archives. 2024-04-19T01:07:10.898 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-19T01:07:10.910 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-19T01:07:10.911 DEBUG:teuthology.orchestra.run.smithi033:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-19T01:07:10.913 DEBUG:teuthology.orchestra.run.smithi110:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-19T01:07:10.915 DEBUG:teuthology.orchestra.run.smithi118:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-19T01:07:10.918 DEBUG:teuthology.orchestra.run.smithi155:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-19T01:07:10.918 INFO:teuthology.orchestra.run.smithi033.stdout: 658756 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 19 01:07 /home/ubuntu/cephtest 2024-04-19T01:07:10.919 INFO:teuthology.orchestra.run.smithi110.stdout: 658624 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 19 01:07 /home/ubuntu/cephtest 2024-04-19T01:07:10.921 INFO:teuthology.orchestra.run.smithi118.stdout: 658640 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 19 01:07 /home/ubuntu/cephtest 2024-04-19T01:07:10.932 INFO:teuthology.orchestra.run.smithi155.stdout: 658634 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 19 01:07 /home/ubuntu/cephtest 2024-04-19T01:07:10.933 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-19T01:07:10.946 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-19T01:07:10.974 INFO:teuthology.run:Summary data: description: rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} duration: 843.2587928771973 flavor: default owner: scheduled_yuriw@teuthology success: true 2024-04-19T01:07:10.974 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-19T01:07:11.074 INFO:teuthology.run:pass